EVALUATION 2014

Evaluation 2014 Conference

Evaluation 2014 Presidential Strand and Plenary Session
Video Recordings Purchase

 

Did you miss a session? Did you think a session would be great to share with your staff or colleague? Online hosted video recordings of the Presidential Strand and Plenaries from the Evaluation 2014 conference are now available for purchase! AEA recorded all Plenary sessions and Presidential Strand presentations so you and your colleagues can continue your learning when you return home.

Below are all of our recordings for purchase. The price for a single recorded session is $35 for AEA Members and $50 for Non-Members. The price for the entire package of recordings is $205 for AEA Members and $290 for Non-Members.

Watch the 5-minute video below for a description of the 2014 conference theme by AEA 2014 President Beverly Parsons. 

 

You must login or create an account to purchase products.

Login to Your Account

Create a New Account or Join AEA

 

 

Eval 2014: Evaluation and Measurement Bundle
Eval 2014: Evaluation and Measurement Bundle

Evaluation and Measuremenet Session Recordings.

$70: Members / $100: Non-Members

This option is designed for those interested in using the videos to aid in classroom or other professional development setting. With this package, you will receive a discussion guide prepared by a fellow evaluator as well as access to an online community to allow you to engage with other leaders in the field to continue the dialogue and share resources. The pre-packaged bundle includes the following sessions:

  • The Significance of Social Impact Measurement for Visionary Evaluation
  • The Significance of Social Impact Measurement For Visionary Evaluation (Plenary Session Follow-up)
  • How to Use Media and Media Analytics in a Visionary Evaluation
  • Discussion guide for how to best use and present the sessions

Eval 2014: Entire Series Package: Presidential Strand Presentations and Plenary Sessions
Eval 2014: Entire Series Package: Presidential Strand Presentations and Plenary Sessions

All Session Recordings.

$205: Members / $290: Non-Members

Eval 2014 Opening Session: Visionary Evaluation for a Sustainable, Equitable Future
Eval 2014 Opening Session: Visionary Evaluation for a Sustainable, Equitable Future

Single Session Recording.

$35: Members / $50: Non-Members

Plenary Leaders: Lovely Dhillon, Matt Keene, Beverly Parsons, Xiuhtezcatl Martinez

Visionary evaluation encourages us to focuses on what we deeply want the world to look like whether or not we can envision it as possible or accessible from where we currently are. Xiuhtezcatl Martinez, 14-year-old youth director of Earth Guardians, joins Beverly Parsons (2014 AEA President), and Matt Keene and Lovely Dhillon (2014 AEA conference program chair and co-chair) to challenge your thinking about a sustainable, equitable future. They will engage you in creating your own version of visionary evaluation that contributes to a future that matters to you, those you serve through your evaluation work, and future generations.

Eval 2014 Plenary: A Vision of Corporations Fit for the Future: Implications for Evaluation
Eval 2014 Plenary: A Vision of Corporations Fit for the Future: Implications for Evaluation

Single Session Recording.

$35: Members / $50: Non-Members

Plenary Leader: Bob Willard

Corporations are increasingly recognizing their place at the intersection of global economic, environmental, and equity issues. Bob Willard will bring us the business context for corporate social and environmental responsibility and help us see the connection to evaluation.

Bob will highlight the importance of standards and benchmarks as he links the corporate world with the field of evaluation. For example, many corporations are lulling themselves into thinking they are making sufficient progress if they keep nudging forward on their current course and at their current speed. Frequently used standards can easily celebrate progress toward anemic goals instead of goals that make a business fit for a sustainable future. Bob highlights standards that encourage organizations to create positive environmental, social, and economic value so that we have the possibility of sustaining a flourishing economy, society, and ecosystem on our finite planet.

 

Eval 2014 Plenary: The Significance of Social Impact Measurement for Visionary Evaluation
Eval 2014 Plenary: The Significance of Social Impact Measurement for Visionary Evaluation

Single Session Recording.

$35: Members / $50: Non-Members

Plenary Leaders: Georgette Wong, Tris Lumley, Nancy McPherson  
Panelists: Ian Davies, Tom Kelly, George Julnes

While public, non-profit, and philanthropic sectors have always been focused on serving the public interest, businesses are increasingly and innovatively creating social impact goals. These businesses include social ventures, social enterprises, corporations, and social investment funds. The increasing number of  businesses with social goals highlights a lack of consensus on how to measure or evaluate “social impact.”

Three presenters involved in social impact investing will illuminate trends in the field of social impact measurement. They will then engage in a lively dialogue with three seasoned evaluators  about the interplay of key evaluation issues and the trends in social impact measurement. The panel is designed to build our collective knowledge in order to move toward a set of guiding principles to inform both the evaluation and social impact communities.

 

Eval 2014 Closing Session: AEA's Role in a Sustainable, Equitable Future - A Community Conversation
Eval 2014 Closing Session: AEA's Role in a Sustainable, Equitable Future - A Community Conversation

Single Session Recording.

$35: Members / $50: Non-Members

Plenary Leaders: John Gargani, Riitta Oksanen, Stewart Donaldson, Rodney Hopson, and Beverly Parsons

Join your conference colleagues for table discussions, chats, and tweets about Now What? What is the role of AEA in supporting a sustainable, equitable future for all? How can AEA support your plans for visionary evaluation? How should AEA contribute to the global evaluation community in a way that, as stated in the AEA Guiding Principles, takes into account “the diversity of general and public interests and values… going beyond analysis of particular stakeholder interests to consider the welfare of society as a whole.”

Following table conversations, past, present, and future AEA presidents (Michael Scriven, Rodney Hopson, Beverly Parsons, Stewart Donaldson, and John Gargani) and the European Evaluation Society president (Claudine Voyadzis) engage in a conversation about AEA’s role in a sustainable, equitable future.

The session ends with our 2014 AEA poem written and read by Susan Robbins based on the conference theme. 

 

Eval 2014: A Sustainability Manifesto for Evaluation
Eval 2014: A Sustainability Manifesto for Evaluation

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Matt Keene 
Presenters: George Julnes (Presenting on behalf of Eleanor Chelimsky), Baljit Wadhwa, Beverly Parsons, Andy Rowe, Alejandro Ortega-Argueta, Claudia Romero

In this session, we will speak with colleagues who are looking at sustainability through an evaluation lens. Two kinds of sustainability come into focus: the sustainability of the intervention and the sustainability of human and natural systems related to the intervention. Panelists will discuss why sustainability turns our attention toward program origins and public interest values; the dynamics of human and natural systems and the tradeoffs that emerge at their nexus; strong versus weak sustainability; and the micro - macro paradox where project level successes are heralded but enveloped by large scale failures. Panelists will present ways for evaluators to think about sustainability. They will begin to clarify the practical implications of the sustainability concept for evaluation practice and research. Panelists will build upon the concepts and discussions introduced in this session to their perspectives on a sustainability manifesto to guide the conduct of evaluation.

Eval 2014: The Evaluator's Role in Building Sustainable, Equitable Communities
Eval 2014: The Evaluator's Role in Building Sustainable, Equitable Communities

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Patricia Rogers
Presenters: Abby Ruskey, Chris Y. Lovato

This session illustrates how evaluators can help communities become sustainable and equitable. One presenter describes a statewide visioning process that helped define what is meant by sustainable communities and the role of education. It includes the establishment of a set of outcomes that position a community to create positive environmental, social, and economic value. Another presenter illustrates how an evaluation is shifting as a community moved rapidly forward in economic and social change stimulated by health care issues. The session illustrates how evaluators can play important roles in assisting social systems to transform their norms, policies, infrastructures and practices to support such communities.

Eval 2014: Local, State, and National Case Examples of Visionary Evaluation
Eval 2014: Local, State, and National Case Examples of Visionary Evaluation

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Susan G Berkowitz 
Discussant: Nick L Smith 
Presenters: Nada Rayyes, Linda Toms Barker, Kay Magill, Susan G Berkowitz 

This panel will draw on case examples of four very different evaluations conducted at the local, state and national levels to explore the question: What do we mean by sustainability in the context of program evaluation? Panelists will examine such issues as: how have evaluators sought to conceptualize, operationalize and measure sustainability; How do we promote sustainability, given budget constraints; who should be sustained, and who decides; how successful have we been at measuring and promoting sustainability in different circumstances ,and why? Panelists  offer “lessons learned” for the future and consider both how evaluation influences sustainability, and how sustainability, in turn, can influence evaluators and evaluation plans and designs. The discussant draw out common points for further discussion and audience participation will be actively encouraged.

Unexpected Aspects of Sustainability in an Evaluation of a Community Based Health Program for Older Lesbian/Bisexual Women

The Office on Women’s Health has funded five projects nationwide to develop and test interventions aimed at improving the health of older lesbian and bisexual (LB) women at risk of weight-related illnesses. This talk will focus on one San Francisco-based program “Doing it for Ourselves”  (DIFO), which includes 12 weekly health education group sessions. While sustainability was built into the program design, it played out in unexpected ways. The evaluator will discuss challenges that arose in maintaining DIFO community partnerships designed to foster sustainability; and the successes evidenced by participants taking ownership and forming their own DIFO groups after the 12-week funded program ended. The presenter, who serves as the program evaluator, is in a unique position to discuss this aspect of sustainability, as she is both part of the DIFO collaboration and outside of it (a non-participant). Implications for evaluation planning and measuring sustainability will be explored.


Sustainability as Institutionalization: Strategies for Adoption of Pono Choices Middle School Curriculum

Pono Choices is a culturally responsive teen pregnancy and Sexually Transmitted Infection (STI) prevention program designed to provide middle school students with the knowledge, attitudes, and skills necessary to reduce their risk of STIs and unintended pregnancy by incorporating medically accurate information, character education and the traditions and practices of the Hawai’ian host culture. Funded by the U.S. Office of Adolescent Health, the University of Hawaiii has partnered with Alu Like and Planned Parenthood to develop and implement the curriculum, with IMPAQ International as the external evaluator. Pono Choices is being tested for efficacy through a random assignment impact evaluation, with the expectation that the partners will sustain the curriculum beyond the grant period if it proves effective. Surprisingly, the Hawaii Department of Education approved use of the curriculum by non-study schools before the evaluation was complete and the program has become the focus of great political controversy.

Sustainability as Systems Change: Evaluating Sustainable Impacts of the Disability Employment Initiative

As of November 2012, the labor force participation rate for individuals with disabilities was 48.5 percent lower than for those who do not have a disability (BLS, 2012). The Disability Employment Initiative (DEI) is a grant program awarded to select states by the US Department of Labor to expand the capacity of the U.S. Department of Labor’s American Job Centers (AJCs) to serve customers with disabilities and to promote system change in the public workforce system. In partnership with Social Dynamics, LLC, IMPAQ International is conducting a mixed methods evaluation of the national initiative that includes data sharing agreements and site visits to participating states. In addition to measuring outcomes for customers with disabilities, the national evaluation includes identifying seven “systems change” domains and developing indicators for measuring grantee success in achieving sustainable systems change outcomes.

Sustainability as Institutionalization: Developing Institutional-Level Indicators of Gender-Equitable Institutional Transformation


The National Science Foundation’s ADVANCE Institutional Transformation (IT) Awards seek to make an impact through organizational change strategies designed to reduce or eliminate organizational barriers to the full participation of women in academic science, technology, engineering and math (STEM) fields. They aim to promote a more hospitable institutional environment for women in STEM. Westat conducted a national evaluation of the first two cohorts of 19 IHEs receiving IT Awards. Drawing on a literature review and a survey and follow-up discussions with institutional representatives, the evaluators developed preliminary institutional-level indicators or markers of progress towards institutionalization, and thus sustainability, of gender-equitable changes. This presentation will highlight these indicators and consider the broader potential applicability of both the indicators and this approach for other efforts to conceptualize and assess sustainability of programs aimed at transforming IHEs and other institutions.

 

Eval 2014: Corporate Sustainability and Evaluation
Eval 2014: Corporate Sustainability and Evaluation

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Cameron Norman
Presenters: Bob Willard, Michael Quinn Patton  

This session will provide an opportunity for questions and reflections related to the plenary session A Vision of Corporations Fit for the Future: Implications for Evaluation in which Bob Willard talks about how businesses are addressing economic, social, and environmental sustainability and the role of standards and benchmarks. He will be joined in this discussion by Michael Patton to look more in depth at various links between evaluation and how businesses are addressing sustainability.

 

Eval 2014: Money Well Spent? Funder, NGO, and Evaluator Perspectives on Funding and Evaluating Advoc
Eval 2014: Money Well Spent? Funder, NGO, and Evaluator Perspectives on Funding and Evaluating Advoc

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Bess Rothenberg [Wellspring Advisors]
Presenters: David Devlin-Foltz, Helena Choi, Mustafa Kudrati, Nan Wehipeihana

Advocacy and policy change evaluation provokes lively debate: Can it be rigorous? Does it add value? Changing policy is tough, especially when the desired change is controversial. Knowing whether advocates contributed to that change is tougher still; multiple factors and actors interact in complex ways. Funders who support policy change, especially in contexts far from home, face additional challenges. The William and Flora Hewlett Foundation collaborated with the Aspen Planning and Evaluation Program on a four-year evaluation of the Money Well Spent portfolio of family planning and reproductive health advocacy grantees, including Pathfinder International’s efforts in Tanzania. Representatives of all three will discuss lessons for funders, evaluators, and advocates from their experience collaboratively applying principles from developmental evaluation to assessing, adjusting and improving advocacy and evaluation. The session will underscore the role of funders, outside evaluators and local advocacy partners.

The evaluator as “critical friend” in a tough neighborhood

 The Aspen Planning and Evaluation Program (APEP) focused initially on policy change processes in the United States. Its best-known evaluation tool, the Advocacy Progress Planner, was initially intended for use by health policy advocates in California. Its earliest users focused on US foreign policy decisions related to the Iraq War, arms control treaties, and Guantanamo detainees. At the Hewlett Foundation’s invitation, APEP worked with the Money Well Spent grantees to apply the Planner in wildly diverse contexts from Abuja to Paris to Dar es Salaam. Working flexibly and collaboratively with grantees including Pathfinder International’s Tanzania office, APEP learned that the “tough questions” about strategy -- and the willingness to answer them honestly and frequently -- were more important than the tool itself. This presentation will focus on the Money Well Spent portfolio from the evaluator’s perspective, that of a “critical friend” to the grantees and funder.

 Funders need to let go and let others lead

Advocacy efforts almost never follow a straight line between points A and B. A savvy advocate knows that success requires constantly adjusting to the changing contexts. But savvy advocates need room to maneuver. Funders can help create that room by agreeing that maintaining a focus on the advocacy goal is more important than adhering to an agreed work plan. Funders need to trust that changing advocacy tactics mid-course is a sign that grantees are learning and adapting as needed. Funders’ own understanding of the context may make them more willing to approve or even recommend changes in the work plans. The presentation will focus on a key lesson learned as an advocacy funder: empower whoever has the best knowledge of the local context – allow them to judge conditions, make the case for strategy change, and even make unilateral decisions when quick adjustments are needed to seize fast-breaking opportunities.

"So what?” Keeping eyes on the prize

Pathfinder’s initial efforts under the Money Well Spent Project focused on strategies and activities identified in the original proposal. However, changes in key decision-makers, cancellation of the Global Fund Rounds, and increased availability of contraceptives from external funders demanded an adjustment in strategies throughout the implementation period. This presentation will share how a process of internal learning, focusing on what the organization was trying to achieve rather than getting stuck on a particular way of doing things, allowed the project to identify and take advantage of a simple and yet strategic opportunity to institutionalize local government budgeting for family planning in Tanzania. The presentation will also share the importance of engagement with the funder and evaluator in an honest dialogue and how their flexibility freed the organization to focus on the intended outcomes.
 



Eval 2014: How to Use Media and Media Analytics in a Visionary Evaluation
Eval 2014: How to Use Media and Media Analytics in a Visionary Evaluation

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Lovely Dhillon 
Presenters: Dan Green, Dana Chinn, Lindsay Green-Barber 

This panel will explore the fast-changing world of media analytics. Panelists will provide examples of frameworks for evaluating the impact of media through different platforms (e.g., social media, television). Panelists will also discuss the use of media analytics in measuring concepts such as audience engagement. This panel will also explore the impact of media, especially social media, on social change and how media analytics can provide an organization insight into its own impact.

 

Eval 2014: Evaluating Complexity while Creating Equitable and Sustainable Systems Change
Eval 2014: Evaluating Complexity while Creating Equitable and Sustainable Systems Change

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Hallie Preskill 
Discussant: Glenda Eoyang 
Presenters: Srik Gopalakrishnan, Katelyn Mack, Joelle Cook 

Traditional evaluation tools and approaches fall short when it comes to evaluating complex and intricate systems change efforts. This session will explore a set of bold “propositions” for evaluating complexity and will seek to illustrate the propositions through real-life examples from three multifaceted interventions including: Building Healthy Communities in California; Challenge Scholars in Michigan; and the Alberta Family Wellness Initiative in Alberta, Canada. Each intervention is unique; however, they all demonstrate two common characteristics: 1. They strive to create equity and sustainability (across health, education, and wellbeing) in their communities and 2. they represent an “emergent” rather than a predictable approach to strategy, and correspondingly, to evaluation. The session will begin with an introduction to the propositions for evaluating complexity by the chair, followed by presentations by an evaluator-client pair from each of the three examples, and will conclude with remarks by the discussant on implications for future evaluations of complex initiatives and audience small group discussions and Q&A.

Challenge Scholars: Lessons Learned in Evaluating a Complex Program in Its Early Stages

The presentation will highlight a specific case example of evaluating complexity – the evaluation of the “Challenge Scholars” program in Grand Rapids, Michigan, which strives to create more equitable access to college access and success that would lead to greater vibrancy and sustainability in the Grand Rapids community. Through the case example, the presentation will illustrate some of the propositions in action – for example, how evaluation design, implementation, and reporting need to change in order to accommodate an emergent approach. In addition, the presentation will discuss successful strategies in partnering with the client on the evaluation, innovative data collection and reporting methods used, and promising techniques for “sense-making” around the data. The presentation will be delivered by Srik Gopalakrishnan, the lead FSG evaluator on the case, along with Cris Kutzli from the Grand Rapids Community Foundation, the leader of the Challenge Scholars initiative and primary evaluation client.

The California Endowment: Embedding Learning Opportunities throughout a Statewide Strategy to Improve Health

Since the start of Building Healthy Communities, The California Endowment (TCE) has taken an innovative and bold approach to improving health in communities statewide. Rather than lay out a predetermined path for their 10-year, billion dollar strategy, TCE has embraced an emergent approach. As a result, TCE has invested considerable resources into learning from and with their staff, grantees, and partners. The presentation will highlight how TCE has chosen to press pause at a mid-course in their strategy through the commissioning of a strategic review, which will continue to inform the Endowment’s future work. Presenters will illustrate the propositions by discussing the successes and challenges with evaluating a complex, multi-faceted, emergent strategy mid-course, such as lingering questions of strategic “clarity” and “cohesion” while evaluating an evolving theory of change. The presentation will be delivered by Katelyn Mack, the lead FSG evaluator on the case, along with either Mona Jhawar or Jim Keddy from TCE.

Alberta Family Wellness Initiative: Lessons Learned in Evaluating an Initiative on the Knife’s Edge

The Norlien Foundation launched AFWI in 2009 in order to catalyze systems change around improving health outcomes for children and families in Alberta – with a particular emphasis on early childhood development and addiction and mental health outcomes. Now in its sixth year, the initiative continues to navigate complex political and relational dynamics as it seeks new and better ways to share knowledge about brain science, change the hearts and minds of direct service providers in the province, and deepen the momentum for broad-based systems change in the province. Presenters will share how the principles of evaluating complexity were used to frame and guide the nine-month evaluation, including designing the evaluation, collecting data, and engaging initiative staff and stakeholders in the findings. The presentation will be delivered by Joelle Cook, who is leading the case for FSG, along with James Radner, who is serving as an advisor to the Norlien Foundation.

 

Eval 2014: Violence Against Women: A global Crisis and A Challenge for Evaluators
Eval 2014: Violence Against Women: A global Crisis and A Challenge for Evaluators

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Donna M Mertens
Discussant: Michael Bamberger
Presenters: Sharlene Hesse-Biber, Brian Heilman, Tessie Tzavaras Catsambas, Shelah Bloom 

Violence against women (VAW) and gender based violence (GBV) are among the most serious political, psycho-social, and economic challenges facing countries in the 21st Century. Despite their pervasiveness, VAW and GBV are often not considered priority issues in evaluations by  sectors such labor markets and employment, education, and access to public services, where GBV may be a serious issue. Many evaluations either do not address these issues or use data collection methods unsuited to capture data on sensitive issues such as domestic or public violence or gender politics in the labor market. In this panel, four leading specialists will discuss the nature of VAW and GBV in different contexts: human trafficking and the sex trade, sexual assaults on college campuses; GBV within HIV prevention; engaging men and boys in the prevention of GBV; and how to improve the quality of VAW and GBV monitoring and evaluation.

A culture of violence and the structural roots of gender-based violence

We adopt a “Culture of Violence” approach targeting societal and community institutions that oppress and devalue women as opposed to focusing on individualist explanations blaming women for their own oppression. 

We briefly examine U.S. domestic sex trafficking. Most women recruited by the U.S. sex trafficking industry, which ranges from large-scale crime organizations and sex tourism to small networks of “pimps”, are American-born minors. We examine several successful programs that focus on the “demand” side of sex trafficking—targeting “Johns” who hire sexual services of female trafficking victims.

We also address the rise in sexual assaults on US college campuses. Approximately one in four college women will experience some form of gender violence, yet colleges and universities are slow to respond, report and prevent these attacks. A student who attended The White House Task Force to Protect Students from Sexual Assault will discuss the impact of bystander intervention prevention programs. 

Engaging men and boys in the prevention of gender-based violence: Opportunities, challenges and risks for programmers and evaluators

Policymakers and programmers working to end the global epidemic of gender-based violence are now embracing the idea that men and boys can be allies, rather than obstacles, to their work. As many programs emerge to support men who strive for communities free of violence, and as evaluations point toward effective ways to transform the harmful attitudes underlying other men’s use of violence, lessons are emerging. Drawing heavily on international examples, this presentation will first summarize the major trends in this program approach, pointing to opportunities for increased scale and effectiveness in future work. Second, the presentation will shed light on salient challenges and risks for practitioners as well as evaluators connected to this approach. Finally, in line with the panel’s objective, it will draw on the same opportunities, challenges, and risks to propose recommendations for evaluators seeking to incorporate an awareness of gender and violence into their work more broadly.

Addressing and responding to gender-based violence in HIV prevention, care and treatment programs

AIDSTAR One (http://www.aidstar-one.com/ )developed a guide to help the U.S. President's Emergency Plan for AIDS Relief (PEPFAR) program managers address and respond to GBV within HIV prevention, care, and treatment programs. The framework presents a comprehensive framework for integrating systemic questions that integrate GBV in evaluation practice including through direct queries of GBV survivors, community participatory inquiry to understand the root causes of violence, capacity assessment for service providers, and analysis of policy to assess the capacity of the enabling environment for preventing, addressing, and ultimately end GBV. The systemic framework presented offers evaluators lines of inquiry they can adapt for any evaluation. The presentation will share the framework, and then provide a field example of its application.

Designing quality monitoring and evaluation systems to prevent and respond to gender-based violence

The last decade has seen an increased commitment to programmatic and policy efforts directed at preventing and responding to GBV, by donor organizations such as USAID, WHO, and other UN affiliates. However, there is a dearth of rigorous evaluations of GBV programming, resulting in a lack of data to support evidence for best practices in the field. Most of these programs do not have quality M&E systems that are capable of effectively tracking progress in attaining their stated objectives. Even fewer programs have been rigorously evaluated. The focus of this presentation will be on what is needed to monitor and evaluate efforts to prevent and respond to GBV.

 

Eval 2014: The Significance of Social Impact Measurement: Keynote Follow-up Conversation
Eval 2014: The Significance of Social Impact Measurement: Keynote Follow-up Conversation

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Jane Linda Reisman,
Presenters: Georgette Wong, Tris Lumley, Nancy McPherson, Ian Davies, Thomas Kelly, Jr., George Julnes  

This session will provide an opportunity for questions and reflections related to the plenary session “The Significance of Social Impact Measurement for Visionary Evaluation”.  The panelists will be available for a deeper dive into how the fields of evaluation and impact measurement converge and diverge.  Participants will also be asked to contribute their viewpoint about the factors to consider in developing a set of principles that would be relevant for the impact measurement enterprises that is rapidly growing in the impact investing, social venture, social enterprise and corporate spaces.  This input will be valuable to the development of guiding principles that this panel is creating and plans to widely share in both the evaluation and impact measurement communities—with a particular eye toward the international year of evaluation in 2015.

 

Eval 2014: Building New Partnerships to Promote the Use of Evaluation Among Public Policymakers
Eval 2014: Building New Partnerships to Promote the Use of Evaluation Among Public Policymakers

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Rakesh Mohan 
Discussant: William Trochim
Presenters: Bill Pound, Michael Caudell-Feagan, Rich Wills 

The conference theme asks us to “envision evaluation as fundamental to a sustainable, equitable future” and to “unleash the power of evaluation…to impact the world in positive ways.” To unleash that power, we will need to build new partnerships with those who have access to policymakers and understand the nuances of the public policy process. These partnerships will help us better understand policymakers’ information needs and will allow us to educate them about the value of evaluation in making informed policy and program decisions that would help create a sustainable and equitable future. This highly interactive session brings together experts from three organizations that have worked with policymakers for decades: the National Conference of State Legislatures, the Council of State Governments, and the Pew Charitable Trust. The audience will hear from a senior legislator who is using the work of these organizations to improve criminal justice policies in his state.

 

Eval 2014: From Visionary Evaluation to Exemplary Evaluations in a Multicultural World - Pre...Part1
Eval 2014: From Visionary Evaluation to Exemplary Evaluations in a Multicultural World - Pre...Part1

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Michael Quinn Patton 
Discussant: Stafford Hood 
Presenters: Beverly Parsons, Stewart Donaldson, Marco Segone, Claudine Voyadzis  

President Beverly Parsons summarizes some of the key findings and insights that have been gleaned about visionary evaluation during 2014. President-elect Stewart Donaldson will introduce AEA’s 2015 conference theme, linking it to both Visionary Evaluation and the International Year of Evaluation. The 2015 theme, Exemplary Evaluations in a Multicultural World, aspires to promote sharing and learning from one another about exemplary evaluations and evaluation approaches across diverse cultures and national boundaries. A special emphasis will be placed on learning from extraordinary examples of influential evaluations and enabling environments for evaluation in both developed and developing countries across the globe.

 

Eval 2014: From Visionary Evaluation to Exemplary Evaluations in a Multicultural World - Pre...Part2
Eval 2014: From Visionary Evaluation to Exemplary Evaluations in a Multicultural World - Pre...Part2

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Michael Quinn Patton
Discussant: Stafford Hood 
Presenters: Beverly Parsons, Stewart Donaldson, Marco Segone, Claudine Voyadzis  

President Beverly Parsons summarizes some of the key findings and insights that have been gleaned about visionary evaluation during 2014. President-elect Stewart Donaldson will introduce AEA’s 2015 conference theme, linking it to both Visionary Evaluation and the International Year of Evaluation. The 2015 theme, Exemplary Evaluations in a Multicultural World, aspires to promote sharing and learning from one another about exemplary evaluations and evaluation approaches across diverse cultures and national boundaries. A special emphasis will be placed on learning from extraordinary examples of influential evaluations and enabling environments for evaluation in both developed and developing countries across the globe.

 

Eval 2014: Cradle to Cradle Evaluation: Efficient and Waste-Free Use of Evaluation Knowledge
Eval 2014: Cradle to Cradle Evaluation: Efficient and Waste-Free Use of Evaluation Knowledge

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Matt Keene
Discussant: Matt Keene 
Presenters: Gabriela Fitz, Nancy MacPherson 

Not long ago,  it was normal to use an aluminum can once and then toss it into a garbage bag destined for a landfill. Today, many producers and consumers increase efficiency and reduce waste through recycling and reuse. How much of evaluation knowledge is destined for a landfill? How much of it could be reused or recycled and how? In this session, we will introduce the lifecycle of evaluation knowledge from an ecosystem perspective, including its production, management, use, reuse and sustainability. We will explore persistent challenges associated with reusing knowledge and present examples of projects and technologies where the principles of reuse have been central and will help to reduce evaluations lost to the cradle to grave single client, single project model. Through fits of unpredictable storytelling and effective presentation, we will show and tell how cradle to cradle evaluation is integral to visionary evaluation.

Supportive Technologies for Evaluation Discovery and Synthesis

Gabriela Fitz will present the work the Foundation Center has undertaken with the Rockefeller Foundation to identify and develop potential technical solutions and methodological paths toward the collection, sharing, and synthesis of existing evaluative knowledge. Her presentation will include a discussion of some of the unique challenges that arise when synthesizing what is a largely unstructured and methodologically diverse body of literature. She will also demonstrate some of new tools, developed in partnership with the Rockefeller Foundation, for identifying common themes across a body of literature as well as drawing formerly excluded voices and perspectives into literature searches and reviews.

Reinventing the wheel? Why funders (should) have an obligation to build on what is known

Nancy MacPherson will discuss the approach that the Rockefeller Foundation is exploring to respond to the challenges and opportunities of using evaluative learning to inform tomorrow’s decisions and development solutions. Drawing from the experience of the Rockefeller Foundation’s Evaluation Office, partners and grantees in Asia and Africa, Nancy will highlight innovative and experimental approaches to evaluative learning in emergent spaces, as well as the use of new technology and practices to bring in a broader range of voices to enrich learning, accountability and sense making.

 

Eval 2014: Leading Toward the Emerging Future: The Issues/Challenges of Integrating Systems Thinking
Eval 2014: Leading Toward the Emerging Future: The Issues/Challenges of Integrating Systems Thinking

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Janice Noga 
Presenters: Mary McEathron, Erin Watson, Ginger Fitzhugh, Margaret Hargreaves

Last year, the Systems in Evaluation TIG hosted an open-space design charrette during its business meeting using an applied example – AEA’s annual meeting – as a focus for designing a systems-based evaluation. Four work teams came together during 2014, each focusing on a different systems approach (soft systems, developmental evaluation, complex adaptive systems, and network analysis). In this panel session, the teams will discuss issues and challenges, lessons learned, and critical considerations related to engaging other stakeholders – clients, beneficiaries, funders, and policy makers – in bringing a systems perspective to program evaluation and improvement. This will be followed by audience discussion facilitated by members of our four teams to learn what other evaluators are grappling with when trying to apply systems thinking to evaluation practice. Our goal: to learn how to better support evaluators in the use of systems thinking for visionary evaluation that contributes to a sustainable, equitable future.

A Soft Systems Approach to Evaluation

Mary McEathron and Bob Williams have been guiding the Soft Systems team in its learning and design process. Using soft systems methods, the team framed the focus of their evaluation work by examining the system’s boundaries, relationships, and perspectives. A key characteristic of soft systems is the underlying assumption that a system can be understood in many ways, depending on who is doing the framing and the perspective from which they are describing the system. Thus, the work of the soft systems team was couched first in defining and modeling the “ideal” system from multiple perspectives as a basis for examining the “real” system.

A Complex Adaptive Systems Approach to Evaluation

Systems team in its learning and design process. A key component of the process was identifying in what ways the AEA conference exhibits key qualities and constructs of a complex adaptive system, including self-organization, emergence, and co-evolution. The work of the Complex Adaptive Systems Team also explored how this evaluation approach could examine ways in which the conference experience influences new patterns of understanding and action across participants and the conditions under which these new patterns emerge and are sustained. In particular, the team focused on evaluation questions related to three conditions influencing these patterns: containers, exchanges, and “differences that made a difference.”

A Developmental Evaluation Approach

Ginger Fitzhugh and Michael Quinn Patton have been guiding the Developmental Evaluation team in its learning and design process. Developmental evaluation is informed by principles for operating in complex adaptive systems, systems thinking, and sensitivity to complex, nonlinear dynamics. The ongoing, iterative nature of the developmental evaluation approach supports social innovation and adaptive management. Thus, rather than focus on developing questions, the team focused on defining a process for generating and answering questions that would be sensitive to context and responsive to existing and emergent patterns and trends.

A Network Analysis Approach to Evaluation

Meg Hargreaves and Todd Honeycutt have been guiding the Network Analysis team in design work and discussion focused on examining the networks that support and interact with the AEA annual meeting. The team based its work in defining a system as “an interacting, interdependent network of actors working together to achieve a common goal.” Starting from a general question concerning the role of AEA’s formal and informal networks in the system bounded by the annual meeting, the team’s design work focused on key characteristics and functions of networks, such as TIGs, in shaping and influencing the conference experience.

Eval 2014: Evaluation Criteria for Judging Value: Application to Sustainability and Equity
Eval 2014: Evaluation Criteria for Judging Value: Application to Sustainability and Equity

Single Session Recording.

$35: Members / $50: Non-Members

Session Chair: Jennifer C Greene 
Discussant: E Jane Davidson 
Presenters: Ghislain Arbour, Mathea Roorda, George Julnes, Ernest Robert House 

Evaluation is generally distinguished from research by the additional task of judging the value of what is being evaluated in terms of merit, worth, or importance. When applied to consumer products, this valuing is conducted in terms of identified criteria relevant to judgments of quality (e.g., safety and performance of a car). However, when evaluating social programs and policies the choice of criteria is often controversial. This lack of consensus on criteria calls into question the value judgments, and, yet, there has been insufficient discussion in the evaluation community of how criteria should be chosen. The papers in this session seek to highlight areas of needed discussion and also offer frameworks to be considered in developing more systematic approaches to selecting and applying criteria in evaluation.

The problem as the source of answers: How criteria of merit can be derived from the systematic study of social problems

 In order to judge a program, we must understand the reasons an intervention is needed. We cannot find relevant criteria without understanding the social problem that might necessitate the intervention. A framework is proposed to investigate social problems in such a way that we can identify good criteria of merit in evaluations. This will be done utilizing a set of questions about a social problem: (a) its moral nature (why is it bad?); (b) its causal mechanics (how does it work?); (c) its intensity (how bad is it?); and (d) the potential collective responsibility we have over them (is it a public or a purely private matter?). These questions, when feasible, will be linked to known methodologies from social science. The framework explains how to organize these questions in such a way they can unveil the justification for governmental programs.

What are the characteristics of criteria? Investigating the requirements for defensible evaluation

Criteria for making evaluative judgments provide us with the dimensions on which performance is rated and ought to be explicit and transparent in every evaluation we do. They are crucial to establishing the basis for defensible evaluative conclusions (Fournier, 1995; Henry, 2002). However, read the methodology section of evaluation reports and you will notice there is rarely explicit explanation of how criteria have been selected, developed and justified.

In this paper, a set of draft ‘criteria for developing criteria’, with particular reference to public sector evaluations, will be presented and discussed. The criteria derive from a modified Delphi study completed as the first stage of a broader research project aimed at strengthening our ability, as professional evaluators, to deliver evaluative conclusions that are warranted.

The impact of methodology on the application of criteria in evaluation: What we know and need to know about a mixed methods approach to valuing

Standard evaluation theory calls for establishing criteria with which to judge the merit and worth of an evaluation. However, this can be problematic in evaluating social programs intended to serve the public interest. First, people differ in the criteria they feel best capture a “good society” — while this year’s conference theme presumes the importance of an “equitable society,” others give primacy to an efficient society. Furthermore, the literature is full of accounts of other values (e.g., security, privacy, and community) that warrant use as criteria for social programs. Accordingly, Chelimsky (2013) recommends “development of measurement systems and data collection on public-interest elements, such as equality of opportunity, equality before the law, protection of minority rights, protection against economic inequality, and so on.” This presentation will review the literature on methodologies for such measurement and suggest needed future research for a mixed methods framework for applying criteria in judging value.