Many programs fail to start with a clear description of the program and its intended outcomes, undermining both program planning and evaluation efforts. The logic model, as a map of what a program is and intends to do, is a useful tool for clarifying objectives, improving the relationship between activities and those objectives, and developing and integrating evaluation plans and strategic plans.

First, we will recapture the utility of program logic modeling as a simple discipline, using cases in public health and human services to explore the steps for constructing, refining and validating models. Then, we'll examine how to improve logic models using some fundamental principles of "program theory", demonstrate how to use logic models effectively to help frame questions in program evaluation, and show some ways logic models can also inform strategic planning. Both days use modules with presentations, small group case studies, and debriefs to reinforce group work.

You will learn:

• To construct simple logic models, To use program theory principles to improve a logic model;
• To identify and engage program stakeholders using a logic model;
• To develop an evaluation focus based on a logic model;
• To use logic models to answer strategic planning.

Session 4: Logic Models
Scheduled: Monday and Tuesday, October 31 and November 1, 9 AM to 4 PM
Level: Beginner, no prerequisites

5. Strategies for Interactive Evaluation Practice

In all of its many forms, evaluation practice requires evaluators to be skilled facilitators of interpersonal interactions. Whether you are completely in charge, working collaboratively with program staff, or coaching individuals conducting their own study, you need to interact with people throughout the course of an evaluation. This workshop will provide theoretical grounding (social interdependence theory, conflict theory, and evaluation use theory) and practical frameworks for analyzing and extending your own practice.

Through presentations, discussion, reflection, and case study, you will learn and experience strategies to enhance involvement and foster positive interaction in evaluation. You are encouraged to bring examples of challenges faced in your own practice to this workshop consistently lauded for its ready applicability to real-world evaluation contexts.

You will learn:

• Strategies to foster effective interaction, including belief sheets; values voting; three-step interview; cooperative rank order; graffiti; jigsaw; and data dialogue;
• Responses to challenges in participatory evaluation practices;
• Four frameworks for reflective evaluation practice.

Jean King has over 30 years of experience as an award-winning teacher at the University of Minnesota. As an evaluation practitioner, she has received AEA’s Myrdal award for outstanding evaluation practice. Laurie Stevahn is associate professor of educational leadership at Seattle University with extensive facilitation experience as well as applied experience in evaluation. The two are co-authors of Interactive Evaluation Practice: Mastering the Interpersonal Dynamics of Program Evaluation (Sage, forthcoming) and Needs Assessment Phase III: Taking Action for Change (Sage, 2010).

Session 5: Interactive Practice
Prerequisites: Basic evaluation skills
Scheduled: Monday and Tuesday, October 31 and November 1, 9 AM to 4 PM
Level: Intermediate

6. TBA

7. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use

Developmental evaluation (DE) is especially appropriate for innovative initiatives or organizations in dynamic and complex environments where participants, conditions, interventions, and context are turbulent, pathways for achieving desired outcomes are uncertain, and conflicts about what to do are high. DE supports reality-testing, innovation, and adaptation in complex dynamic systems where relationships among critical elements are nonlinear and emergent. Evaluation use in such environments focuses on continuous and ongoing adaptation, intensive reflective practice, and rapid, real-time feedback. The purpose of DE is to help develop and adapt the intervention (different from improving a model).

This evaluation approach involves partnering relationships between social innovators and evaluators in which the evaluator’s role focuses on helping innovators embed evaluative thinking into their decision-making processes as part of their ongoing design and implementation initiatives. DE can apply to any complex change effort anywhere in the world. Through lecture, discussion, and small-group practice exercises, this workshop will position DE as an important option for evaluation in contrast to formative and summative evaluations as well as other approaches to evaluation.

You will learn:

• The specific niche for which developmental evaluation is appropriate and useful;
• To understand and distinguish five different types of DE and the implications of those types;
• To identify the dimensions of complexity that affect how DE is;
• Practical frameworks and innovative methods for use in DE.

Michael Quinn Patton is an independent consultant and professor at the Union Institute. An internationally known expert on Utilization-focused Evaluation, this workshop is based on his just published new book, Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use (Guilford, 2010).

Session 7: Developmental Evaluation
Scheduled: Monday and Tuesday, October 31 and November 1, 9 AM to 4 PM
Level: Beginner, no prerequisites

8. Building Evaluation Capacity of Community Organizations

Are you working with local community groups (coalitions, nonprofits, social service agencies, local health departments, volunteers, school boards) that are trying to evaluate the outcomes of their work to meet a funding requirement, an organizational expectation, or to enhance their own program performance?

In this highly interactive workshop, you will practice and reflect on a variety of activities and adult learning techniques for building basic evaluation skills related to the core components of evaluation: engaging stakeholders, focusing the evaluation, data collection, data analysis and use. Try the activities out, assess their appropriateness for your own situation, and expand your toolbox. Bring your own ‘best practices' to share as we work towards building the evaluation capacity of community practitioners and organizations. This year’s workshop also will include a section on organizational capacity building that goes beyond individual competencies and skills – strategies to build resources and support and the organizational environment for sustaining evaluation.

You will learn:

• Activities to use in building essential evaluation knowledge and skills;
• Methods and techniques that facilitate evaluation learning;
• What to consider in choosing among options to better suit needs, requests and realities;
• Strategies, beyond teaching and training, for building organizational evaluation capacity.

Ellen Taylor-Powell is widely recognized for her work in evaluation capacity building. Her 20 years in Extension have focused continuously on evaluation training and capacity building with concentration on individual, team, and organizational learning.

Session 8: Building Evaluation Capacity
Prerequisites: Involvement in evaluation capacity building at the community level
Scheduled: Monday and Tuesday, October 31 and November 1, 9 AM to 4 PM
Level: Intermediate

Tuesday Workshops, November 1, 9 AM to 4 PM

9. Nontoxic Slideshows: Practical Methods for Improving Evaluation Communication

"Death by Powerpoint" won't literally kill your audience. But it will cause them to check their phone messages, flip through the conference program, and fall asleep. In this workshop, attendees will learn the science behind good slideshows and will leave with direct, pointed changes that can be immediately administered to their own conference presentations. Beyond the scope of the conference, the workshop will address principles of presentation design that support legibility, comprehension, and retention of our evaluation work in the minds of our stakeholders.

Grounded in visual processing theory, the principles will enhance attendees' ability to communicate more effectively with peers, colleagues, and clients through a focus on the proper use of color, placement, and type in presentations. Attendees are strongly encouraged to maximize the value of this highly interactive workshop experience by bringing printouts of slides they intend to present at the conference.

You will learn:

• Visual processing theory and why it is relevant for evaluators;
• Graphic design best practices based in visual processing theory;
• How to apply graphic design best practices to enhance evaluation communication;
• Practical, immediate steps to take to enhance conference presentations.

Stephanie Evergreen's dissertation work examines the role of graphic design in evaluation communication. She is the founding chair of AEAs Data Visualization and Reporting TIG and in that role I has started various initiatives to enhance communication among fellow evaluators. An excellent facilitator, recent feedback from an AEA Coffee Break Webinar left attendees informed, excited, and wanting more!

Session 9: Nontoxic Slideshows
Prerequisites: You should regularly develop slideshows for evaluation work and be comfortably able to navigate slideshow software unassisted.
Scheduled: Tuesday, November 1, 9 AM to 4 PM
Level: Intermediate

10. Hearing Silenced Voices: Techniques to Include Traditionally Disenfranchised Populations in Evaluation

While evaluators understand the importance of multiple stakeholder perspectives, many struggle with how to ensure the participation of those traditionally 'without a voice,' vulnerable or disenfranchised populations. For example, children and youth, persons with disabilities, or those having emerging literacy in majority languages, hold important views regarding the programs, services, and situations which affect them, but their perspectives are not always included.

This workshop will be grounded in theory, will explore ethical considerations, and will be highly participatory and interactive. Stemming from a rights-based approach, the workshop will explore the why and how of including traditionally disenfranchised populations in evaluation. Through their work in Canada and internationally, the facilitators will share a variety of techniques and how these can be coupled with more traditional methods. You will be introduced to the use of rubrics, case studies, visual methods, mapping, and learning walks as tools for eliciting often-silenced voices.

You will learn:

• About different approaches and techniques/methods that are appropriate for evaluations that include marginalized or disenfranchised peoples;
• What to take into account when considering use and application of these techniques;
• About the ethical considerations surrounding the inclusion of often silenced populations in evaluation;
• . . . . from each other's experiences as well as from the facilitators

Linda Lee and Larry Bremner of Proactive Information Services are experienced facilitators and have worked in the field of evaluation for over 30 years. Their practice has including working with those from rural, immigrant, and native communities, in North America as well as in Europe, Asia, and South America, in ways that ensure that evaluation approaches and methods are culturally, socially, and developmentally appropriate.

Session 10: Hearing Silenced Voices
Prerequisites: Basic knowledge of and experience with qualitative methods, and experience working with marginalized or disenfranchised communities or populations
Scheduled: Tuesday, November 1, 9 AM to 4 PM
Level: Intermediate

11: What's New in RealWorld Evaluation?

In graduate school, you might have learned how to do research using randomized control trials. Then you were asked to design an evaluation of a “real” program. But, what if there was no comparative baseline data on the project participants and people from the same population had not been randomly assigned to treatment and control groups? Now, what if the client won’t allocate sufficient time – or money – to conduct what you feel would be a properly rigorous evaluation? How can you conduct adequately valid evaluations under such circumstances?

Welcome to the real world of program evaluation! This workshop will explore the approaches advocated in the new 2nd edition of the popular RealWorld Evaluation book. The authors will share personal examples from their own extensive international experiences and, through participatory processes, attendees will engage with techniques that help evaluators and clients ensure the best quality evaluations possible in spite of RealWorld constraints.

You will learn:

• The seven steps of the RealWorld Evaluation approach;
• How to identify and assess various design options - and select what’s best or feasible under the circumstances;
• How to identify and address threats to the validity or adequacy of evaluation designs;
• How to reconstruct baseline data when needed;
• How to evaluate complex programs where conventional evaluation designs can’t be applied.

Jim Rugh has more than 45 years of experience in international development and for 12 years headed the evaluation department of CARE International. Michael Bamberger spent a decade working with NGOs in Latin America, almost 25 of them working on evaluation with the World Bank. They, along with Linda Mabry, first co-authored RealWorld Evaluation: Working Under Budget, Time, Data and Political Constraints in 2006. The 2nd edition will be released in 2011, just in time for the AEA conference!

Session 11: RealWorld Evaluation
Prerequisites: Experience in conducting evaluations and pre-conference review of background materials
Scheduled: Tuesday, November 1, 9 AM to 4 PM
Level: Intermediate

12. Evaluation 101

This workshop is designed to introduce new evaluators and persons who may be responsible organizationally for evaluation to the processes of thinking about, planning, implementing, and reporting an evaluation across a variety of settings. We'll work together to demystify the jargon, explore evaluation standards and principles that improve practice, and understand what is needed to conduct evaluations in complex settings.

A central focus of the workshop will be the inclusion of issues related to cultural competency and responsiveness in terms of evaluation thinking, purposes, planning processes and use of standards for good evaluations. Through a combination of brief didactic presentations and small group work throughout the workshop, we'll explore a framework for designing the evaluation from problem identification to use.

You will learn:

• Ways to 'think' like a competent, ethical, culturally responsive, evaluator;
• The driving forces for committing to conduct a meaningful evaluation from the perspectives of multiple stakeholder groups;
• A framework for conceptualizing, implementing, reporting and using evaluations from a variety of perspectives, including those that emphasize social justice and human rights;
• How to identify, respect, and incorporate contextual influences into the evaluation design in culturally appropriate ways.

John McLaughlin of the College of William and Mary, and Donna Mertens of Gallaudet University, bring over 40 year of combined experience in the delivery of workshops and seminars focused on all facets of program evaluation. They are noted authors and active evaluators and are currently engaged in the enterprise across a variety of settings.

Session 12: Evaluation 101
Scheduled: Tuesday, November 1, 9 AM to 4 PM
Level: Beginner

13. Getting Started: Introductory Consulting Skills for Evaluators

Are you a program evaluator contemplating venturing out on your own? For many, that may be both an exciting but intimidating prospect. Taught by an independent consultant, his practical workshop will reveal the simple but important skills needed to be successful.

Through lecture, anecdote, discussion, small-group exercises, and independent reflection, this workshop will help participants problem solve – and develop an action plan. The teaching template combines a synthesis of management consulting literature along with evaluation and applied research processes, and entrepreneurial and small business skills. It also provides valuable samples and worksheets along with insider tips, trade secrets, and personal anecdotes to help participants identify and address the unique issues they may face. This course may go hand-in-hand with Staying the Course, which also addresses the challenges faced by independent consultants.

You will learn:

• The personal characteristics and essential values of the independent consultant;
• The pros and cons of setting up shop;
• How to develop a business plan;
• How to prepare proposals;
• How to set fees and track time;
• Important marketing secrets and ways to find work.

Gail Barrington has more than 25 years of practical experience. She founded Barrington Research Group, Inc. in 1985 and has conducted more than 100 program evaluation studies. In 2008, she won the Canadian Evaluation Society award for her Contribution to Evaluation in Canada. A frequent presenter, she will be the author of a new book entitled Consulting Start-up & Management, due out by SAGE in October 2011.

Session 13: Intro Consulting Skills
Scheduled: Tuesday, November 1, 9 AM to 4 PM
Level: Beginner

14: Tools and Techniques of Feminist and Gender Responsive Evaluation

While most development and many U.S.-based agencies are committed to gender equality, gender continues to be one of the areas inadequately addressed in many evaluations. And, conventional research tools are often not well suited for understanding how different social groups are affected by program interventions. This workshop will help evaluators recognize the impact of the lens through which evaluative decisions are made and the role that governmental, organizational, community and personal values play at every stage of the process.

Tools and techniques for developing gender sensitive conceptual frameworks, evaluation design, sampling, data collection, analysis, dissemination and use of evaluations will be presented. Case studies using different gender sensitive approaches in both a U.S. and international context will be shared and critiqued and participants will apply the techniques through hands-on activities.

You will learn:

• The similarities between feminist evaluation and gender analysis, human rights and equity focused evaluations;
• The distinct differences between feminist evaluations and evaluations that focus on gender analysis, human rights and/or equity;
• The importance of incorporating gender responsive approaches into program evaluation;
• The operational and policy consequences of ignoring gender;
• The tools and techniques for the design, implementation and use of gender responsive evaluation;
• The techniques/methods characteristic of gender-responsive evaluation and a feminist evaluation.

Kathryn Bowen is Director of Program Evaluation at Centerstone Research Institute and has conducted feminist evaluations of community based prevention and treatment programs for 11 years. Michael Bamberger is an independent consultant who worked for 9 years as the Senior Sociologist in the World Bank Gender and Development Department. Tessie Catsambas is President of EnCompass LLC and brings 25 years experience in planning, evaluation and management of international programs and activities. Belen Sanz Luque is an avid trainer and facilitator with UN Women.

Session 14: Feminist/Gender Responsive Evaluation
Scheduled: Tuesday, November 1, 9 AM to 4 PM
Level: Beginner

15: Systems Thinking for Evaluation Practice

Systems thinking can help evaluators understand the world – in all its diversity – in ways that are practical, comprehensive, and wise. For those interested in making sense of the complex and sometimes messy situations we often encounter in practice, this workshop provides an overview of system thinking and how it can be used in evaluation. A systems approach is particularly useful in situations where rigorous rethinking, reframing, and unpacking complex realities and assumptions are required. Using systems approaches provides a broader perspective and helps the evaluator see the interconnectedness of component parts in a coordinated manner that emphasizes balance and fit. Evaluations based on systems concepts generate rich descriptions of complex, interconnected situations based on multiple perspectives that help participants build deeper meanings and understanding that can inform choices for subsequent action.

Through mini-lectures, group activities, and hands-on practice, this workshop teaches fundamental concepts of systems thinking and provides opportunities to apply learnings to everyday practice – making sense of the world, using systems to understand things better, and orienting ourselves towards the world in a way that embraces complexity and ambiguity.

You will learn:

• Basic concepts underlying systems thinking;
• How to use systems perspectives in evaluation practice and evaluative thinking;
• How to apply systems thinking to evaluation practice.

Janice Noga, an independent consultant with Pathfinder Evaluation and Consulting, has taught graduate level courses in statistics, research methods, human learning, and classroom assessment and evaluation. Margaret Hargreaves, a senior health researcher at Mathematica Policy Research, has program evaluation methods at the graduate level and is currently serving as program co-chair for the Systems in Evaluation Topical Interest Group.

Session 15: Systems Thinking
Prerequisites: Experience designing and implementing evaluation project but may be new to systems thinking
Scheduled: Tuesday, November 1, 9 AM to 4 PM
Level: Intermediate

16. Propensity Score Matching: Theories and Applications

When experimental designs are infeasible, evaluation researchers often use observational data to estimate treatment effects. Propensity score matching, used to improve covariate balance, has been gaining popularity. This workshop will introduce you to basic theories and principles, demonstrate the process, and provide the tools necessary to perform the work. We’ll demonstrate using the free statistical software program R and provide attendees with a CD containing both the R software and example data.

The workshop will include a review of experimental and non-experimental designs, an overview of commonly used matching methods, and a step-by-step demonstration as well as team activities.

You will learn:

• The theoretical basis of propensity score matching and related models;
• How to import, export, and process data files between Excel and R;
• How to preprocess data;
• Hot to estimate the program effect;
• How to evaluate potential bias.

Ning Rui from Research for Better Schools and Haiyan Bai from the University of Central Florida have written in multiple venues on the application of propensity score matching. New to workshop facilitation at AEA, they’ll bring with them their experience from facilitation at their home university and the Eastern Evaluation Research Society.

Session 16: Propensity Score Matching
Scheduled: Tuesday, November 1, 9 AM to 4 PM
Level: Beginner

Wednesday Workshops, November 2, 8 AM to 3 PM

17: Logic Models - Beyond the Traditional View: Metrics, Methods, Expected and Unexpected Change

When should we use (or not use) logic models? What kind of information can we put in logic models? What is the value of different forms and scales of models for the same program? What different uses do logic models have across a program's life cycle? What kinds of relationships can we represent? What are the relationships between logic models, metrics, and methodology? How can we manage multiple uses of logic models -- evaluation, planning advocacy; explanation versus prediction? How can we peg the detail in a model to our actual state of knowledge about a program? How does graphic design relate to information richness, and why does it matter? What choices do evaluators have for working with stakeholders to develop and use logic models?

Evaluators need to know how to respond to these questions. Through lecture and discussion, this workshop will endeavor to provide answers.

You will learn:

• The essential nature of a 'model', its strengths and weaknesses;
• When and how to use logic models across the entire evaluation life cycle;
• The value of using multiple forms and scales of a model for the same evaluation;
• Relationships among logic models, measurement, methodology, and various types of theory;
• Principles of good graphic design for logic models - balancing readability and information richness;
• Techniques for working with stakeholders to develop a logic model.

Jonathan Morrell of Fulcrum Corporation has produced and delivered a wide range of training workshops to professional associations, government, and private sector settings. Morell employs logic models throughout his evaluation practice and consulting work. The issue of program logic, and how to represent that logic, figures prominently in his recent book Evaluation in the Face of Uncertainty.

Session 17: Logic Models +
Prerequisites: Experience constructing logic models
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Intermediate

18: Longitudinal Data Analysis: From the Classics to Modern Structural Equation Modeling

Many evaluation studies make use of longitudinal data. However, while much can be learned from repeated measures, the analysis of change is also associated with a number of special problems. The workshop takes up these issues and reviews how traditional methods in the analysis of change, such as the paired t-test, repeated measures ANOVA or MANOVA, address these problems.

A mixture of PowerPoint presentation, group discussion, and exercises with a special focus on model specification will help us to explore LGM in contrast to more traditional approaches to analyzing change. The core of the workshop will be an introduction to SEM-based latent growth curve modeling (LGM). We will show how to specify, estimate and interpret growth curve models. In contrast to most traditional methods, which are restricted to the analysis of mean changes, LGM allows the investigation of unit specific (individual) changes over time. Towards the end of the workshop, we will discuss more recent advancements of LGM, including multiple group analyses, the inclusion of time-varying covariates, and cohort sequential designs. We'll give detailed directions for model specification, and all analyses will be illustrated by practical examples.

You will learn:

• How to detect reliable sources of variance in the analysis of change;
• Special problems associated with the analysis of longitudinal data;
• How to specify, estimate and interpret latent growth curve models (LGM) using SEM;
• The relationship of traditional methods to new methods;
• Recent developments in latent growth curve modeling;
• More advanced models such as multiple group analysis, the inclusion of time-varying covariates, or cohort sequential designs.

Manuel C Voelkle is a research scientist at the Max Planck Institute in Berlin, Germany. He teaches courses on advanced multivariate data analysis and research design and research methods. Werner W Wittmann is professor of psychology at the University of Mannheim, where he heads a research and teaching unit specializing in research methods, assessment and evaluation research.

Session 18: Longitudinal Data Analysis
Prerequisites: Familiarity with structural equation models (SEM) and regression analytic techniques.
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Intermediate

19. Introduction to GIS (Geographic Information Systems) and Spatial Analysis in Evaluation

This workshop introduces Geographic Information Systems (GIS) and spatial analysis concepts and uses in environmental/ecological and community health/public health program and policy evaluation. It defines and covers steps for undertaking vector and raster mapping projects plus challenges of doing GIS cost effectively. It will provide information about pros and cons of a variety of mapping software (some free) and how to obtain base maps and data for map contents.

Using case study examples from environmental/ecological and community health/public health content areas, we will demonstrate and you will practice designing a GIS project, setting up spatial analysis, and using GIS approaches to evaluate programs or policy initiatives. The workshop will discuss ways to involve evaluation stakeholders (e.g., staff, program clients) in mapping projects. We’ll use case studies, demonstration, small group exercises, and traditional presentation to introduce attendees to the value and use of GIS for evaluation; however, please note that this is not taught in a computer lab and there will be no hands-on work with GIS software.

You will learn:

• To understand GIS as an important tool for evaluators;
• To distinguish two general types of computer maps (vector and raster) and determine which is most appropriate to use in an evaluation;
• To identify kinds of data to map, and the spatial analysis needed to use GIS in an evaluation;
• To decide among GIS software options;
• How to involve clients in "participatory GIS.

Both Arlene Hopkins and Stephen Maack are experienced university teachers and regularly present to adults in professional venues, in conference sessions and as workshop facilitators. Stephen Maack has a Ph.D. in anthropology with a research specialty in social change. Ms. Hopkins has an MA in education, and is also a former K-12 teacher. Both use GIS in their work as practicing evaluators.

Session 19: Introduction to GIS
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Beginner

20: Transformative Mixed Methods Evaluations

What are the implications of the transformative paradigm for the use of mixed methods directly focused on the furtherance of social justice? How can you apply the methodological implications of the transformative paradigm in the design of an evaluation? What approaches are useful for an evaluator to undertake a transformative mixed methods evaluation in diverse contexts? How can transformative mixed methods be applied to increase the probability of social justice goals being achieved? What sampling and data collection strategies are appropriate? What does it mean to address the myth of homogeneity?

We'll explore the answers to these and other questions, and the implications of positioning oneself as an advocate for social justice and human rights in an evaluation context. We'll demonstrate, discuss, and share examples of evaluations that use transformative mixed methods that focus on dimensions of diversity such as race/ethnicity, religion, language, gender, indigenous status, and disability. The philosophical assumptions of the transformative paradigm will be used to derive methodological implications that can be applied to the design of evaluations that seek to further social justice in marginalized communities.

You will learn:

• To critically examine the transformative paradigm's assumptions in culturally diverse communities;
• To identify different methodological approaches within a transformative mixed methods model;
• To apply critical skills associated with selecting the design and use of transformative mixed methods evaluation.

Donna Mertens is a Past President of the American Evaluation Association who teaches evaluation methods and program evaluation to deaf and hearing graduate students at Gallaudet University in Washington, D.C. Mertens recently authored Transformative Research and Evaluation (Guilford). Katrina L Bledsoe is an evaluator with Education Development Center, Inc., conducting and managing evaluations in culturally complex communities nationally.

Session 20: Transformative Mixed Methods
Prerequisites: Knowledge of basic evaluation practice
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Intermediate

21. Applications of Multiple Regression for Evaluators: Mediation, Moderation, and More

Multiple regression is a powerful and flexible tool that has wide applications in evaluation and applied research. Regression analyses are used to describe relationships, test theories, make predictions with data from experimental or observational studies, and model linear or nonlinear relationships.

Issues we'll explore include selecting models that are appropriate to your data and research questions, preparing data for analysis, running analyses, interpreting results, and presenting findings to a nontechnical audience. The facilitator will demonstrate applications from start to finish with live SPSS and Excel, and then you will tackle multiple real-world case examples in small groups. Detailed handouts include explanations and examples that can be used at home to guide similar applications.

You will learn:

• How to prepare data for regression analysis;
• How to identify where regression analysis would be useful;
• Procedures for conducting computer analysis, including SPSS code;
• How to conduct mediation and moderation analyses;
• How to interpret SPSS REGRESSION output;
• How to present regression findings to a lay audience.

Dale Berger of Claremont Graduate University is a lauded teacher of workshops and classes in statistical methods. Recipient of the outstanding teaching award from the Western Psychological Association, he is also the author of "Using Regression Analysis" in The Handbook of Practical Program Evaluation.

Session 21: Multiple Regression
Prerequisites: Basic inferential and descriptive statistics, including correlation and regression; familiarity with SPSS
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Intermediate

22: How to Prepare an Evaluation Dissertation Proposal

Developing an acceptable dissertation proposal often seems more difficult than conducting the actual research. Further, proposing an evaluation as a dissertation study can raise faculty concerns of acceptability and feasibility. This workshop will lead you through a step-by-step process for preparing a strong, effective dissertation proposal with special emphasis on the evaluation dissertation.

The workshop will cover such topics as the nature, structure, and multiple functions of the dissertation proposal; how to construct a compelling argument; how to develop an effective problem statement and methods section; and how to provide the necessary assurances to get the proposal approved. Practical procedures and review criteria will be provided for each step. The workshop will emphasize application of the knowledge and skills taught to the participants’ personal dissertation situation through the use of an annotated case example, multiple self-assessment worksheets, and several opportunities for questions of personal application.

You will learn:

• The pros and cons of using an evaluation study as dissertation research;
• How to construct a compelling argument in a dissertation proposal;
• The basic process and review criteria for constructing an effective problem statement and methods section;
• How to provide the assurances necessary to guarantee approval of the proposal;
• How to apply all of the above to your personal dissertation needs.

Nick L Smith is the co-author of How to Prepare a Dissertation Proposal (Syracuse University Press) and a past-president of AEA. He has taught research and evaluation courses for over 20 years at Syracuse University and is an experienced workshop presenter. He has served as a dissertation advisor to multiple students and is the primary architect of the curriculum and dissertation requirements in his department

Session 22: Dissertation Proposal
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Beginner

23. Utilization-Focused Evaluation

Evaluations should be useful, practical, accurate and ethical. Utilization-Focused Evaluation is a process that meets these expectations and promotes use of evaluation from beginning to end. With a focus on carefully targeting and implementing evaluations for increased utility, this approach encourages situational responsiveness, adaptability and creativity. This training is aimed at building capacity to think strategically about evaluation and increase commitment to conducting high quality and useful evaluations.

Utilization-Focused evaluation focuses on the intended users of the evaluation in the context of situational responsiveness with the goal of methodological appropriateness. An appropriate match between users and methods should result in an evaluation that is useful, practical, accurate, and ethical, the characteristics of high quality evaluations according to the profession's standards. With an overall goal of teaching you the process of Utilization-Focused Evaluation, the session will combine lectures with concrete examples and interactive case analyses.

You will learn:

• Basic premises and principles of Utilization-Focused Evaluation (U-FE);
• Practical steps and strategies for implementing U-FE;
• Strengths and weaknesses of U-FE, and situations for which it is appropriate.

Michael Quinn Patton is an independent consultant and professor at the Union Institute. An internationally known expert on Utilization-focused Evaluation, this workshop is based on the newly completed fourth edition of his best-selling evaluation text, Utilization Focused Evaluation: The New Century Text (SAGE).

Session 23: Utilization-Focused Evaluation
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Beginner

24: Reframing Evaluation Through Appreciative Inquiry

In this one-day workshop, participants will be introduced to Appreciative Inquiry and will explore ways in which it may be applied in their own evaluation work.

You will experience various phases of Appreciative Inquiry by using appreciative interviews to focus an evaluation, to structure and conduct interviews, and to develop indicators. In skills building sessions, you will experience the application of "reframing" in a practical and grounded way. Through real-world case examples, practice case studies, exercises, discussion and short lectures, participants will identify how to incorporate AI into their evaluation contexts.

You will learn:

• To appreciate the added value of using Appreciative Inquiry in evaluation;
• To understand the principles and applications of appreciative inquiry to evaluation;
•  To formulate evaluation goals and questions through appreciative inquiry processes;
•  To conduct interviews and analyze interview data using an appreciative inquiry approach;
• To understand how to develop indicators to measure program impact through provocative propositions/possibility statements;
• To identify appropriate applications of an appreciative inquiry approach and methods in evaluation work.

Tessie Tzavaras Catsambas is President of EnCompass LLC and brings 25 years experience in planning, evaluation and management of international programs and activities. Hallie Preskill is the Executive Director of FSG's Strategic Learning & Evaluation Center, and works on a variety of evaluation and learning projects that span multiple sectors and program areas. Catsambas and Preskill are co-authors of the recent book Reframing Evaluation Through Appreciative Inquiry.

Session 24: Appreciative Inquiry
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Beginner

25: Designing and Conducting Case Study Evaluation
This session is now full. There is no waiting list for the session. Please choose an alternative.

Case studies are inherently complex, investigating a case in depth and from many angles to meet the interests of a range of stakeholders. This workshop will explore how to design and conduct rigorous case study evaluation that offers unique insights into multifaceted policies and programs. Specific issues to be covered include how to establish validity and credibility in case study, ethics and politics in the field, how to generalize from the case; theorize within the case; and how to get beyond the perception of case study as storytelling, important though this is, to offer critique of programs and policies.

The workshop will be conducted in four sessions with an interactive pedagogy, including short presentations, critique, dialogue, and group activities. It will start by examining the purposes, types, and justification for case study evaluation and then focus on how to plan and design case studies that produce rigorous, valid knowledge. In the afternoon the first session will be on analysis, interpretation and generalization; the second on reporting case studies in diverse ways for different audiences to maximize their use. Ethical issues will be discussed throughout. The course aims to provide you with a strong justification for the methodology of case study evaluation and the knowledge and skills to conduct case studies that will promote in-depth understanding of complex programs and inform future policy-making.

You will learn:

• How to design and conduct case study evaluation that is rigorous and credible;
• How to manage difficult political and ethical issues in the field;
• Different modes of analyzing and interpreting case study data;
• How to report case study evaluation in ways that are useful to policymakers and practitioners.

Hailing from the University of Southampton, United Kingdom, Helen Simons has been teaching and conducting workshops on case study evaluation for over thirty years in higher education and policy contexts as well as conducting case study evaluations in the field. She has conducted case study evaluation and/or directed evaluation training in over 20 countries and is widely published in the field of evaluation and on the topic of case study methods, including in 2009 Case Study Research in Practice from SAGE.

Session 25: Case Study Evaluation
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Beginner

26: Theory-Driven Evaluation for Assessing and Improving Program Planning, Implementation, and Effectiveness

Designed for evaluators who already have basic training and experience in program evaluation, this workshop will expand your knowledge on the theory-driven evaluation approach for strengthening your practical skills. We'll explore the conceptual framework of program theory and its related evaluation taxonomy, which will facilitate effective communication between evaluators and stakeholders regarding stakeholders' evaluation needs and evaluation options to address those needs.

The workshop starts from an introduction of an evaluation taxonomy that encompasses the full program cycle, including program planning, initial implementation, mature implementation, and outcomes. We'll then focus on how program theory and theory-driven evaluation are useful in the assessment and improvement of a program at each of these stages. The workshop also covers recent developments on the integrative validity model and bottom-up approach for enhancing usefulness of evaluation.

You will learn:

• How to apply the conceptual framework of program theory to design different types of theory-driven evaluations;
• How to apply the theory-driven approach to select an evaluation that is best suited to a particular program stage and to meet stakeholders' evaluation needs;
• How to apply the theory-driven approach to provide credible evidence on viability, effectuality, and transferability.

Huey Chen is a Senior Evaluation Scientist at the Centers for Disease Control and Prevention (CDC). Previously he was a Professor at the University of Alabama at Birmingham. He has taught workshops, as well as undergraduate and graduate evaluation courses in universities. His 1990 book, Theory-Driven Evaluations, is considered the classic text for understanding program theory and theory-driven evaluation. His 2005 book, Practical Program Evaluation: Assessing and Improving Planning, implementation, and Effectiveness, provides a major expansion of the scope and usefulness of theory-driven evaluations.

Session 26: Theory-Driven Evaluation
Prerequisites: Basic background in logic modeling or program theory
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Intermediate

Session 27: Participatory Program Implementation
Prerequisites: Community or stakeholder facilitation experience; prior experience working with or participating in community groups
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Intermediate

28: An Introduction to Quasi-Experimental Methods

Quasi-experimental designs provide creative researchers a host of options sometimes unavailable in Randomized Controlled Trials (RCTs). The incapacity to assign participants randomly to study conditions is an admitted inconvenience, but not a death knell. Many designs are available to the quasi-experimentalist that provide evidence with causal implications are so compelling that they are practically undeniable. Quasi-experiments can be performed in contexts in which experiments cannot, owing to practical, political, or other constraints.

In this workshop, we will explore a variety of designs, ranging from simple and complex interrupted time series studies to nonrandomized pretest/posttest/control group designs to regression discontinuity approaches. Real research problems that do not lend themselves to the RCT approach will be discussed, and participants will be invited to solve them while learning about the possibilities available in the class of designs described as quasi-experimental.

You will learn:

• How to address important social issues that do not admit to random assignment of participants to conditions;
• How quasi-experimental designs, properly applied, may be used to assess the outcome of social interventions;
• How and when quasi-experimental designs may be preferred over experimental methods;
• A variety of designs that may be usefully employed in a host of difficult research environments, to answer a range of questions of important, socially relevant questions.

William Crano first encountered quasi-experimental methods under the instruction of Donald Campbell. He has taught quasi-experimental methods regularly since coming to Claremont Graduate University 13 years ago, stressing the practical application of scientific knowledge. Quasi-experimental methods have been an important part of his research for many years, and he is keen to show others how these techniques can materially improve our science.

Session 28: Quasi-experimental Methods
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Beginner

29: Using Effect Size and Association Measures in Evaluation

Improve your capacity to understand and apply a range of measures including: standardized measures of effect sizes from Cohen, Glass, and Hedges; Eta-squared; Omega-squared; the Intraclass correlation coefficient; and Cramer’s V. Answer the call to report effect size and association measures as part of your evaluation results. Together we will explore how to select the best measures, how to perform the needed calculations, and how to analyze, interpret, and report on the output in ways that strengthen your overall evaluation.

Through mini-lecture, hands-on exercises, and computer-based demonstration, you will improve your understanding of the theoretical foundation and computational procedures for each measure as well as ways to identify and correct for bias.

You will learn:

• How to select, compute, and interpret the appropriate measure of effect size or association;
• Considerations in the use of confidence intervals;
• SAS and SPSS macros to compute common effect size and association measures;
• Basic relationships among the measures.

Jack Barnette is Professor of Biostatistics at the University of Colorado School of Public Health. He has taught courses in statistical methods, program evaluation, and survey methodology for more than 30 years. He has been conducting research and writing on this topic for more than ten years. Jack is a regular facilitator both at AEA's annual conference and the CDC/AEA Summer Evaluation Institute. He was awarded the Outstanding Commitment to Teaching Award by the University of Alabama and is a member of the ASPH/Pfizer Academy of Distinguished Public Health Teachers.

Session 29: Effect Size and Association Measures
Prerequisites: Univariate statistics through ANOVA & understanding and use of confidence levels
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Advanced

30: Multilevel Models in Program and Policy Evaluation

Multilevel models open the door to understanding the inter-relationships among nested structures and the ways evaluands change across time. This workshop will demystify multilevel models and present them at an accessible level, stressing their practical applications in evaluation.

Through discussion and hands-on demonstrations, the workshop will address four key questions: When are multilevel models necessary? How can they be implemented using standard software? How does one interpret multilevel results? What are recent developments in this arena?

You will learn:

• The basics of multilevel modeling;
• When to use multilevel models in evaluation practice;
• How to implement models using widely available software;
• The importance of considering multilevel structures in understanding program theory.

Sanjeev Sridharan of the University of Toronto has repeatedly taught multilevel models for AEA as well as for the SPSS software company. His recent work on this topic has been published in the Journal of Substance Abuse Treatment, Proceedings of the American Statistical Association and Social Indicators Research. Known for making the complex understandable, his approach to the topic is straightforward and accessible.

Session 30: Multilevel Models
Prerequisites: Basic statistics
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Intermediate

31: Conflict Resolution Skills for Evaluators

Unacknowledged and unresolved conflict can undermine even the most skilled evaluators, if they are unprepared to address the underlying issues and are untrained in conflict-resolution skills. In this workshop, participants will learn the practical applications of conflict resolution theory as they apply to conflict situations in program evaluation through a hands-on, experiential approach using real-life examples from program evaluation. Eleven of the Essential Competencies for Program Evaluators (Stevahn, King, Ghere, & Minnema, 2004), falling within the Situational Analysis, Project Management, Reflective Practice, and Interpersonal Competence skills areas, stress the need for evaluators to have strong conflict-resolution and communication skills.

The workshop employs skill practice and role-plays, and you will receive feedback to enhance your skill development. This training will cover: problem-solving skills; personal conflict styles; communication, listening, and facilitation skills, and will introduce you to a streamlined process of conflict resolution that may be used with clients and stakeholders.

You will learn:

• To understand and diagnose conflict in program evaluation;
• To examine positive outcomes possible as a result of the use of effective conflict-resolution skills;
• To incorporate the five styles of conflict-resolution as part of reflective practice in program evaluation;
• To problem solve effectively, including win-win guidelines, clarifying, summarizing, and reframing;
• To understand individual differences in resolving conflict and how this might affect your ability to resolve conflict with others from diverse backgrounds and experience;
• Techniques for handling anger and high emotion that may result from conflict.

A professional trainer for more than a decade, Jeanne Zimmer has presented variations on the Basic Communication and Conflict Resolution/Mediation Skills Training to hundreds of individuals, most recently at the Minnesota Evaluation Studies Institute. Conflict-resolution trainings are customized for each group. For the past ten years Jeanne has provided basic mediation skills training for volunteer mediators, in compliance with the MN Supreme Court.

Session 31: Conflict Resolution
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Beginner

32: Qualitative Research Strategies for Deeper Understanding

In an ideal world, qualitative research provides rich insights and deep understanding of respondents' perceptions, attitudes, behaviors and emotions. However, it is often difficult for research respondents to remember and report their experiences and to access their emotions.

This practical workshop offers cutting edge approaches to conducting qualitative interviews and focus groups that have proven their effectiveness across a wide range of topics. These strategies engage research participants and help them articulate more thoroughly, thus leading to deep insights for the researcher. Through demonstration and hands-on participation, we will examine techniques that help respondents to reconstruct their memories - such as visualization, mind-mapping, diaries and storytelling; articulate their emotions, through metaphorical techniques such as analogies, collage and photo-sorts; and explore different perspectives through "word bubbles" and debate.

You will learn:

• How to conduct a variety of activities that lead to richer qualitative data;
• Which activities best fit with what kinds of research objectives;
•  How to think creatively about crafting moderating guides.

Deborah Potts is a qualitative researcher who has led thousands of focus groups and one-on-one interviews. She is co-author of Moderator to the Max: A full-tilt guide to creative, insightful focus groups and depth interviews. Deborah is the senior qualitative researcher at the research company, InsightsNow, Inc. and has taught workshops in conducting qualitative research for nearly two decades.

Session 32: Qualitative Research Strategies
Prerequisites: Basic qualitative interviewing skills
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Intermediate

33: The Rest of the Iceberg: The Logic of Evaluation

It would be a mistake to think that teaching statistics can be done by only covering descriptive statistics; one must teach inferential statistics as well, since it is the part that bears most strongly on the most common and important practical questions. We have made this mistake in evaluation, by teaching only the tools that provide the factual i.e., non-evaluative component of our results. That by itself cannot provide answers to evaluative questions such as, Does this program teach reading or basic mathematics well, compared to other options?, or What is the best way to treat police training of violent offenders?

Through lecture and discussion, this workshop provides a systematic way to identify and validate and incorporate the relevant values that fill the gap, and explores some of the problems with and consequence of doing this.

You will learn:

• Why the present texts and courses and competency bases for teaching evaluation are crucially incomplete;
• Why this makes evaluations based on those foundations vulnerable to serious attack for invalidity, and for a failure to address extremely important practical evaluation problems;
• Why it is extremely difficult to provide the missing material that fills the methodological and foundational gap;
• How that difficulty provides the best reason for the doctrine of value-free science; and why the usual reasons for that position do not provide serious support for it and can be easily repudiated;
• How one can and must fill the gap by showing how to: (i) identify the values needed to support valid evaluative conclusions: (ii) validate those values; (iii) combine those values with the empirical data usually gathered in the course of conventional evaluations.

Michael Scriven has served as the President of AEA and has taught evaluation in schools of education, departments of philosophy and psychology, and to professional groups, for 45 years. A senior statesman in the field, he has authored over 90 publications focusing on evaluation, and received AEA's Paul F Lazarsfeld Evaluation Theory award in 1986.

Session 33: Logic of Evaluation
Prerequisites: Some knowledge of the usual social science methodologies that are used in evaluation.
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Intermediate

34: Survey Design 101

Building on proven strategies that work in real-world contexts, this workshop will help you plan and execute all aspects of the survey design process. Designed for true beginners with little or no background in survey development, you will be introduced to the fundamentals of survey design and administration, and leave with tools needed to develop and improve your own surveys. Perfect for those who need a refresher on survey design or are new to the field of evaluation.

This interactive workshop will use a combination of direct instruction with hands-on opportunities for participants to apply what is learned to their own evaluation projects. We'll explore different types of surveys, the advantages and challenges associated with various administration methods, as well as how to choose the right one, and problems to avoid. You will receive also handouts with sample surveys, item writing tips, checklists, and resource lists for further information.

You will learn:

• The various types and formats of surveys;
• Procedures for high quality survey design;
• Strategies for increasing response rates;
• Tips for items writing and survey formatting.

Courtney L. Malloy is on faculty at the Rossier School of Education at the University of Southern California, and is a consultant at Vital Research, a research and evaluation firm operating for more than 25 years and specializing in survey design. She have developed and administered numerous surveys and has extensive experience facilitating workshops and training sessions on research and evaluation for diverse audiences.

Session 34: Survey Design
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Beginner

35: Using Concept Mapping for Evaluation in Organizations and Communities: A Participatory Mixed Methods Approach

Through applications of structured group concept mapping, this workshop will introduce key principles of community-centered participation in evaluation, describing and illustrating major decisions and steps in engaging stakeholders. Concept mapping is a well-tested, mixed methods methodology that integrates familiar qualitative group processes with multivariate statistical analyses to help a group describe and organize its thinking on a topic. Ideas are represented visually in a series of easy-to-read graphics that capture specific ideas generated by a group; relationships between ideas; how ideas cluster together; and how those ideas are valued.

The steps in the methodology will be illustrated with project examples. There will be a particular focus on the planning stages of a project, as the decisions at this stage are applicable to any participatory project. A secondary focus will be on the unique analyses that create a shared conceptual framework for complex, systems-based issues and represent that in easy-to-read visuals.

You will learn:

• The place of community or organizational engagement in developing program evaluation structures, and providing evaluation feedback;
• To recognize appropriate applications of the concept mapping methodology;
•  The steps in the concept mapping methodology and how those can be adapted to various situations;
• How the concept mapping analysis converts qualitative input into quantitative data that is useful in evaluation projects;
• To describe a structured group concept mapping methodology as one of a variety of tools for engaging communities of interest, noting similarities and differences to related methodologies;
• To apply the methodology to your own projects.

Mary Kane, Scott Rosas, and Katy Hall hail from Concept Systems, Inc, a consulting company that uses the concept mapping methodology as a primary tool in its planning and evaluation consulting projects. The presenters have extensive experience with concept mapping and are among the world's leading experts on this approach.

Session 35: Concept Mapping
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Beginner

36: Dominators, Cynics, and Wallflowers: Practical Strategies for Moderating Meaningful Focus Groups

Have you ever watched a recording of yourself moderating a focus group and said: "I should have cut him off earlier?" Have you ever had a client say: "We can't trust focus groups because one or two people dominate and contaminate the rest of the discussion?" If so, this workshop is for you.

In this engaging session, Kahle describes ten problem types of behavior commonly encountered in focus groups. He offers field-tested, practical strategies and specific techniques for preventing, managing and leveraging difficult behavior. A sometimes humorous and certainly insightful view of the impact of problem behavior on focus group research is presented by sharing real-life stories from the trenches. Come ready to describe the "participants from hell" that you have encountered and share your tips for handling difficult situations. This is a high energy, fun and interactive workshop with strategies and tactics you can start using immediately.

You will learn:

• Ten common types of problem behavior in focus groups;
• Practical strategies for managing and leveraging problem behavior in focus groups;
• Methods of identifying and preventing problem behavior before it occurs;
• Approaches to effective management of group dynamics to achieve client/evaluation objectives.

Robert Kahle is the author of Dominators, Cynics, and Wallflowers: Practical Strategies for Moderating Meaningful Focus Groups (Paramount Market Publishing, 2007). Kahle has been working on the issue of managing difficult behavior in focus groups and other small group settings since the mid 1990s.

Session 36: Focus Group Moderation
Prerequisites: Have moderated at least a couple of focus groups; understand fundamentals of qualitative research
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Intermediate

37: A Framework for Developing High Quality Performance Measurement Systems of Evaluation

Program funders are increasingly emphasizing the importance of evaluation, often through performance measurement. This workshop explores how to develop high quality project objectives and performance measures which are critical to both good proposals and successful evaluations.

Through group discussion, mini-lectures, and practice, you will increase your understanding of the relationships between project activities and intended program outcomes through the development of logic models. This will assist in the development of more sound evaluation designs, thus allowing for the collection of higher-quality and more meaningful data. Participants will be provided with a framework with practical strategies and planning devices to use when writing project objectives and measures, and planning evaluations focused on performance measurement. This framework is useful for a wide array of programs assessing impact and evaluating program outcomes from both single-site and multi-site studies as well as for locally and federally-funded projects.

You will learn:

•  To create a logic model that feeds into the evaluation;
• To identify and create measurable, feasible, and relevant project objectives related to evaluation;
• To identify and write high quality performance measures that are complete, measureable, practical and pertinent to the corresponding objective;
• To understand the difference between process and outcome measures and to ensure there is a balance of the two;
• To see how objectives and performance measures can easily fit in to the performance reports required by federal, local, and for profit funders;
•  A framework that can be applied to a variety of evaluation settings which allows for clear, measurable, and agreed upon plan.

Mindy Hightower King and Courtney Brown are faculty members at Indiana University where they are Senior Associates directing and managing evaluations for local and state, and federal agencies, foundations, and non-profit organizations. Drawing on their extensive experience developing and implementing performance measurement systems, they have provided over 20 workshops and lectures to program staff and grantees, and individual technical assistance to at least 40 representatives of grant-receiving agencies

Session 37: Performance Measurement Systems
Prerequisites: Basic evaluation skills
Scheduled: Wednesday, November 2, 8 AM to 3 PM
Level: Intermediate

Wednesday Morning Workshops, November 2, 8 AM to 11 AM

38: An Executive Summary is Not Enough: Effective Reporting Techniques for Evaluators

As an evaluator you are conscientious about conducting the best evaluation possible, but how much thought do you give to communicating your results effectively? Do you consider your job complete after submitting a lengthy final report? Reporting is an important skill for evaluators who care about seeing their results disseminated widely and recommendations actually implemented.

Drawing on current research, this interactive workshop will present an overview of three key principles of effective reporting and engage participants in a discussion of its role in effective evaluation. Participants will leave with an expanded repertoire of innovative reporting techniques and will have the opportunity to work on a real example in groups.

You will learn:

• The role of communication and reporting in good evaluation practice;
• Three key principles for communicating results effectively;
• Four innovative reporting techniques.

Kylie Hutchinson has served since 2005 as the trainer for the Canadian Evaluation Society's Essential Skills Series (ESS) in British Columbia. Her interest in dissemination and communications stems from twenty years of experience in the field of evaluation.

Session 38: Effective Reporting
Scheduled: Wednesday, November 2, 8 AM to 11 AM
Level: Beginner

39: How to Improve Data Quality of Mixed Method Evaluation Designs

40: Basics of Sampling for Evaluators

Evaluators are frequently called upon to analyze population data. But, performing such tasks requires a basic understanding of the principles of sampling. This workshop targets evaluators who do not have significant experience in quantitative analysis but do have an interest in better understanding this fundamental technique. Beginning with clarifying the scientific language and operational lingo of the field, participants will review the basic approaches and identify the benefits and pitfalls of each. Using illustrations from domestic and international projects to explore the relationship between program implementation and sampling methodology and the role that the evaluator plays in facilitating practical data collection and analysis, attendees will discuss how the conclusions drawn can vary as a consequence of different sampling protocols.

From introduction and demonstration to interactive breakout exercises, participants will apply sampling on Real World examples and translate from “statistics” into English. Useful resources and an annotated bibliography will be distributed.

You will learn:

• About different types of sampling mechanisms and when they should be applied;
• How field sampling and program implementation interact;
• How to apply selected sampling techniques into practical examples.

Michael Cohen holds a doctorate in Developmental Biology and initially used the techniques described in this workshop to explain stochastic elements in the visual processing system. By translating this knowledge to the evaluation of international development programs, he has had the opportunity to design complex sampling paradigms around the world.

Session 40: Basics of Sampling
Scheduled: Wednesday, November 2, 8 AM to 11 AM
Level: Beginner

41: Scoring Performance Assessments

Evaluators increasingly use performance assessments in the review of program outcomes. The reliability of scores from these performance assessments, meanwhile, has been improved by training raters to more effectively use scoring guides or rubrics. This workshop provides an overview of scoring methods that improve the reliability and accuracy associated with performance assessments.

This workshop will highlight how to develop scoring materials, how to train raters and how to conduct a scoring session while sharing research across disciplines and field to describe scoring methods that participants can apply to improve their own assessments. Examples of scoring guides, anchors, and monitoring reports will be shared along with new developments in the field.

You will learn:

• The elements of scoring guides;
• The strengths and limitations of various scoring guides;
• Methods for selecting;
• The logistics of conducting a scoring session;
• How to monitor quality control during scoring.

Robert Johnson, a professor at the University of South Carolina, leads workshops at conferences of the Association of Test Publishers and the American Educational Research Association and is author of Assessing Performance: Developing, Scoring, and Validating Performance Tasks published by Guilford Publications. Min Zhu, also with USC, is a co-presenter.

Session 41: Scoring Performance Assessments
Scheduled: Wednesday, November 2, 8 AM to 11 AM
Level: Beginner

42: Staying Power: Intermediate Consulting Skills for Evaluators

This session is now full. There is no waiting list for the session. Please choose an alternative.

Evaluators and applied researchers may be well versed in academic research processes, but uncertain about how to handle the entrepreneurial and small business management skills needed to be successful independent consultants. For those aspiring to have a thriving practice, this workshop addresses common problems encountered in the day-to-day running of a small consulting business and identifies some of the key issues faced by consultants

Through discussion, lecture, anecdote, independent reflection and teamwork, this workshop will help participants problem solve around the ongoing challenges of a consulting practice. For those looking to fine-tune their current professional practice, a seasoned practitioner/award winner/author will share from her more than two decades of experience. This course may go hand-in-hand with Getting Started, geared toward those contemplating the prospect of going into independent consulting.

You will learn:

• Ways to avoid burnout and foster personal growth;
• How to manage multiple projects;
• Ways to re-tool and reinvigorate marketing initiatives;
• How to hire and manage staff;
• How to expand the practice;
• How to deal with the contract blues.

Gail Barrington founded Barrington Research Group, Inc. in 1985 and has conducted more than 100 program evaluation studies. She is a Certified Management Consultant and a Credentialed Evaluator. In 2008, she won the Canadian Evaluation Society award for her Contribution to Evaluation in Canada. A frequent presenter, she is author of a new book entitled Consulting Start-up & Management: A Guide for Evaluators & Applied Researchers, due out by SAGE in October 2011.

Session 42: Intermediate Consulting Skills
Scheduled: Wednesday, November 2, 8 AM to 11 AM
Level: Intermediate

Wednesday Afternoon Workshops, November 2, 12 PM to 3 PM

43: Empowerment Evaluation

Empowerment Evaluation builds program capacity and fosters program improvement. It teaches people to help themselves by learning how to evaluate their own programs. The basic steps of empowerment evaluation include: 1) establishing a mission or unifying purpose for a group or program; 2) taking stock - creating a baseline to measure future growth and improvement; and 3) planning for the future - establishing goals and strategies to achieve goals, as well as credible evidence to monitor change. The role of the evaluator is that of coach or facilitator in an empowerment evaluation, since the group is in charge of the evaluation itself.

Employing lecture, activities, demonstration and case examples, the workshop will introduce you to the steps of empowerment evaluation and tools to facilitate the approach.

You will learn:

• The basic steps to planning and conducting empowerment evaluations;
• The 10 guiding principles of empowerment evaluation;
• The key concepts guiding the approach (focusing on accountability);
• How to prioritizing program activities;
• How to help organizations plan for the future;
• How to select appropriate technological tools to facilitate an empowerment evaluation.

David Fetterman hails from Stanford University and is the editor of (and a contributor to) the recently published Empowerment Evaluation Principles in Practice (Guilford). He Chairs the Collaborative, Participatory and Empowerment Evaluation AEA Topical Interest Group and is a highly experienced and sought after facilitator.

Session 43: Empowerment Evaluation
Scheduled: Wednesday, November 2, 12 PM to 3 PM
Level: Beginner

Session 45: Cost-Effectiveness Analysis of HHS
Prerequisites: Basic program evaluation skills
Scheduled: Wednesday, November 2, 12 PM to 3 PM
Level: Intermediate

Session 46: Waawiyeyaa (Circular) Evaluation Tool
Prerequisites: Need to know basics of evaluation practice
Scheduled: Wednesday, November 2, 12 PM to 3 PM
Level: Intermediate

Session 47: Desktop Software
Prerequisites: Some knowledge of Word (especially Tables) and familiarity with Access would be helpful
Scheduled: Wednesday, November 2, 12 PM to 3 PM
Level: Intermediate

Christina Christie has taught professional development workshops for many organization, including the Claremont Graduate Professional Development Workshop Series, the AEA/CDC Summer Institute and the Canadian Evaluation Society (annual meeting workshops). She has published both peer-reviewed and invited papers on the topic of evaluation theory.

Session 48: Theories and Frameworks of Practice
Scheduled: Wednesday, November 2, 12 PM to 3 PM
Level: Beginner

Sunday Morning Workshops, November 6, 9 AM to 12 PM