Two Day Workshops, Monday and Tuesday, November 5 and 6, 9 AM to 4 PM

1. Qualitative Methods

Qualitative data can humanize evaluations by portraying people and stories behind the numbers. Qualitative inquiry involves using in-depth interviews, focus groups, observational methods, and case studies to provide rich descriptions of processes, people, and programs. When combined with participatory and collaborative approaches, qualitative methods are especially appropriate for capacity-building-oriented evaluations.

Through lecture, discussion, and small-group practice, this workshop will help you to choose among qualitative methods and implement those methods in ways that are credible, useful, and rigorous. It will culminate with a discussion of new directions in qualitative evaluation.

You will learn:

• Types of evaluation questions for which qualitative inquiry is appropriate,

• Purposeful sampling strategies,

• Interviewing, case study, and observation methods,

• Analytical approaches that support useful evaluation.

Michael Quinn Patton is an independent consultant and professor at the Union Institute. An internationally known expert on utilization-focused evaluation and qualitative methods, he published the third edition of Qualitative Research and Evaluation Methods (SAGE) in 2001.

Session 1: Qualitative Methods
Scheduled: Monday and Tuesday, November 5 and 6, 9 AM to 4 PM
Level: Beginner, no prerequisites

2. Quantitative Methods

Quantitative data offers opportunities for numerical descriptions of populations and samples. The challenge is in knowing which analyses are best for a given situation. Designed for the practitioner needing a refresher course and/or guidance in applying quantitative methods to evaluation contexts, the workshop covers the basics of parametric and nonparametric statistics, as well as how to report your findings.

Hands-on exercises and computer demonstrations interspersed with mini-lectures will introduce methods and concepts. The instructor will review examples of research and evaluation questions and the statistical methods appropriate to developing a quantitative data-based response.

You will learn:

• The conceptual basis for a variety of statistical procedures,

• How more sophisticated procedures are based on the statistical basics,

• Which analyses are most applicable for a given data set or evaluation question,

• How to interpret and report findings from these analyses.

Katherine McKnight applies quantitative analysis as Director of Program Evaluation for Pearson Achievement Solutions. Additionally, she teaches Research Methods, Statistics, and Measurement in Public and International Affairs at George Mason University in Fairfax, VA.

Session 2: Quantitative Methods
Scheduled: Monday and Tuesday, November 5 and 6, 9 AM to 4 PM
Level: Beginner, no prerequisites

3. Evaluation 101: Intro to Evaluation Practice

This workshop is full. There is no waiting list available for this workshop. Please choose another.

Begin at the beginning and learn the basics of evaluation from an expert trainer. The session will focus on the logic of evaluation to answer the key question: "What resources are transformed into what program evaluation strategies to produce what outputs for which evaluation audiences, to serve what purposes." Enhance your skills in planning, conducting, monitoring, and modifying the evaluation so that it generates the information needed to improve program results and communicate program performance to key stakeholder groups.

A case-driven instructional process, using discussion, exercises, and lecture will introduce the steps in conducting useful evaluations: Getting started, Describing the program, Identifying evaluation questions, Collecting data, Analyzing and reporting, and Using results.

You will learn:

• The basic steps to an evaluation and important drivers of program assessment,

• Evaluation terminology,

• Contextual influences on evaluation and ways to respond,

• Logic modeling as a tool to describe a program and develop evaluation questions and foci,

• Methods for analyzing, and using evaluation information.

John McLaughlin has been part of the evaluation community for over 30 years working in the public, private, and non-profit sectors. He has presented this workshop in multiple venues and will tailor this two-day format for Evaluation 2007.

Session 3: Evaluation 101
Scheduled: Monday and Tuesday, November 5 and 6, 9 AM to 4 PM
Level: Beginner, no prerequisites

4. Logic Models for Program Evaluation and Planning

First, we will recapture the utility of program logic modeling as a simple discipline, using cases in public health and human services to explore the steps for constructing, refining and validating models. Then, we'll examine how to improve logic models using some fundamental principles of "program theory", and, finally, demonstrate how to use logic models effectively to help frame questions in evaluation, performance measurement, and strategic planning. Both days use modules with presentations, small group case studies, and debriefs to reinforce group work.

You will learn:

• To construct logic models,

• To use program theory principles to improve a logic model,

• To develop an evaluation focus based on a logic model,

• To use logic models to answer strategic planning questions and select and develop performance measures.

Thomas Chapel is the central resource person for planning and program evaluation at the Centers for Disease Control and Prevention and a sought after trainer. Tom has taught this workshop for the past four years to much acclaim.

Session 4: Logic Models
Scheduled: Monday and Tuesday, November 5 and 6, 9 AM to 4 PM
Level: Beginner, no prerequisites

5. Participatory Evaluation

Participatory evaluation practice requires evaluators to be skilled facilitators of interpersonal interactions. This workshop will provide you with theoretical grounding (social interdependence theory, conflict theory, and evaluation use theory) and practical frameworks for analyzing and extending your own practice.

Through presentations, discussion, reflection, and case study, you will experience strategies to enhance participatory evaluation and foster interaction. You are encouraged to bring examples of challenges faced in your practice for discussion to this workshop consistently lauded for its ready applicability to real world evaluation contexts.

You will learn:

• Strategies to foster effective interaction, including belief sheets; values voting; three-step interview; cooperative rank order; graffiti; jigsaw; and data dialogue,

• Responses to challenges in participatory evaluation practices,

• Four frameworks for reflective evaluation practice.

Jean King has over 30 years of experience as an award-winning teacher at the University of Minnesota. As an evaluation practitioner, she has received AEA’s Myrdal award for outstanding evaluation practice. Laurie Stevahn is a professor at Seattle University with extensive facilitation experience as well as applied experience in participatory evaluation.

Session 5: Participatory Evaluation
Prerequisites: Basic evaluation skills
Scheduled: Monday and Tuesday, November 5 and 6, 9 AM to 4 PM
Level: Intermediate

What do you do when asked to perform an evaluation on a program that is well underway? When time and resources are few, yet expectations high? When questions about baseline data and control groups are met with blank stares? When time and resources are few, yet clients expect “rigorous impact evaluation”? When there are political pressures to address?

The RealWorld Evaluation approach will be introduced and its practical utility assessed through presentations and discussion, and through examples drawn from the experiences of presenters and participants. This well-developed seven-step approach seeks to ensure the best quality evaluation under real-life constraints.

You will learn:

• The seven steps of the RealWorld Evaluation approach,

• Context-responsive evaluation design alternatives,

• Ways to reconstruct baseline data,

• How to identify, and overcome threats to the validity or adequacy of evaluation methods.

Jim Rugh and Michael Bamberger recently co-authored, with Linda Mabry, the book Real World Evaluation, Working Under Time, Data and Political Constraints (SAGE 2006). The two presenters bring over eighty years of professional evaluation experience, mostly in developing countries around the world.

Session 10: RealWorld Evaluation
Prerequisites: Academic or practical knowledge of the basics of evaluation.
Scheduled: Tuesday, November 6, 9:00 AM to 4:00 PM
Level: Intermediate

12. Growing Your Evaluation Business from Surviving to Thriving

Interested in growing your evaluation business? This workshop brings together business, service, and marketing concepts from recent publications such as Jim Collin's Good to Great, Robert Schwartz and John Mayne's Quality Matters, and the recent New Directions for Evaluation issue on Independent Evaluation Consulting, and applies them to small evaluation consulting firms.

The workshop begins with a self-assessment where attendees rate their evaluation businesses in terms of being a sustainable asset. You will then look into the future and dream about what you would like your business to become in terms of sales, profitability and sustainability. The remainder of the workshop focuses on ways to get from where the business is today to where you would like your business to be in the future. The workshop will be highly interactive and use numerous real-life situations for analysis and recommendations for ways to proactively and deliberately grow.

• To move your evaluation businesses forward in terms of sales, profitability and sustainability,

• To build a sustainable plan for marketing your evaluation services,

• Methods for assuring the highest quality of evaluation services to clients,

• Ways to structure your company and services so that your business becomes a saleable asset.

Melanie Hwalek is the founder and owner of SPEC Associates, a program evaluation and research company that has thrived over the past 27 years. She is co-author of the 2006 New Directions for Evaluation article "Building Your Evaluation Business into a Valuable Asset.”  Victoria Essenmacher is a partner and business manager of SPEC Associates and has provided extensive consulting to non-profit organizations on issues of high-quality performance measurement systems.

Session 12: Growing your Eval Business
Prerequisites: Experience conducting evaluations as a small business owner or self-employed contractor.
Scheduled: Tuesday, November 6, 9:00 AM to 4:00 PM
Level: Intermediate

13. Best Practices in Quantitative Methods: Attending to the Little Things Makes a Big Difference

Learn the latest advances in data management, statistical testing, and outcome measurement! Best practices are common in every field. Program evaluators and, in particular, quantitatively-oriented evaluators, ought to have the same benefit of keeping abreast of these "best" practices as their professional counterparts in other fields.

Through short lectures, didactic inquiry, and demonstrations, the session will explore data-handling including coding and transforming variables and computing new variables, working with missing data, statistical testing including statistical power and effect size estimation, and quantitatively capturing outcomes in program and policy implementation.

You will learn:

• Best practices for handling and managing data including coding and transformation of variables,

• Best practices for statistical testing including estimating statistical power and effect size,

• Best practices for capturing outcomes including designing useful measures for relevant outcomes.

Patrick McKnight is a professor of psychology at George Mason University where he teaches statistics and methods courses, and is the co-chair of AEA's Quantitative Methods Topical Interest Group (TIG). An experienced facilitator, his engaging style renders the complex accessible and well worth the time and investment.

Session 13: Best Practices in Quant
Prerequisites: Working knowledge of a statistical package and a sound understanding of univariate, bivariate (correlations and t-tests), and multivariate (GLM, ANOVA, multiple regression) statistical procedures.
Scheduled: Tuesday, November 6, 9:00 AM to 4:00 PM
Level: Intermediate

16. Evaluation Strategies for Communicating and Reporting

Communicating evaluation processes and results is one of the most critical aspects of evaluation practice. Yet, evaluators continually experience frustration with hours spent on writing reports that are seldom read or shared. While final reports will continue to be an expectation of many evaluation contracts, there are other ways in which evaluators can communicate and report on the progress and findings from an evaluation.

Using hands-on demonstrations and real-world examples, we will explore how a variety of strategies for communicating and reporting can increase learning from the evaluation’s findings, stakeholders’ understanding of evaluation processes, the evaluation’s credibility, and action on the evaluation’s recommendations.

You will learn:

• Reasons for communicating and reporting throughout an evaluation’s life cycle,

• How stakeholders’ information needs influence your choice of communicating approaches,

• More than 15 strategies for communicating and reporting evaluation processes and findings.

Rosalie T Torres is president of Torres Consulting Group, a research, evaluation and management consulting firm specializing in the feedback-based development of programs and organizations since 1992. She has authored/co-authored numerous books and articles including,  Evaluation Strategies for Communicating and Reporting (Torres, Preskill, & Piontek, 2005), and Evaluative Inquiry for Learning in Organizations (Preskill & Torres, 1999). 

Session 16: Communicating and Reporting
Scheduled: Tuesday, November 6, 9:00 AM to 4:00 PM
Level: Beginner, no prerequisites

One Day Workshops, Wednesday, November 7, 8 AM to 3 PM

19. Using Effect Size and Association Measures

Answer the call to report effect size and association measures as part of your evaluation results. Improve your capacity to understand and apply a range of measures including: standardized measures of effect sizes from Cohen, Glass, and Hedges; Eta-squared; Omega-squared; the Intraclass correlation coefficient; and Cramer’s V.

Through mini-lecture, hands-on exercises, and demonstration, you will improve your understanding of the theoretical foundation and computational procedures for each measure as well as ways to identify and correct for bias.

You will learn:

• How to select, compute, and interpret the appropriate measure of effect size or association,

• Considerations in the use of confidence intervals,

• SAS and SPSS macros to compute common effect size and association measures,

• Basic relationships among the measures.

Jack Barnette hails from The University of Alabama at Birmingham. He has been conducting research and writing on this topic for the past ten years. Jack has won awards for outstanding teaching and is a regular facilitator both at AEA's annual conference and the CDC/AEA Summer Evaluation Institute.

Session 19: Effect Size Measures
Prerequisites: Univariate statistics through ANOVA and understanding of and use of confidence levels.
Scheduled: Wednesday, November 7, 8:00 am to 3:00 pm
Level: Advanced

20. Evaluating Advocacy and Policy Change Efforts

This workshop is full. There is no waiting list available for this workshop. Please choose another.

Evaluations of advocacy, community organizing, and other policy change efforts present unique challenges for evaluators, particularly those looking to use evaluation for ongoing learning. On the ground, change can take years to happen, and when it does, it may occur in fits and starts. Outside forces can affect efforts in unforeseen ways, causing advocates' strategies to shift, goals to be abandoned, and new goals to be taken up. And current policy losses can belie gains that spell future success. Evaluators need strategies for addressing these challenges in evaluation design and implementation.

Through lecture, discussion, demonstration, and hands-on activities, this workshop will walk participants through a variety of strategies for evaluating advocacy and policy change efforts.  We will draw from specific case studies that address real-world challenges and discuss ways to overcome them. 

You will learn:

• Frameworks and guidelines for conducting advocacy evaluations,

• Ways to create nimble and flexible evaluations that allow for real-time improvement,

• Practical tools that can assist with evaluation efforts,

• Techniques to identify outcomes that can be used as milestones for success.

Justin Louie, a consultant with Blueprint Research & Design, Inc., works with nonprofits and foundations to help them evaluate their advocacy efforts, and has conducted leading research on this topic for The California Endowment. Ehren Reed, a Senior Associate with Innovation Network, Inc., leads a number of evaluations of policy change initiatives and conducts field-building research for national foundations.

Session 20: Evaluating Advocacy/Policy
Scheduled: Wednesday, November 7, 8:00 am to 3:00 pm
Prerequisites: Basic Evaluation Skills
Level: Intermediate

21. Utilization-focused Evaluation

This workshop is full. There is no waiting list available for this workshop. Please choose another.

Evaluations should be useful, practical, accurate and ethical. Utilization-focused Evaluation is a process that meets these expectations and promotes use of evaluation from beginning to end. With a focus on carefully targeting and implementing evaluations for increased utility, this approach encourages situational responsiveness, adaptability and creativity.

With an overall goal of teaching you the process of Utilization-focused Evaluation, the session will combine lectures with concrete examples and interactive case analyses, including cases provided by the participants.

You will learn:

• Basic premises and principles of Utilization-focused Evaluation (U-FE),

• Practical steps and strategies for implementing U-FE,

• Strengths and weaknesses of U-FE, and situations for which it is appropriate.

Michael Quinn Patton is an independent consultant and professor at the Union Institute. An internationally known expert on Utilization-focused Evaluation, this workshop is based on the newly completed fourth edition of his best-selling evaluation text, Utilization Focused Evaluation: The New Century Text (SAGE). 

Session 21: Utilization-focused
Scheduled: Wednesday, November 7, 8:00 am to 3:00 pm
Level: Beginner, no prerequisites

22. Evaluating Program Implementation: Concepts, Methods, and Applications

25. Performance Measurement in the Public and Nonprofit Sectors

This workshop is full. There is no waiting list available for this workshop. Please choose another.

Managing for Results! Performance-Based Budgeting! Balanced Score Cards! Dash Boards! Program managers and executives in the public and non profit sectors are being pushed to embrace these approaches to assessing how well their  programs and agencies are doing. The unifying thread linking all of these efforts is performance measurement. So what is needed to measure performance in an effective and useful manner?

This workshop will provide you with instruction, materials and exercises to increase your understanding of what constitutes performance measurement and how to measure program performance in the public and nonprofit sectors.

You will learn:

• How to identify pressures and opportunities for measuring performance,

• The political challenges to measuring performance and how to respond to them,

• Ways to assess the reliability and validity of performance measures,

• How to identify performance measures for social services,

• Approaches to getting performance measures used.

Kathryn Newcomer is the Director of the Phd in Public Policy and Administration program at the George Washington University where she teaches public and nonprofit program evaluation, research design, and applied statistics. She conducts research and training for federal and local government agencies on performance measurement and program evaluation, and has published five books and numerous articles about performance measurement in the government and nonprofit sectors.

Session 25: Performance Measurement
Scheduled: Wednesday, November 7, 8:00 AM to 3:00 PM
Level: Beginner, no prerequisites

26. How to Prepare an Evaluation Dissertation Proposal

Developing an acceptable dissertation proposal often seems more difficult than conducting the actual research. Further, proposing an evaluation as a dissertation study can raise faculty concerns of acceptability and feasibility. This workshop will lead you through a step-by-step process for preparing a strong, effective dissertation proposal with special emphasis on the evaluation dissertation.

The workshop will cover such topics as the nature, structure, and multiple functions of the dissertation proposal; how to construct a compelling argument; how to develop an effective problem statement and methods section; and how to provide the necessary assurances to get the proposal approved. Practical procedures and review criteria will be provided for each step. The workshop will emphasize application of the knowledge and skills taught to the participants’ personal dissertation situation through the use of an annotated case example, multiple self-assessment worksheets, and several opportunities for questions of personal application.

You will learn:

• The pros and cons of using an evaluation study as dissertation research,

• How to construct a compelling argument in a dissertation proposal,

• The basic process and review criteria for constructing an effective problem statement and methods section,

• How to provide the assurances necessary to guarantee approval of the proposal.

Nick L Smith is the co-author of How to Prepare a Dissertation Proposal from Syracuse University Press and a past-president of AEA. He has taught research and evaluation courses for over 20 years at Syracuse University and is an experienced workshop presenter through NOVA University's doctoral program in evaluation.

Session 26: Evaluation Dissertation
Scheduled: Wednesday, November 7, 8:00 AM to 3:00 PM
Level: Beginner, no prerequisites

27. Theory-Driven Evaluation for Assessing and
Improving Program Planning, Implementation, and Effectiveness

Learn the theory-driven approach for assessing and improving program planning, implementation and effectiveness. You will explore the conceptual framework of program theory and its structure, which facilitates precise communication between evaluators and stakeholders regarding evaluation needs and approaches to addressing those needs. From there, the workshop moves to how program theory and theory-driven evaluation are useful in the assessment and improvement of a program at each stage throughout its life-cycle.

Mini-lectures, group exercises and case studies will illustrate the use of program theory and theory-driven evaluation for program planning, initial implementation, mature implementation and outcomes. In the outcome stages, you will explore the differences among outcome monitoring, efficacy evaluation and effectiveness evaluation.  

You will learn:

• How to apply the conceptual framework of program theory and theory-driven evaluations,

• How to conduct theory-driven process and outcome evaluations,

• How to conduct integrative process/outcome evaluations,

• How to apply program theory to improve program planning processes.

Huey Chen, professor at the University of Alabama at Birmingham, is the author of Theory-Driven Evaluations (SAGE), the classic text for understanding program theory and theory-driven evaluation and most recently of Practical Program Evaluation (2005). He is an internationally know workshop facilitator on the subject.

Session 27: Theory-Driven Evaluation
Prerequisites: Basic background in evaluation.
Scheduled: