Two Day Workshops, Monday and Tuesday, November 5 and 6, 9 AM to 4 PM

1. Qualitative Methods

Qualitative data can humanize evaluations by portraying people and stories behind the numbers. Qualitative inquiry involves using in-depth interviews, focus groups, observational methods, and case studies to provide rich descriptions of processes, people, and programs. When combined with participatory and collaborative approaches, qualitative methods are especially appropriate for capacity-building-oriented evaluations.

Through lecture, discussion, and small-group practice, this workshop will help you to choose among qualitative methods and implement those methods in ways that are credible, useful, and rigorous. It will culminate with a discussion of new directions in qualitative evaluation.

You will learn:

• Types of evaluation questions for which qualitative inquiry is appropriate,

• Purposeful sampling strategies,

• Interviewing, case study, and observation methods,

• Analytical approaches that support useful evaluation.

Michael Quinn Patton is an independent consultant and professor at the Union Institute. An internationally known expert on utilization-focused evaluation and qualitative methods, he published the third edition of Qualitative Research and Evaluation Methods (SAGE) in 2001.

Session 1: Qualitative Methods
Scheduled: Monday and Tuesday, November 5 and 6, 9 AM to 4 PM
Level: Beginner, no prerequisites

2. Quantitative Methods

Quantitative data offers opportunities for numerical descriptions of populations and samples. The challenge is in knowing which analyses are best for a given situation. Designed for the practitioner needing a refresher course and/or guidance in applying quantitative methods to evaluation contexts, the workshop covers the basics of parametric and nonparametric statistics, as well as how to report your findings.

Hands-on exercises and computer demonstrations interspersed with mini-lectures will introduce methods and concepts. The instructor will review examples of research and evaluation questions and the statistical methods appropriate to developing a quantitative data-based response.

You will learn:

• The conceptual basis for a variety of statistical procedures,

• How more sophisticated procedures are based on the statistical basics,

• Which analyses are most applicable for a given data set or evaluation question,

• How to interpret and report findings from these analyses.

Katherine McKnight applies quantitative analysis as Director of Program Evaluation for Pearson Achievement Solutions. Additionally, she teaches Research Methods, Statistics, and Measurement in Public and International Affairs at George Mason University in Fairfax, VA.

Session 2: Quantitative Methods
Scheduled: Monday and Tuesday, November 5 and 6, 9 AM to 4 PM
Level: Beginner, no prerequisites

3. Evaluation 101: Intro to Evaluation Practice

This workshop is full. There is no waiting list available for this workshop. Please choose another.

Begin at the beginning and learn the basics of evaluation from an expert trainer. The session will focus on the logic of evaluation to answer the key question: "What resources are transformed into what program evaluation strategies to produce what outputs for which evaluation audiences, to serve what purposes." Enhance your skills in planning, conducting, monitoring, and modifying the evaluation so that it generates the information needed to improve program results and communicate program performance to key stakeholder groups.

A case-driven instructional process, using discussion, exercises, and lecture will introduce the steps in conducting useful evaluations: Getting started, Describing the program, Identifying evaluation questions, Collecting data, Analyzing and reporting, and Using results.

You will learn:

• The basic steps to an evaluation and important drivers of program assessment,

• Evaluation terminology,

• Contextual influences on evaluation and ways to respond,

• Logic modeling as a tool to describe a program and develop evaluation questions and foci,

• Methods for analyzing, and using evaluation information.

John McLaughlin has been part of the evaluation community for over 30 years working in the public, private, and non-profit sectors. He has presented this workshop in multiple venues and will tailor this two-day format for Evaluation 2007.

Session 3: Evaluation 101
Scheduled: Monday and Tuesday, November 5 and 6, 9 AM to 4 PM
Level: Beginner, no prerequisites

4. Logic Models for Program Evaluation and Planning

First, we will recapture the utility of program logic modeling as a simple discipline, using cases in public health and human services to explore the steps for constructing, refining and validating models. Then, we'll examine how to improve logic models using some fundamental principles of "program theory", and, finally, demonstrate how to use logic models effectively to help frame questions in evaluation, performance measurement, and strategic planning. Both days use modules with presentations, small group case studies, and debriefs to reinforce group work.

You will learn:

• To construct logic models,

• To use program theory principles to improve a logic model,

• To develop an evaluation focus based on a logic model,

• To use logic models to answer strategic planning questions and select and develop performance measures.

Thomas Chapel is the central resource person for planning and program evaluation at the Centers for Disease Control and Prevention and a sought after trainer. Tom has taught this workshop for the past four years to much acclaim.

Session 4: Logic Models
Scheduled: Monday and Tuesday, November 5 and 6, 9 AM to 4 PM
Level: Beginner, no prerequisites

5. Participatory Evaluation

Participatory evaluation practice requires evaluators to be skilled facilitators of interpersonal interactions. This workshop will provide you with theoretical grounding (social interdependence theory, conflict theory, and evaluation use theory) and practical frameworks for analyzing and extending your own practice.

Through presentations, discussion, reflection, and case study, you will experience strategies to enhance participatory evaluation and foster interaction. You are encouraged to bring examples of challenges faced in your practice for discussion to this workshop consistently lauded for its ready applicability to real world evaluation contexts.

You will learn:

• Strategies to foster effective interaction, including belief sheets; values voting; three-step interview; cooperative rank order; graffiti; jigsaw; and data dialogue,

• Responses to challenges in participatory evaluation practices,

• Four frameworks for reflective evaluation practice.

Jean King has over 30 years of experience as an award-winning teacher at the University of Minnesota. As an evaluation practitioner, she has received AEA’s Myrdal award for outstanding evaluation practice. Laurie Stevahn is a professor at Seattle University with extensive facilitation experience as well as applied experience in participatory evaluation.

Session 5: Participatory Evaluation
Prerequisites: Basic evaluation skills
Scheduled: Monday and Tuesday, November 5 and 6, 9 AM to 4 PM
Level: Intermediate

What do you do when asked to perform an evaluation on a program that is well underway? When time and resources are few, yet expectations high? When questions about baseline data and control groups are met with blank stares? When time and resources are few, yet clients expect “rigorous impact evaluation”? When there are political pressures to address?

The RealWorld Evaluation approach will be introduced and its practical utility assessed through presentations and discussion, and through examples drawn from the experiences of presenters and participants. This well-developed seven-step approach seeks to ensure the best quality evaluation under real-life constraints.

You will learn:

• The seven steps of the RealWorld Evaluation approach,

• Context-responsive evaluation design alternatives,

• Ways to reconstruct baseline data,

• How to identify, and overcome threats to the validity or adequacy of evaluation methods.

Jim Rugh and Michael Bamberger recently co-authored, with Linda Mabry, the book Real World Evaluation, Working Under Time, Data and Political Constraints (SAGE 2006). The two presenters bring over eighty years of professional evaluation experience, mostly in developing countries around the world.

Session 10: RealWorld Evaluation
Prerequisites: Academic or practical knowledge of the basics of evaluation.
Scheduled: Tuesday, November 6, 9:00 AM to 4:00 PM
Level: Intermediate

12. Growing Your Evaluation Business from Surviving to Thriving

Interested in growing your evaluation business? This workshop brings together business, service, and marketing concepts from recent publications such as Jim Collin's Good to Great, Robert Schwartz and John Mayne's Quality Matters, and the recent New Directions for Evaluation issue on Independent Evaluation Consulting, and applies them to small evaluation consulting firms.

The workshop begins with a self-assessment where attendees rate their evaluation businesses in terms of being a sustainable asset. You will then look into the future and dream about what you would like your business to become in terms of sales, profitability and sustainability. The remainder of the workshop focuses on ways to get from where the business is today to where you would like your business to be in the future. The workshop will be highly interactive and use numerous real-life situations for analysis and recommendations for ways to proactively and deliberately grow.

• To move your evaluation businesses forward in terms of sales, profitability and sustainability,

• To build a sustainable plan for marketing your evaluation services,

• Methods for assuring the highest quality of evaluation services to clients,

• Ways to structure your company and services so that your business becomes a saleable asset.

Melanie Hwalek is the founder and owner of SPEC Associates, a program evaluation and research company that has thrived over the past 27 years. She is co-author of the 2006 New Directions for Evaluation article "Building Your Evaluation Business into a Valuable Asset.”  Victoria Essenmacher is a partner and business manager of SPEC Associates and has provided extensive consulting to non-profit organizations on issues of high-quality performance measurement systems.

Session 12: Growing your Eval Business
Prerequisites: Experience conducting evaluations as a small business owner or self-employed contractor.
Scheduled: Tuesday, November 6, 9:00 AM to 4:00 PM
Level: Intermediate

13. Best Practices in Quantitative Methods: Attending to the Little Things Makes a Big Difference

Learn the latest advances in data management, statistical testing, and outcome measurement! Best practices are common in every field. Program evaluators and, in particular, quantitatively-oriented evaluators, ought to have the same benefit of keeping abreast of these "best" practices as their professional counterparts in other fields.

Through short lectures, didactic inquiry, and demonstrations, the session will explore data-handling including coding and transforming variables and computing new variables, working with missing data, statistical testing including statistical power and effect size estimation, and quantitatively capturing outcomes in program and policy implementation.

You will learn:

• Best practices for handling and managing data including coding and transformation of variables,

• Best practices for statistical testing including estimating statistical power and effect size,

• Best practices for capturing outcomes including designing useful measures for relevant outcomes.

Patrick McKnight is a professor of psychology at George Mason University where he teaches statistics and methods courses, and is the co-chair of AEA's Quantitative Methods Topical Interest Group (TIG). An experienced facilitator, his engaging style renders the complex accessible and well worth the time and investment.

Session 13: Best Practices in Quant
Prerequisites: Working knowledge of a statistical package and a sound understanding of univariate, bivariate (correlations and t-tests), and multivariate (GLM, ANOVA, multiple regression) statistical procedures.
Scheduled: Tuesday, November 6, 9:00 AM to 4:00 PM
Level: Intermediate

16. Evaluation Strategies for Communicating and Reporting

Communicating evaluation processes and results is one of the most critical aspects of evaluation practice. Yet, evaluators continually experience frustration with hours spent on writing reports that are seldom read or shared. While final reports will continue to be an expectation of many evaluation contracts, there are other ways in which evaluators can communicate and report on the progress and findings from an evaluation.

Using hands-on demonstrations and real-world examples, we will explore how a variety of strategies for communicating and reporting can increase learning from the evaluation’s findings, stakeholders’ understanding of evaluation processes, the evaluation’s credibility, and action on the evaluation’s recommendations.

You will learn:

• Reasons for communicating and reporting throughout an evaluation’s life cycle,

• How stakeholders’ information needs influence your choice of communicating approaches,

• More than 15 strategies for communicating and reporting evaluation processes and findings.

Rosalie T Torres is president of Torres Consulting Group, a research, evaluation and management consulting firm specializing in the feedback-based development of programs and organizations since 1992. She has authored/co-authored numerous books and articles including,  Evaluation Strategies for Communicating and Reporting (Torres, Preskill, & Piontek, 2005), and Evaluative Inquiry for Learning in Organizations (Preskill & Torres, 1999). 

Session 16: Communicating and Reporting
Scheduled: Tuesday, November 6, 9:00 AM to 4:00 PM
Level: Beginner, no prerequisites

One Day Workshops, Wednesday, November 7, 8 AM to 3 PM

19. Using Effect Size and Association Measures

Answer the call to report effect size and association measures as part of your evaluation results. Improve your capacity to understand and apply a range of measures including: standardized measures of effect sizes from Cohen, Glass, and Hedges; Eta-squared; Omega-squared; the Intraclass correlation coefficient; and Cramer’s V.

Through mini-lecture, hands-on exercises, and demonstration, you will improve your understanding of the theoretical foundation and computational procedures for each measure as well as ways to identify and correct for bias.

You will learn:

• How to select, compute, and interpret the appropriate measure of effect size or association,

• Considerations in the use of confidence intervals,

• SAS and SPSS macros to compute common effect size and association measures,

• Basic relationships among the measures.

Jack Barnette hails from The University of Alabama at Birmingham. He has been conducting research and writing on this topic for the past ten years. Jack has won awards for outstanding teaching and is a regular facilitator both at AEA's annual conference and the CDC/AEA Summer Evaluation Institute.

Session 19: Effect Size Measures
Prerequisites: Univariate statistics through ANOVA and understanding of and use of confidence levels.
Scheduled: Wednesday, November 7, 8:00 am to 3:00 pm
Level: Advanced

20. Evaluating Advocacy and Policy Change Efforts

This workshop is full. There is no waiting list available for this workshop. Please choose another.

Evaluations of advocacy, community organizing, and other policy change efforts present unique challenges for evaluators, particularly those looking to use evaluation for ongoing learning. On the ground, change can take years to happen, and when it does, it may occur in fits and starts. Outside forces can affect efforts in unforeseen ways, causing advocates' strategies to shift, goals to be abandoned, and new goals to be taken up. And current policy losses can belie gains that spell future success. Evaluators need strategies for addressing these challenges in evaluation design and implementation.

Through lecture, discussion, demonstration, and hands-on activities, this workshop will walk participants through a variety of strategies for evaluating advocacy and policy change efforts.  We will draw from specific case studies that address real-world challenges and discuss ways to overcome them. 

You will learn:

• Frameworks and guidelines for conducting advocacy evaluations,

• Ways to create nimble and flexible evaluations that allow for real-time improvement,

• Practical tools that can assist with evaluation efforts,

• Techniques to identify outcomes that can be used as milestones for success.

Justin Louie, a consultant with Blueprint Research & Design, Inc., works with nonprofits and foundations to help them evaluate their advocacy efforts, and has conducted leading research on this topic for The California Endowment. Ehren Reed, a Senior Associate with Innovation Network, Inc., leads a number of evaluations of policy change initiatives and conducts field-building research for national foundations.

Session 20: Evaluating Advocacy/Policy
Scheduled: Wednesday, November 7, 8:00 am to 3:00 pm
Prerequisites: Basic Evaluation Skills
Level: Intermediate

21. Utilization-focused Evaluation

This workshop is full. There is no waiting list available for this workshop. Please choose another.

Evaluations should be useful, practical, accurate and ethical. Utilization-focused Evaluation is a process that meets these expectations and promotes use of evaluation from beginning to end. With a focus on carefully targeting and implementing evaluations for increased utility, this approach encourages situational responsiveness, adaptability and creativity.

With an overall goal of teaching you the process of Utilization-focused Evaluation, the session will combine lectures with concrete examples and interactive case analyses, including cases provided by the participants.

You will learn:

• Basic premises and principles of Utilization-focused Evaluation (U-FE),

• Practical steps and strategies for implementing U-FE,

• Strengths and weaknesses of U-FE, and situations for which it is appropriate.

Michael Quinn Patton is an independent consultant and professor at the Union Institute. An internationally known expert on Utilization-focused Evaluation, this workshop is based on the newly completed fourth edition of his best-selling evaluation text, Utilization Focused Evaluation: The New Century Text (SAGE). 

Session 21: Utilization-focused
Scheduled: Wednesday, November 7, 8:00 am to 3:00 pm
Level: Beginner, no prerequisites

22. Evaluating Program Implementation: Concepts, Methods, and Applications

25. Performance Measurement in the Public and Nonprofit Sectors

This workshop is full. There is no waiting list available for this workshop. Please choose another.

Managing for Results! Performance-Based Budgeting! Balanced Score Cards! Dash Boards! Program managers and executives in the public and non profit sectors are being pushed to embrace these approaches to assessing how well their  programs and agencies are doing. The unifying thread linking all of these efforts is performance measurement. So what is needed to measure performance in an effective and useful manner?

This workshop will provide you with instruction, materials and exercises to increase your understanding of what constitutes performance measurement and how to measure program performance in the public and nonprofit sectors.

You will learn:

• How to identify pressures and opportunities for measuring performance,

• The political challenges to measuring performance and how to respond to them,

• Ways to assess the reliability and validity of performance measures,

• How to identify performance measures for social services,

• Approaches to getting performance measures used.

Kathryn Newcomer is the Director of the Phd in Public Policy and Administration program at the George Washington University where she teaches public and nonprofit program evaluation, research design, and applied statistics. She conducts research and training for federal and local government agencies on performance measurement and program evaluation, and has published five books and numerous articles about performance measurement in the government and nonprofit sectors.

Session 25: Performance Measurement
Scheduled: Wednesday, November 7, 8:00 AM to 3:00 PM
Level: Beginner, no prerequisites

26. How to Prepare an Evaluation Dissertation Proposal

Developing an acceptable dissertation proposal often seems more difficult than conducting the actual research. Further, proposing an evaluation as a dissertation study can raise faculty concerns of acceptability and feasibility. This workshop will lead you through a step-by-step process for preparing a strong, effective dissertation proposal with special emphasis on the evaluation dissertation.

The workshop will cover such topics as the nature, structure, and multiple functions of the dissertation proposal; how to construct a compelling argument; how to develop an effective problem statement and methods section; and how to provide the necessary assurances to get the proposal approved. Practical procedures and review criteria will be provided for each step. The workshop will emphasize application of the knowledge and skills taught to the participants’ personal dissertation situation through the use of an annotated case example, multiple self-assessment worksheets, and several opportunities for questions of personal application.

You will learn:

• The pros and cons of using an evaluation study as dissertation research,

• How to construct a compelling argument in a dissertation proposal,

• The basic process and review criteria for constructing an effective problem statement and methods section,

• How to provide the assurances necessary to guarantee approval of the proposal.

Nick L Smith is the co-author of How to Prepare a Dissertation Proposal from Syracuse University Press and a past-president of AEA. He has taught research and evaluation courses for over 20 years at Syracuse University and is an experienced workshop presenter through NOVA University's doctoral program in evaluation.

Session 26: Evaluation Dissertation
Scheduled: Wednesday, November 7, 8:00 AM to 3:00 PM
Level: Beginner, no prerequisites

27. Theory-Driven Evaluation for Assessing and
Improving Program Planning, Implementation, and Effectiveness

Learn the theory-driven approach for assessing and improving program planning, implementation and effectiveness. You will explore the conceptual framework of program theory and its structure, which facilitates precise communication between evaluators and stakeholders regarding evaluation needs and approaches to addressing those needs. From there, the workshop moves to how program theory and theory-driven evaluation are useful in the assessment and improvement of a program at each stage throughout its life-cycle.

Mini-lectures, group exercises and case studies will illustrate the use of program theory and theory-driven evaluation for program planning, initial implementation, mature implementation and outcomes. In the outcome stages, you will explore the differences among outcome monitoring, efficacy evaluation and effectiveness evaluation.  

You will learn:

• How to apply the conceptual framework of program theory and theory-driven evaluations,

• How to conduct theory-driven process and outcome evaluations,

• How to conduct integrative process/outcome evaluations,

• How to apply program theory to improve program planning processes.

Huey Chen, professor at the University of Alabama at Birmingham, is the author of Theory-Driven Evaluations (SAGE), the classic text for understanding program theory and theory-driven evaluation and most recently of Practical Program Evaluation (2005). He is an internationally know workshop facilitator on the subject.

Session 27: Theory-Driven Evaluation
Prerequisites: Basic background in evaluation.
Scheduled: Wednesday, November 7, 8:00 AM to 3:00 PM
Level: Intermediate

28. Introduction to IRT/Rasch Measurement in Evaluation

Program evaluation has great need for the development of valid measures, e.g. of the quantity and quality of services and of the outcomes of those services. Many evaluators are frustrated when existing instruments are not well tailored to the task and do not produce the needed sensitive, accurate, valid findings. 

Through an extensive presentation, followed by discussion and hands-on work with data sets and computer-generated output, this workshop will explore Rasch Measurement as a means to effectively measure program services. It provides an overview of “modern” measurement as practiced using item response theory with a focus on Rasch measurement. Rasch analysis provides the social sciences with the kind of measurement that characterizes measurement in the natural sciences.

You will learn:

• Differences between Classical Test Theory and Rasch Measurement,

• Why, when, and how to apply Rasch measurement,

• Why and how Rasch seeks to create linear, interval measures,

• Interpretation of Rasch/Winsteps output

Kendon Conrad and Barth Riley are from the University of Illinois at Chicago. They bring extensive experience in both teaching about, and applying, Rasch measurement to evaluation. Their workshops have won high praise from participants for their down-to-earth, clear, applied presentation with discussion.

Session 28: Rasch Measurement
Prerequisites: Basic background in evaluation.
Scheduled: Wednesday, November 7, 8:00 AM to 3:00 PM
Level: Intermediate

29. Applications of Multiple Regression in Evaluation: Mediation, Moderation, and More

Multiple regression is a powerful tool that has wide applications in evaluation and applied research. Regression analyses are used to describe relationships, test theories, make predictions with data from experimental or observational studies, and model linear or nonlinear relationships. Issues we’ll explore include selecting specific regression models that are appropriate to your data and research questions, preparing data for analysis, running the analyses, interpreting the results, and presenting findings to a nontechnical audience.

The facilitator will demonstrate applications from start to finish with SPSS and Excel, and then you will tackle multiple real-world case examples in small groups. Detailed handouts include explanations and examples that can be used at home to guide similar applications.

You will learn:

• Concepts important for understanding regression,

• Procedures for conducting computer analysis, including SPSS code,

• How to conduct mediation and moderation analyses,

• How to interpret SPSS regression output,

• How to present findings in useful ways.

Dale Berger is Professor of Psychology at Claremont Graduate University where he teaches a range of statistics and methods courses for graduate students in psychology and evaluation. He was President of the Western Psychological Association and recipient of the WPA Outstanding Teaching Award.

Session 29: Multiple Regression
Prerequisites: Basic understanding of Statistics
Scheduled: Wednesday, November 7, 8:00 AM to 3:00 PM
Level: Intermediate

30. Public Health Evaluation: Getting to the Right Questions

In 1999, the Centers for Disease Control and Prevention published the Evaluation Framework to provide public health professionals with a common evaluation frame of reference. Public health practitioners have successfully used the framework in a variety of settings and contexts. Beyond the framework, however, there are nuances and complexities to planning and implementing evaluations in public health settings.

Employing discussions, real examples, and activities, this workshop will focus on topics for evaluators to consider and strategies for approaching public health evaluations to get to the right questions to be addressed in a variety of evaluation contexts. This session will go beyond the CDC Evaluation Framework to examine confounders and complexities of public health evaluation.

You will learn

• Unique aspects of evaluability assessment in public health settings,

• Elements to consider to get to the right questions for the evaluation, including politics, accountability, ongoing evaluations, and rotating personnel,

• Strategies to work with stakeholders to identify what types of evidence will have credibility,

• Strategies to develop indicators for chosen evaluation questions.

Mary V Davis is Director of Evaluation Services at the North Carolina Institute for Public Health and Adjunct Faculty in the University of North Carolina School of Public Health where she teaches several advanced evaluation courses. Diane Dunet is a Senior Program Evaluator in the Division of Nutrition and Physical Activity where she conducts and supervises public health evaluations.

Session 30: Public Health Eval
Prerequisites: Basic understanding of Evaluation
Scheduled: Wednesday, November 7, 8:00 AM to 3:00 PM
Level: Intermediate

31. Managing Experimental Designs in Evaluation

This workshop is full. There is no waiting list available for this workshop. Please choose another.

Evaluators and administrators are increasingly expected to conduct studies using what are called scientifically-based methods. This workshop will provide you with the knowledge and ability to design and implement both random assignment experiments and alternative rigorous designs that can satisfy demands for scientifically-based methods.

With an emphasis on hands-on exercises and individual consultation within the group setting, this workshop will provide you with concrete skills in improving your current or anticipated work with experimental design studies. 

You will learn:

• How to conduct evaluability assessments of experimental and quasi-experimental designs,

• How to write or evaluate proposals to satisfy demands for scientifically-based research methods,

• How to modify experimental designs to respond to specific contexts,

• How to conduct quantitative analyses to strengthen the validity of conclusions and reveal hidden program impacts.

George Julnes, Associate Professor of Psychology at Utah State University, has been contributing to evaluation theory for over 15 years and has been working with federal agencies, including the Social Security Administration, on the design and implementation of randomized field trials. Fred Newman is a Professor at Florida International University with over thirty years of experience in performing front line program evaluation studies.

Session 31: Experimental Design
Prerequisites: Understanding of threats to validity and the research designs used to minimize them, practical experience with eval helpful.
Scheduled: Wednesday, November 7, 8:00 AM to 3:00 PM
Level: Intermediate

32. Visual Presentations of Quantitative Data 

Presenting data through graphics, rather that numbers, can be a powerful tool for understanding data and disseminating findings. Unfortunately, this method is commonly used to confuse audiences, complicate research and obscure findings.  

This workshop will enable participants to capitalize on the benefits of visual representation by providing them with tools for displaying data graphically for presentations, evaluation reports, publications and continued dialogue with program funders, personnel and recipients. This workshop will teach you about cognitive processing and heighten your awareness of the common errors made when visually displaying multivariate relationships, making you a more critical consumer of quantitative information.

You will learn:

• How people process visual information,

• Ways to capitalize in cognitive processing and minimize cognitive load,

• Quick and easy methods for presenting data,

• Innovative methods for graphing data,

• Ways to decide if quantitative information should be presented graphically/visually versus in words/text.

Stephanie Reich is an assistant professor in the department of education at the University of California, Irvine where her research focuses on cognitive development and how people process information. David Streiner is a Professor of Psychiatry at the University of Toronto and has authored four widely used books in statistics, epidemiology and scale development. 

Session 32: Visual Presentations
Scheduled: Wednesday, November 7, 8:00 AM to 3:00 PM
Level: Beginner, no prerequisites

33. The "State of the Art" in Evaluation

Together we will explore four 'hot topics' in the field today: (1) The Geography of the Discipline -- What are its components, what differentiates it from other disciplines, from a collection of bits and pieces, leaving 'the brand,' etc.; (2) Current Models and Theories -- their variety, strengths and weaknesses, best uses, possible future directions, etc.; (3) Methodologies and Uses -- what have we borrowed and extended or improved, which if any are new and distinctive of evaluation, what is still needed; and (4) The Rest of the Story -- the political, psychological, economic, educational dimensions, and what else; and, finally, what should we treat as the priorities for future development?

This will be a participatory workshop aimed at developing everyone's individual perspective on the present situation in the discipline of evaluation. The facilitator will 'open the bidding' on each topic with a summary and then chair a discussion including all, to take the topic further. We'll engage in discussion about the current state of the art in evaluation, the directions the discipline is taking, and what we can do to set and realize priorities for future development.

You will learn:

• What evaluators bring to the table that our research counterparts do not,

• How one's identity as an evaluator provides a unique stance and framing for client and stakeholder relationships,

• The state of the art in evaluation - what innovations lie out there on the cutting edge,

• Where is the discipline headed and how might we contribute to a valued and valuable future for evaluation.

Michael Scriven is among the most well-known professionals in the field today with 25 years of work on the philosophy of science. He has over 90 publications in the field of evaluation. Michael is excited to offer this brand new workshop at Evaluation 2007.

Session 33: State of the Art
Scheduled: Wednesday, November 7, 8:00 AM to 3:00 PM
Level: Beginner, no prerequisites

Half Day Workshops, Wednesday, November 7, 8 AM to 11 AM

36. Conducting Online Surveys

You will learn:

• A variety of cross-disciplinary and international definitions of racism,

• Strategies for removing/averting racism's presence in evaluation processes,

• Common places where racism may hide and influence the context of programs and problems,

• How to collect five broad types of data concerning racism as a variable,

• Strategies for collecting data on eight of the several dozen types of racism described in contemporary cross-disciplinary English-language research literature.

Pauline Brooks is an evaluator and researcher by formal training and practice. She has had years of university-level teaching and evaluation experience in both public and private education, particularly in the fields of education, psychology, social work and public health. For over 20 years, she has worked in culturally diverse settings focusing on issues pertaining to underserved populations, class, race, gender, and culture.

Session 38: Racism in Evaluation
Prerequisites: Previous thinking, work, or study in the area of discrimination's influence on programs and processes and an openness to further dialogue and exploration of racism.
Scheduled: Wednesday, November 7, 8:00 AM to 11:00 AM
Level:  Intermediate

39. Evaluating Large Scale Initiatives Using Systems Thinking

You will learn:

• How to consider three different system dynamics—organized, self-organizing, and unorganized—within large scale initiatives,

• How to align different evaluation designs with different types of system dynamics,

• How to design and apply nonlinear systems dynamics modeling to organized system dynamics,

• How to design and apply evaluation methods to self-organizing system dynamics,

• How this systems orientation relates to commonly used evaluation methods (e.g., outcomes, process, participatory, and empowerment evaluation methods).

Beverly Parsons is the executive director of InSites in Colorado and has over 20 years experience in evaluating education and social service initiatives. She focuses on the evaluation and planning of systemic change. Teresa Behrens is the  director of evaluation at the WK Kellogg Foundation. She recently co-edited a special issue of the American Journal of Community Psychology on systems change.

Session 39: Using Systems Thinking
Prerequisites: Knowledge of or experience in conducting or planning cluster and/or multi-site evaluations.
Knowledge of multiple theories of evaluation
Scheduled: Wednesday, November 7, 8:00 AM to 11:00 AM
Level: Intermediate

40. Using Stories in Evaluation

Stories are an effective means of communicating the ways in which individuals are influenced by educational, health, and human service agencies and programs. Unfortunately, the story has been undervalued and largely ignored as a research and reporting procedure. Stories are sometimes regarded with suspicion because of the haphazard manner in which they are captured or the cavalier promise of what the story depicts.

Through short lecture, discussion, demonstration, and hands-on activities, this workshop explores effective strategies for discovering, collecting, analyzing and reporting stories that illustrate program processes, benefits, strengths or weaknesses.

You will learn:

• How stories can reflect disciplined inquiry,

• How to capture, save, and analyze stories in evaluation contexts,

• How stories for evaluation purposes are often different from other types of stories.

Richard Krueger is a senior fellow at the University of Minnesota and has been actively listening for evaluation stories for over a decade. He has offered well-received professional development workshops at AEA and for non-profit and government audiences for over 15 years. Richard is a past president of AEA.

Session 40: Using Stories
Scheduled: Wednesday, November 7, 8:00 AM to 11:00 AM
Level: Beginner, no prerequisites

41. Level Best: How to Help Small and Grassroots Organizations Tackle Evaluation

Small and grassroots organizations usually have much different needs and funding sources  than larger organizations.  This workshop is based on the presenters' new book, Level Best: How Small and Grassroots Organizations Can Tackle Evaluation and Talk Results. This workshop introduces the concept of "rolling evaluation," and emphasizes that evaluation at its best is about learning rather than judging, about improving rather than proving, and that overall, evaluation does not need to be costly, overwrought or burdensome.

Through lecture, discussion and handouts, we will address the myths and misperceptions surrounding evaluation of nonprofit programs, and through the sharing of specific tools and strategies, will teach you how to support nonprofits in their evaluation efforts.

42. Empowerment Evaluation

Empowerment Evaluation builds program capacity and fosters program improvement. It teaches people to help themselves by learning how to evaluate their own programs. The basic steps of empowerment evaluation include: 1) establishing a mission or unifying purpose for a group or program; 2) taking stock - creating a baseline to measure future growth and improvement; and 3) planning for the future - establishing goals and strategies to achieve goals, as well as credible evidence to monitor change. The role of the evaluator is that of coach or facilitator in an empowerment evaluation, since the group is in charge of the evaluation itself.

Employing lecture, activities, demonstration and case examples ranging from townships in South Africa to a $15 million Hewlett-Packard Digital Village project, the workshop will introduce you to the steps of empowerment evaluation and tools to facilitate the approach.

You will learn:

• How to plan and conduct an empowerment evaluation,

• Ways to employ new technologies as part of empowerment evaluation including use of digital photography, quicktime video, online surveys, and web-based telephone/videoconferencing,

• The dynamics of process use, theories of action, and theories of use.

David Fetterman hails from Stanford University and is the editor of (and a contributor to) the recently published Empowerment Evaluation Principles in Practice (Guilford). He Chairs the Collaborative, Participatory and Empowerment Evaluation AEA Topical Interest Group and is a highly experienced and sought after facilitator.

Session 42: Empowerment Evaluation
Scheduled: Wednesday, November 7, 12:00 PM to 3:00 PM
Level: Beginner, no prerequisites

43. Strategies to Respond to the Top 10 Problems, Challenges and Headaches
with Focus Group Interviewing

Focus groups don't always work as expected. Find out what leading focus group practitioners say are their top 10 problems, challenges and headaches AND how they solve those concerns. Also, find out the concerns among AEA members. AEA workshop participants will be invited to submit their top concerns as well. So, make your list, attend the workshop and discover helpful solution strategies.

Through presentations, question-and-answer, and exploration of responses to attendee challenges, this workshop will help you move your focus group facilitation skills to the next level.

You will learn:

• Challenges with focus groups as seen by professional focus group moderators,

• Challenges with focus groups as seen by AEA members,

• Solution strategies to the top 10 problems, challenges, and headaches.

Richard Krueger is a senior fellow at the University of Minnesota. In 30+ years of practice he has conducted thousands of focus group interviews and he still gets excited about listening to people. He is the author of 6 books on focus group interviewing and is a past president of AEA.

Session 43: Focus Group Challenges
Prerequisites: Experience in conducting focus groups
Scheduled: Wednesday, November 7, 12:00 PM to 3:00 PM
Level: Intermediate

46. Getting to Outcomes in Public Health

Getting To Outcomes: Methods and Tools for Planning, Evaluation, and Accountability (GTO) was developed as an approach to help practitioners plan, implement, and evaluate their programs to achieve results. GTO is based on answering 10 accountability questions. By answering the questions well, program developers increase their probability of achieving outcomes and demonstrate their accountability to stakeholders.
Addressing the 10 questions involves a comprehensive approach to results-based accountability that includes evaluation and much more.

Through lecture, demonstration, and hands-on activities, this workshop will explore the basics of the GTO approach and provide resources for further investigation and action. Research funded by a grant from CDC has shown that use of the GTO model can improve individual capacity and program performance to facilitate the planning, implementation, and evaluation of prevention programs. GTO has been customized for several areas of public health including: substance abuse prevention, underage drinking prevention, positive youth development, teen pregnancy prevention, and emergency preparedness.

You will learn:

• A comprehensive approach to results-based accountability

• Ten questions to ask to improve program planning, accountability, and results

• How to select evidence-based models and best practices

• Strategies for continuous program improvement

Abraham Wandersman is a Professor of Psychology at the University of South Carolina-Columbia. He is a co-author of Prevention Plus III and a co-editor of Empowerment Evaluation: Principles in Practice. Catherine Lesesne is a Behavioral Scientist in the Division of Reproductive Health at the Centers for Disease Control and Prevention. She is the lead author of the newly developed GTO manual, Promoting Science-based Approaches to Teen Pregnancy Prevention Using Getting to Outcomes.

Session 46: Getting to Outcomes
Scheduled: Sunday, November 11, 9:00 AM to 12:00 PM
Level: Beginner, no prerequisites

You will learn: