KEYNOTES

Keynote: Attributing “Cause” in Complex Programs: If We Want More Evidence-Based Practice, We Need More Practice-Based Evidence

The evidence-based medicine movement since the 1970s has insinuated its way into American public health thinking, first via the Canadian Task Force on the Periodic Health Examination and the U.S. Preventive Services Task Force, then through the Cochran Collaboration and most recently the US Task Force for Community Preventive Services. A tenet of these systematic reviews leading to the identification and recommendation of “best practices” is that they must give greatest weight to evidence that attributes the causal strength of interventions based on randomized controlled trials (RCTs). The pursuit of randomization and other controls in RCTs has produced evidence that is strong on internal validity, but often weak on external validity. The traditions of evaluation of actual programs in vivo and in real time, rather than in situ under artificially controlled circumstances, promises greater relevance of the findings for practice and greater external validity in generalizing the findings to other “real world” practices and populations. This presentation will examine ways to supplement “best practices” from strong evidence with “best processes” of applying the always-limited evidence in real world settings and diverse populations, and ways that systems modeling of outcomes (“in sylico”) might help bridge the gaps.

Prof Lawrence W. Green co-leads the Social and Behavioral Sciences Program at the University of California at San Francisco Comprehensive Cancer Center. He recently retired from CDC as Distinguished Fellow-Visiting Scientist and Director of the Office of Science and Extramural Research. He has served on the medical and public health faculties at Berkeley, Johns Hopkins, Harvard, Texas, UBC, Emory, and Maryland Universities. He is a past President and Distinguished Fellow of the Society for Public Health Education, recipient of the American Public Health Association's highest awards, the Distinguished Career Award and Award of Excellence, and the American Academy of Health Behavior Research Laureate Medal.

Click here for the handouts! 

Attendees might also be interested in the handouts from Larry Green's Training Session.

Keynote: How to Work With FORECAST and Logic Models as an Ecological Empowerment Evaluation Strategy 

The Formative Evaluation, Consultation, and Systems Technique (FORECAST) is a participatory approach to the evaluation of complex community initiatives.  FORECAST is an empowerment evaluation method because community stakeholders are part of the design, planning, implementation, and reflection of findings in the evaluation process.  Attendees at the plenary session will receive an overview of the FORECAST method by illustration of its application to an actual community-based project to prevent first time male sexual violence against adolescent women.  The evaluation was funded by the Centers for Disease Control and Prevention Division of Violence Prevention.  The program example will illustrate how FORECAST uses program logic models that are developed locally as a means for developing evaluation markers and measures that are employed in real time in order to provide the program with timely feedback for continuous quality improvement.  The FORECAST approach will be introduced by a discussion that illustrates why alternative evaluation approaches to randomized, controlled and quasi-experimental designs often are more appropriate for the evaluation of complex community programs.

Robert M. Goodman, Ph.D., M.P.H., M.A., is a Professor and Chair, Department of Behavioral and Community Health Sciences, Graduate School of Public Health, University of Pittsburgh. Dr. Goodman has written extensively on issues concerning community health development, community capacity, community coalitions, evaluation methods, organizational development, and the institutionalization of health programs. He has been the principal investigator and evaluator on projects for CDC, The National Cancer Institute, The Centers for Substance Abuse Prevention, The Children’s Defense Fund, and several state health departments. In 2004, Dr. Goodman received the Distinguished Fellow Award from the Society for Public Health Education, the highest honor it bestows. Currently, Dr. Goodman is consulting on community-based public health practices and empowerment evaluation with the Diabetes Translation and Injury Prevention Branches at CDC. Also, he is leading an evaluation of community-based approaches to increasing interest in cancer clinical trials.

Click here for the handouts!

Keynote: Exploratory Evaluation: Efficient Allocation of Evaluation Resources

Exploratory evaluation, also known as evaluability assessment, is a pre-evaluation technique that can improve the chances of obtaining useful information from evaluation. It has been extensively used in health and education programs and is becoming increasingly common in recent years. This presentation will outline the technique and illustrate with applications at federal, state and local levels. The participant will gain a better understanding of the advantages and limitations of this technique.

Laura C. Leviton, Ph.D. is a Senior Program Officer of the Robert Wood Johnson Foundation.  She has overseen evaluations in most of the areas of focus for the Foundation, and now works primarily on initiatives in preventing childhood obesity and in improving the quality of medical care.  Before joining the Foundation she was a Professor of Public Health at University of Alabama at Birmingham (UAB) and before that, on faculty of the University of Pittsburgh School of Public Health.  Dr. Leviton is a leading writer on evaluation methods and practice, in particular for disease prevention.  She was President of the American Evaluation Association in the year 2000, coauthored a leading evaluation text, and serves on several editorial boards for evaluation journals.  She received her Ph.D. in social psychology from University of Kansas and postdoctoral training in evaluation research from Northwestern University.

Click here for the handouts!

MONDAY  TRAININGS

Using the Balanced Scorecard: A Tool in Performance Monitoring

Level: All

The emphases on accountability and continuous quality improvement in private, public, and non-profit sectors has created renewed interest  in performance measurement and systems of “dashboard metrics.”  Yet, unless these are linked organically to an organization’s mission and strategy, time and effort put into measurement will not pay off in program improvement.  This session will showcase the Balanced Scorecard as an approach to translating organizational mission and strategy into a strategic measurement and management system. The session will provide an overview of the Balanced Scorecard approach, which assesses functions across four “balanced” perspectives: financial, customers, internal business processes, and learning and growth, and discuss the necessary modifications to the approach to make it most useful for public and non-profit organizations. The presentation will allow the participants to better understand and implement the Balanced Scorecard process, with specific emphasis on: effectively planning the Balanced Scorecard process and exercising flexibility in adapting the process to your performance management framework;  identifying the critical performance measures, targets and initiatives,  and creating and sustaining overall Balanced Scorecard success.

Harry E. Chambers is President of Trinity Solutions, Inc., an Atlanta-area consulting and training firm specializing in organizational development, leadership, and team coaching and building. He is a nationally-known presenter and trainer, and an award-winning author on topics in leadership and management, most recently, No Fear Management:  Rebuilding Trust, Performance, and Commitment in the New American Workplace and The Bad Attitude Survival Guide. Harry has trained, facilitated and consulted with leaders of America’s top corporations as well as public sector and non-profit agency leadership. He is a frequent presenter at Inc. Magazine’s Leadership Development Conferences and his work has been featured on the web sites of CNN FN and Business Week OnLine. His articles, excerpts, and direct quotes appear in many publications and journals, including Executive Excellence, Investors Business Daily, Success and Cosmopolitan magazines. He has been interviewed on NBC’s Today Show, ABC’s Morning News and National Public Radio’s Morning Edition.

Click here for the handouts!

Evaluation 101: An Overview for New Evaluation Practitioners

This training session is full. Registration is no longer available for this session.

Level: Beginner
 
This course will provide an introduction to program evaluation and evaluation terminology for Institute participants with little or no prior background in program evaluation. The teaching framework for the course will be the Centers for Disease Control and Prevention’s (CDC) six-step Framework for Program Evaluation in Public Health as well as the four sets of evaluation standards from the Joint Commission on Evaluation Standards. The course will touch on all six steps of the CDC Framework but will emphasize the early steps, including identification and engagement of stakeholders, creation of logic models, and selecting/focusing evaluation questions. Several public health case studies will be used both as illustrations and as an opportunity for participants to apply the content of the course.

Thomas Chapel, M.A., M.B.A., is a Senior Health Scientist in the Office of the Director, Office of Program Planning and Evaluation at the Centers for Disease Control and Prevention. He is responsible for helping CDC programs build evaluation capacity by providing training and technical assistance on strategic planning and evaluation design; building networks on evaluation and planning across CDC; and providing a range of training and facilitation in strategic planning and evaluation design. Before joining CDC, Mr. Chapel was Vice-President of the Atlanta office of ORC Macro where he directed and managed projects in program evaluation, strategic planning, and evaluation design for public and non-profit organizations. He is a frequent presenter at national meetings and facilitated and serves on expert panels on a host of public health and evaluation topics. He is active in the local affiliate of the American Evaluation Association (AEA), currently as president and formerly as chair of the Professional Development Committee. Mr. Chapel holds a BA degree from Johns Hopkins University and MA in public policy and MBA degrees from the University of Minnesota.

Click here for the handouts!

Enhanced Group Facilitation: Techniques and Process

This training session is full. Registration is no longer available for this session.

Level: All

This workshop will familiarize participants with a variety of group facilitation techniques as well as the management of the facilitation process. Participants will learn how to choose a facilitation technique based on goals and objectives, anticipated outcome, type and number of participants, and logistics. Two to three facilitation techniques for generating ideas and focusing thoughts on both defined and undefined issues will be explored in greater detail. These techniques, and their variations, will include focus groups, item writing, and nominal group technique. Participants will also learn more about the different roles and responsibilities they may have in group facilitation (there are more than you think!), and how these roles intersect with the tasks inherent in planning and managing a group facilitation experience. Job aides and reference lists will be provided.

Jennifer Dewey, Ph.D., is a Technical Director with the research and evaluation professional services firm of ORC Macro. Jennifer leads Site Management, overseeing ongoing training and technical assistance to local evaluation teams for the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program. Prior positions include Director of Internal Evaluation at Learning Point Associates, Senior Consultant at Andersen, and post-doctoral scholar at the Center for Prevention Research at the University of Kentucky. Jennifer holds a doctorate in Applied Experimental Psychology with a specialization in program evaluation. Her knowledge and skills encompass project management, proposal development, methodological and statistical design, qualitative and quantitative analysis, needs assessment, survey  development, telephone and in-person interviews, and group facilitation. Jennifer has published in the Journal of Primary Prevention, American Journal of Evaluation, Advances in Developing Human Resources, and has made over 40 professional conference presentations.

Click here for the handouts!

Exploring Diverse Approaches to Causal Attribution and Practice-Based Evidence

(Register for: Causal Attribution and Practice-Based Evidence)

Level: Intermediate

Following on the keynote presentation in the previous hour, this interactive discussion session will provide opportunities to explore the experience of evaluators and other participants in constructing evidence and using evidence together with theory, professional experience, and indigenous colloquial knowledge in the planning of programs and the design of interventions. The gaps inevitably found between science and practice, and the fit of scientific findings with the situations in which such findings would seem to recommend specific “best practices,” will be examined to discover ways in which such gaps might be closed and the fit improved. We will challenge the conventions of inferring “best practices” from specific types of systematic literature review criteria, and the peer review mechanisms that support their perpetuation in funding and publication decisions. We will explore some approaches to assuring greater external validity in the decisions leading to the funding and publication of research, giving greater weight to context and to moderating variables to balance the current emphasis on internal validity, context-free universal truths implied by the weight given to controlled trial evidence.

Professor Lawrence W. Green co-leads the Social and Behavioral Sciences Program at the University of California at San Francisco Comprehensive Cancer Center. He recently retired from CDC as Distinguished Fellow-Visiting Scientist and Director of the Office of Science and Extramural Research. He has served on the medical and public health faculties at Berkeley, Johns Hopkins, Harvard, Texas, UBC, Emory, and Maryland Universities. He is a past President and Distinguished Fellow of the Society for Public Health Education, recipient of the American Public Health Association's highest awards, the Distinguished Career Award and Award of Excellence, and the American Academy of Health Behavior Research Laureate Medal.

Click here for the handouts!

Sampling 101: Basics of Probability and Purposeful Sampling

Level: All

Choosing and implementing an appropriate sampling strategy can affect the validity, credibility and cost of an evaluation.  Some studies require sophisticated probability sampling methods to produce accurate estimates of the characteristics of the populations served or of the size of the effects of the program or policy on the target population.  Other studies may appropriately use purposeful samples to support theory development or to do detailed case analysis.  In this workshop, participants will be exposed to alternative sampling strategies that are frequently used in evaluation and social research.  In the workshop, the instructor will address the 14 questions from his book Practical Sampling (Sage, 1990) that should be answered prior to sample design, as a part of sample design, and prior to analysis of the data.  The participants will become acquainted with ways to plan and implement sampling strategies that meet the needs of an evaluation.  Examples will be used to illustrate the designs and issues that arise in implementation.  Participants will have the opportunity to raise specific sampling issues that they have encountered in their own work.

Gary T. Henry is a professor in the Andrew Young School of Policy Studies at Georgia State University.  He previously served as the Director of Evaluation and Learning Services for the David and Lucile Packard Foundation.  Henry has evaluated a variety of public policies and programs and has published extensively in the field of evaluation and education policy.  He received the Evaluation of the Year Award from the American Evaluation Association in 1998 for his work with the Georgia’s Council for School Performance and the Joseph S. Wholey Distinguished Scholarship Award in 2001 from the American Society for Public Administration and the Center for Accountability and Performance.  Dr. Henry currently serves on scientific peer review panels for the Institute of Education Sciences, U.S. Department of Education and on a National Research Council/National Academy of Science panel assessing the effects of “green schools” on the health and productivity of teachers and students.

Click here for the handouts!

Systems Thinking and Public Health

This training session is full. Registration is no longer available for this session.

Level: Intermediate 

The persistent challenges that show up most in public health often share common characteristics. They defy quick fixes, they build and dissipate slowly over time, they are rarely “owned” by any one player in the system but affect all, and their components are tightly coupled. The analytical approaches of systems thinking, which was invented in the 1950s at MIT, was created to address such “systems” problems. Systems thinking uses diagramming and simulation modeling to help groups of people improve their understanding of how to improve the performance of a range of social/physical systems such as a city, an ecosystem, an industry, or, in this context, a population facing a threat to their health. In this interactive session, participants will learn about a systems thinking perspective on public health, learn about causal mapping techniques, learn about insights generated by a computer-based “management flight simulator” created with the Division of Diabetes Translation at CDC, and have the opportunity to reflect on applying systems thinking to the public health challenge they face.

Drew Jones is a project director with Sustainability Institute. His work focuses on applying systems thinking approaches to a wide range of societal concerns, particularly in the public health and environmental areas. Working with CDC’s Division of Diabetes Translation, he has led the development of the Diabetes System Model, which is shaping diabetes strategies at the national and state level. A paper on the work was recently published in the March 2006 issue of AJPH. Mr. Jones received a B.A. in Engineering Sciences and Environmental Studies from Dartmouth College and earned a M.S. from M.I.T., where he studied System Dynamics simulation modeling under John Sterman and Peter Senge.

Click here for the handouts!

Logic Models as a Platform for Program Evaluation Planning, Implementation, and Use of Findings

Level: All

Practitioners use logic models to describe important components of a program; make visible a theory of change; and link activities to intended outcomes.  For the purposes of evaluation practice, a well-constructed logic model provides a program-specific foundation for identifying evaluation questions; prioritizing data needs; and translating findings into recommendations for ongoing program improvement.  Aimed directly at improving the utility of logic models and quality of evaluation practice in your setting, the workshop addresses 2 questions:

(1)   What are the hallmarks of a well-constructed, scientifically-sound and useful logic model?

(2)   How do we maximize the use of logic models for program evaluation planning, implementation and use of findings?

Workshop Objectives:

• Demystify and define the logic model as a starting point for everyday evaluation practice

• Identify the hallmarks of a well-constructed, scientifically-sound logic model

• Clarify the relevance of process and outcome evaluation to preparing and using program logic models

• Demonstrate the use of logic models to identify and prioritize evaluation questions and data needs

• Examine the use of logic models to identify opportunities/options for demonstrating accountability for scarce resources

• Demonstrate use of a logic model to guide preparation of findings/recommendations aimed at ongoing program improvement

• Pinpoint additional resources for continued study/application

Goldie MacDonald, Ph.D. is the National Evaluation Coordinator for the Steps to a HealthierUS Cooperative Agreement Program in CDC’s National Center for Chronic Disease Prevention and Health Promotion. Dr. MacDonald is an experienced facilitator and provides leadership and technical expertise on program evaluation planning, implementation, and the use of findings to inform program improvement at both the national and community levels. Much of her work focuses on identifying appropriate strategies for documenting program processes and outcomes toward increased accountability to diverse program stakeholders. She also provides expertise on participatory approaches to program evaluation, utilizing qualitative methods of data collection, and the use of logic models as a platform for program planning and evaluation. She is lead author of “Introduction to Program Evaluation for Comprehensive Tobacco Control Programs.” For their work on this publication, the authors received the Alva and Gunnar Myrdal Award for Government from AEA in November 2002.

Sorry- Handouts are posted online at the presenters discretion and are not available for this session.

Principled Discovery: Learning From Your Data, with Reason 

Level: Intermediate - one should take the Training or the Breakout with Dr. Mark, but not both

Too often, quantitatively-trained evaluators are taught to test only a priori hypotheses. But evaluation data are too valuable not to try to learn the unanticipated lessons they may have to tell. These can be lessons about how the program operates differently in different contexts or with different subgroups of client.  Or lessons about why the program works (or doesn’t).  Or lessons about different subgroups of clients, or service providers, that we didn’t realize existed in advance.  Some evaluators have learned ways to "snoop" in or “mine” their data. However, without ways to make the snooping/ mining principled, there are serious risks of getting it wrong.

We will explore many ways of learning from one's data, including tests of moderation; graphical methods; post hoc use of clustering methods; exploratory investigation of higher-level variables in multilevel models; and exploratory searching for mediators and countervailing effects. In addition, we will discuss multiple ways of making new discoveries principled, other than the familiar notions of replication.  The session will also provide a conceptual introduction to (or review of) some current quantitative methods such as multilevel models and structural equation modeling.

Melvin M. Mark, Ph.D. is Professor of Psychology at Penn State University , Editor Emeritus of the American Journal of Evaluation and President-elect of the American Evaluation Association. He has conducted federally funded evaluations in the areas of prevention programs for at-risk youth, federal personnel policies, and industrial modernization, and has been involved in evaluations of state and local programs.  An award-winning teacher, he has published numerous papers and chapters on the theory and design of evaluation.  Among his books are Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs (Jossey-Bass, 2000) and a forthcoming Handbook of Evaluation (SAGE London).

Click here for the handouts!

Focus Group Research: Understanding, Designing and Implementing

This training session is full. Registration is no longer available for this session.

Level: All

As a qualitative research method, focus groups are an important tool to help researchers understand the motivators and determinants of a given behavior. This course provides a practical introduction to focus group research. At the completion of this course, participants will be able to 1) identify and discuss critical decisions in designing a focus group study,  2) understand how research or study questions influence decisions regarding segmentation, recruitment, and screening; and, 3) identify and discuss different types of analytical strategies and focus group reports.

Michelle Revels and Bonnie Bates are technical directors at ORC Macro specializing in focus group research and program evaluation.  Ms. Revels attended Hampshire College in Amherst, MA and the Hubert H. Humphrey Institute of Public Affairs at the University of Minnesota. Ms. Bates, also a trained and experienced focus group moderator and meeting facilitator, received her bachelor’s and master’s degree in criminal justice from the University of Maryland.

Click here for the handouts!

WEDNESDAY TRAININGS

Exploring Effect Size and Measures of Association

Level: Intermediate

Answer the call to report effect size and association measures as part of your evaluation results. This workshop will improve your capacity to understand and apply a range of measures including: standardized measures of effect sizes proposed by Cohen, Glass, and Hedges; Eta-squared; Omega-squared; the Intraclass correlation coefficient; and Cramer’s V. Through mini-lecture and demonstration you will improve your understanding of the theoretical foundation and computational procedures for each measure. The session will include: definitions of and procedures for computing a range of effect size and association measures, a presentation that examines the relationships among the common measures, and description of computation of selected confidence intervals for effect sizes and association measures. You will receive SPSS and SAS software program codes for performing many of the computations related to the measures and common confidence intervals.

Jack Barnette, PhD has served as a faculty member at Penn State University , University of Virginia , University of Memphis , University of Alabama at Tuscaloosa , University of Iowa , and is now Senior Associate Dean for Academic Affairs and Professor of Biostatistics at the University of Alabama at Birmingham . He has served as an APHA Statistics Council Member and Section representative to the APHA Action Board. Presently, he is chairing the ASPH biostatistics competency workgroup and is co-chair elect of the ASPH Biostatistics/Epidemiology Section. He has more than 30 years experience in teaching, advising students, and applying research, evaluation, and statistical methods to a wide variety of educational and public health projects. He has conducted evaluations of projects funded by CDC, HRSA, SAMHSA, NHLBI, and NIOSH. He serves on three of the ASPH/CDC Preparedness Exemplar Groups: Education and Evaluation Methods, Certificate Programs, and University-based Student Preparedness. He has been conducting research on the use of effect sizes and measures of association for the past seven years and he has presented pre-sessions on this topic at the last three AEA annual meetings. He holds the PhD in Educational Research and Development from Ohio State (1972).

Click here for the handouts!

Integrating Goals, Performance Measurement, and Evaluation
This training session is full. Registration is no longer available for this session.

Level: Intermediate

While this session will be taught by CDC staff, the focus is on how any large organization integrates evaluation, planning and performance monitoring.  CDC, like many large organizations, is moving to “performance-informed” management, where its program components align their efforts with enterprise-wide goals and objectives, progress is monitored through good performance measurement, and findings are continuously fed back for target evaluation and program improvement.  There are challenges in any organization at each step in this cycle, and then a second set of challenges in integrating the steps so that they are mutually supportive.  In this session,  using some cross-cutting cases, we’ll walk through the development of enterprise goals and objectives, how to align program efforts with goals, how to identify a strong set of performance measures, how to use performance measurement to guide program evaluation, and how to ensure that findings from performance measurement and evaluation feed program improvement.  We’ll focus especially on the key roles evaluators and evaluation skills can play in this continuous quality improvement cycle.

The CDC/OSI Goals Team has responsibility for coordinating and supporting the goals management process at CDC.  Currently, that means identifying, training, and providing analytic and logistical support to more than 20 goal action teams; helping those teams produce quality and timely products; creating knowledge management systems to support their efforts; and doing CDC-wide communication and training to build knowledge and support for the goals process among CDC programs and staff.

Click here for the handouts!

Designing Evaluations for Impact

This training session is full. Registration is no longer available for this session.

Level: Intermediate

Many evaluations have as a goal to impact decision-making or policy formulation by providing systematic empirical feedback.  Information can be intended for a variety of audiences including sponsors, client-groups, administrators, staff, and other relevant constituencies. Most often, evaluation information is considered to have an impact if it aids in decision-making. However, the relationship between an evaluation and its impact is not simple.  Based on empirical knowledge, we know that there are steps that an evaluator can take to help to ensure that an evaluation has impact.  The goal of this workshop is to provide participants with an understanding of, a framework for, and specific strategies that will increase the likelihood than an evaluation will have impact.  In this session, using participants’ own programs and experiences, we will first explore the conditions that can influence an evaluation’s impact.  Participants will then engage in interactive discussions about the specific methods used for increasing evaluation impact.  Small group activities and simulation and role-play exercises will be used to explore the use of specific methods for increasing impact and to foster original and rigorous thinking about how to apply the particular strategies presented to participants’ own contexts and programs.        

Christina A. Christie is an Assistant Professor, Director of the Masters of Arts Program in Psychology and Evaluation, and Associate Director of the Institute of Organizational and Program Evaluation Research at Claremont Graduate University.  Her research, which has been supported by several funders including the Haynes Foundation, focuses on investigating the relationship between evaluation theory and practice and issues related to the development of descriptive theories of evaluation.  She has also received funding from a variety of sources, including the National Science Foundation, the U.S. Department of Education, and the Hewlett-Carnegie Foundation to evaluate social, education, and health behavior programs targeting high-risk and underrepresented populations.  Christie co-founded the Southern California Evaluation Association, a local affiliate of the American Evaluation Association, and is the former Chair of the Theories of Evaluation Division of the American Evaluation Association.  In 2004, Christie received the American Evaluation Association’s Marcia Guttentag Early Career Achievement Award.

Sorry- Handouts are posted online at the presenters discretion and are not available for this session.

Advanced Applications of Program Theory

Level: Intermediate

A top 5 offering at Evaluation 2005! While simple logic models are an adequate way to gain clarity and initial understanding about a program, sound program theory can enhance understanding of the underlying logic of the program by providing a disciplined way to state and test assumptions about how program activities are expected to lead to program outcomes. 

Lecture, exercises, discussion, and peer-critique will help you to develop and use program theory as a basis for decisions about measurement and evaluation methods, to disentangle the success or failure of a program from the validity of its conceptual model, and to facilitate the participation and engagement of diverse stakeholder groups. 

You will learn:

• To employ program theory to understand the logic of a program,

• How program theory can improve evaluation accuracy and use,

• To use program theory as part of participatory evaluation practice.

Stewart I. Donaldson, Ph.D. is Professor and Chair of Psychology, Director of the Institute of Organizational and Program Evaluation Research, and Dean of the School of Behavioral and Organizational Sciences, Claremont Graduate University. He has conducted numerous evaluations, developed one of the largest university-based evaluation training programs, and taught and published widely on applied research and evaluation topics. His recent work includes a book with Michael Scriven about the future of evaluation practice - Evaluating Social Programs and Problems: Visions for the New Millennium (2003), a book Applied Psychology: New Frontiers and Rewarding Careers (2006), and a forthcoming book Program Theory-Driven Evaluation Science: Strategies and Applications. He is co-founder of the Southern California Evaluation Association, has served as Co-Chair of AEA's Theory-Driven Evaluation and Program Theory Topical Interest Group, and is on the Editorial Boards of the American Journal of Evaluation and New Directions for Evaluation.

Click here for the handouts!

Essential Competencies for Program Evaluators

Level: All

This half-day session is designed to teach participants the recently-developed Essential Competencies for Program Evaluators, a set of knowledge, skills, and attitudes in six categories, developed over the past six years.  The session will begin with the analysis of program evaluation vignettes representing diverse areas of practice to show both the common competencies across settings and those unique to specific contents or contexts.  Following a brief history of how the competencies were developed, the group will then collectively examine the competencies in all six categories: professional practice, systematic inquiry, situational analysis, project management, reflective practice, and interpersonal skills.  This activity, which builds on the continuum of interpersonal evaluation practice, will ground participants in the competencies’ content and allow people to ask questions as they think about their own work.  After a short break, participants will divide into role-alike or content-alike groups and develop concept maps to explore how the competencies make sense in their role or content area.  Comparative discussion will further illuminate the competencies, and then participants will complete a self-assessment tool and discuss how to set priorities and action steps for professional development.  Most of the session will consist of interactive exercises with just enough lecture to frame the discussion.

Jean A. King, Ph.D., is a Professor in the Department of Educational Policy and Administration at the University of Minnesota where she serves as the Director of Graduate Studies and Coordinator of the Evaluation Studies Program.  She holds an M.S. and Ph.D. from Cornell University and prior to her graduate study taught middle school English for a number of years.  In 1995, her work using participatory evaluation methods resulted in the Myrdal Award for Evaluation Practice from the American Evaluation Association, and in 1999, she was awarded the Association’s Robert Ingle Award for Extraordinary Service.  Professor King received the University of Minnesota, College of Education and Human Development’s Beck Award for Outstanding Instruction in 1999, the College’s 2002 Distinguished Teaching Award, and the 2005 Community Service Award.  She is the author of numerous articles and chapters and, with Laurie Stevahn, continues writing a book on interactive evaluation practice.

Click here for the handouts!

Strengthening Evaluation Through Cultural Relevance and Cultural Competence
This training session is full. Registration is no longer available for this session.

Level: Intermediate

This skill-building session addresses the centrality of culture in evaluation. It is organized in two segments. The opening segment addresses the relevance of culture to all stages of the evaluation process, to the fundamental validity of our work as evaluators, and to ethical standards and guidelines of our profession. Presenters will use an FAQ format to raise questions and address common misconceptions that marginalize discussions of culture within the evaluation community (e.g., Is “culture” really just a code-word for “race”? How does culture apply to me as a white evaluator working within predominantly white populations? What is the “value added” of culture in evaluation? Why should I care?) The second segment extends cultural relevance to present strategies for building cultural competence through experience, education and self-awareness. Theoretical frameworks that situate culture in evaluation (e.g., Frierson, Hood & Hughes, 2002; Hall & Hood, 2005; Kirkhart, 2005) are presented as advance organizers for practice and application purposes. Presenters use case scenarios and participants’ own examples to integrate workshop content with participants’ field experience, interests, and concerns. They rely on various theoretical frameworks to guide the two segments in tangible and practical ways. Additional resources are provided to extend and reinforce participant learning.

Karen E. Kirkhart holds a Ph.D. in Social Work and Psychology from The University of Michigan and is currently Professor, School of Social Work, College of Human Services and Health Professions, Syracuse University. Rodney K. Hopson has undergraduate and graduate degrees in English Literature, Educational Evaluation, and Linguistics from the University of Virginia, and he is Associate Professor and Chair, Department of Educational Foundations and Leadership and faculty member in the Center for Interpretive and Qualitative Research at Duquesne University. Karen and Rodney have served in positions of leadership within the American Evaluation Association, and both are actively involved in education and scholarship on culture, diversity, and social justice in evaluation. Rodney serves as Project Director for the American Evaluation Association/Duquesne University Graduate Education Diversity Internship Program. Karen is a member of the AEA Diversity Committee task force charged with developing a public interest statement on the subject of cultural competence and evaluation.

Click here for the handouts!

Edward Liebow, PhD, is Senior Research Scientist on the staff of Battelle's Centers for Public Health Research and Evaluation. Liebow has conducted policy-related and evaluation research throughout the western US and in South Australia focusing on applying ethnographic research methods to understand the distinctive response of disadvantaged communities to potential environmental and public health hazards posed by development programs and policies. Dr. Liebow is affiliated with the University of Washington, where he teaches courses in ethnographic research methods, American Indian Studies, and comparative urban politics. He is also a visiting professor of applied anthropology and comparative economics at the Università Carlo Cattaneo in Castellanza, Varese (Italy). He recently completed a term as President of the National Association for the Practice of Anthropology, and he also serves as Book Review Editor for the Journal of Political Ecology.

Click here for the handouts!

RealWorld Evaluation: Overcoming Constraints in International and National Contexts

Level: Intermediate

What do you do when asked to perform an evaluation on a program that is well underway? When your questions about baseline data and control groups are met with blank stares? When time and resources are few, yet clients expect “rigorous impact evaluation”? When there are political expectations and pressures to deal with? This workshop presents a seven-step approach that seeks to ensure the best quality evaluation under real-life constraints.

Through presentations and discussion, with real-world examples drawn from international development evaluation, participants in this workshop will be introduced to the RealWorld Evaluation approach. The workshop focuses on developing country evaluation, but the techniques are applicable to evaluators working in any context with budget, time, data and political constraints.

The fundamental framework for the session is an exploration of the seven steps of the RealWorld Evaluation approach, including ways to work negotiate RealWorld solutions with clients. We’ll also investigate methodological issues applicable to RealWorld Evaluations including approaches to reconstructing baseline data and how to identify and address threats to the validity or adequacy of evaluation methods. Finally, we’ll identify seven evaluation designs appropriate for RealWorld evaluations and discuss their appropriate context-dependent applications.

Jim Rugh brings 42 years of experience in international development, including 26 years specializing in program evaluation. For 11 years he has been the Coordinator of program Design, Monitoring and Evaluation for CARE, a large international NGO with programs in 70 countries reaching over 48 million beneficiaries. Under Jim’s leadership CARE has adopted an evaluation policy, strategies and standards to promote learning for improved program quality and accountability for effectiveness. An active member of AEA, and a founding member of AaEA (the Atlanta-area Evaluation Association), Jim has co-facilitated or facilitated evaluation workshops at AEA’s annual conference as well as in Cape Town South Africa, Kiev Ukraine, Washington DC and Atlanta. He is the co-author with Michael Bamberger and Linda Mabry of the RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints (SAGE Publications).

Click here for the handouts!

BREAKOUTS - Click on a breakout title or scroll down for descriptions