SCHEDULE OVERVIEW ¨ WORKSHOPS ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS

PRE-INSTITUTE WORKSHOPS

• NEW! PI1: (Taylor-Powell) The What and How of Outcome Measurement

• PI2: (Chapel) Introduction to Evaluation

• NEW! PI3: (Vincent-Mark) An Introduction to Participatory Monitoring and Evaluation: An Approach to Engaging Stakeholders and Communities in M&E

Note that these pre-institute workshops are not included in standard Institute registration, and require an additional payment, but may be registered for on the same form. We also welcome registrants who wish to attend only a pre-institute workshop and that, too, may be done using the regular registration form.

NEW! PI1: The What and How of Outcome Measurement

Level: Advanced Beginner
 
Description: Outcomes continue to be ‘in demand’ within the framework of results-based accountability. This workshop focuses specifically on outcome measurement within community programs and will cover content in three areas: (1) Choosing outcomes to measure; (2) Specifying indicators; and (3) Collecting data – data sources and data collection methods. Through hands-on, participatory techniques using case studies and practical examples, you will build your capacity in the what and how of outcome measurement. We will also cover issues, challenges, and emerging practices from the world of community-based outcome measurement.

Audience: People with some experience in outcome measurement, including background/experience in logic modeling/outcome paths/theory of change, who want to increase their skills.

Ellen Taylor-Powell has over 25 years’ experience in evaluation and program development with a focus on capacity building in the public and nonprofit sectors. She trains widely on logic modeling and all aspects of evaluation pertinent to community-based program performance. As Distinguished Evaluation Specialist at the University of Wisconsin-Extension, she authored or co-authored numerous practical evaluation resources including the online logic model course (Enhancing Program Performance with Logic Models) and the manuals, Evaluating Collaboratives and Building Capacity in Evaluating Outcomes. Ellen spent over a decade working in international settings and currently is an independent consultant. She served on the American Evaluation Association Board and past chair of the Extension Evaluator's TIG. Ellen holds a M.S. and Ph.D. from Cornell University (program evaluation focus).

Offered:

• Sunday, June 12, 9:00 - 4:00 (1 hour break for lunch within)

PI2: Introduction to Evaluation

Level: Advanced Beginner

Description: This workshop will provide an overview of program evaluation for Institute participants with some, but not extensive, prior background in program evaluation. The session will be organized around the Centers for Disease Control and Prevention’s (CDC) six-step Framework for Program Evaluation in Public Health as well as the four sets of evaluation standards from the Joint Commission on Evaluation Standards. The six steps constitute a comprehensive approach to evaluation. While its origins are in the public health sector, the Framework approach can guide any evaluation. The course will touch on all six steps, but particular emphasis will be put on the early steps, including identification and engagement of stakeholders, creation of logic models, and selecting/focusing evaluation questions. Several case studies will be used both as illustrations and as an opportunity for participants to apply the content of the course and work through some of the trade-offs and challenges inherent in program evaluation in public health and human services.

Audience: Attendees with some background in evaluation, but who desire an overview and an opportunity to examine challenges and approaches. Cases will be from public health but general enough to yield information applicable to any other setting or sector.

Thomas Chapel is a Senior Evaluation Scientist in the Office of Workforce and Career Development, at the Centers for Disease Control and Prevention. He serves as a central resource on strategic planning and program evaluation for CDC programs and their partners. Before joining CDC, Tom was Vice-President of the Atlanta office of Macro International where he directed and managed projects in program evaluation, strategic planning, and evaluation design for public and nonprofit organizations. He is a frequent presenter at national meetings, a frequent contributor to edited volumes and monographs on evaluation, and has facilitated or served on numerous expert panels on public health and evaluation topics.

Offered:

• Sunday, June 12, 9:00 - 4:00 (1 hour break for lunch within)

NEW! PI3: An Introduction to Participatory Monitoring and Evaluation: An Approach to Engaging Stakeholders in M&E

Level: Advanced Beginner

Description: This session will demonstrate how to integrate participatory monitoring and evaluation (PM&E) approaches into conventional evaluation activities utilizing participatory tools, techniques, methods, strategies and processes. The presentation is highly interactive and participants will have the opportunity to practice developing commonly used participatory tools from categories including: mapping, diagramming, ranking & classification, processes & change, and structured observation, and work in small groups to develop a sample PM&E plan. You will also learn how this approach can be used to engage stakeholders effectively, and to build capacity of local communities. Come prepared to participate and comfortably dressed!

Audience: Those working in state and local health departments, community-based organizations, marginalized communities, and/or with vulnerable populations both domestically and in developing countries, with responsibilities for designing, developing, implementing, or evaluating community development interventions or community based programs.

Arlene Vincent-Mark has served as a Social Scientist/Senior Service Fellow, and Public Health Advisor with CDC. She has worked in social and economic development, business management and policy development for more than 20 years. With CDC, she has has provided technical assistance to state health departments and community-based organizations in areas relating to: Bioterrorism Preparedness and Emergency Response programs, Pandemic Flu Preparedness, and HIV/AIDS. She has used participatory approach frameworks in projects including: drug adherence and quality of life studies in HIV/AIDS, maternal and child health, education and agricultural development. Arlene holds a PhD and a Master of Arts in International Affairs and Development, a Master of Business Administration Degree, and a Bachelor of Arts Degree in Economics.

Offered:

• Sunday, June 12, 9:00 - 4:00 (1 hour break for lunch within)

SCHEDULE OVERVIEW ¨ WORKSHOPS ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS

SCHEDULE OVERVIEW ¨ WORKSHOPS ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS

Monday, June 13, Keynote 1: 10 Steps to Data Vizardry - Making People Fall in Love with your Data

People think visually. However, most of the tools that evaluators, data analysts, and statisticians use actually confuse our natural visual interpretation skills. Today, we will share practical guidance that makes data presentation more effective, engaging, and valuable. Some of the topics we’ll be discussing include research-based information about improving layout and workflow, information visualization, chart selection, and styling. We’ll show real-life examples to demonstrate how applying these best practices enable charting laymen and experts alike to work like visualization ninjas.

Zach Gemignani is co-founder and CEO of Juice Analytics. His focus on making client’s data more usable and valuable has helped Juice provide solutions with real benefit. His writings and speaking engagements have earned him a tremendous reputation in the Data Visualization arena. Before Zach helped found Juice, he led efforts to bring analytical rigor to AOL and was a consultant with Diamond Consultants and Booz Allen.

Wednesday, June 15, Keynote 2: Wiggling in or Sliding by: Quality and the “Good Enough” Rule with Leslie Cooksy

Rossi and Freeman (1989) advocated using the “good enough” rule in evaluation design, choosing “the best possible design, given the potential importance of the program and considerations of practicality and feasibility” (p. 232). But Rossi later said that he feared that the good enough rule may be misused to excuse sloppy work. In this talk, Cooksy will ask how we know what “good enough” is, and suggest ways that even the just-barely-good enough evaluation can make a meaningful difference.

SCHEDULE OVERVIEW ¨ WORKSHOPS ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS

CONCURRENT SESSION DESCRIPTIONS

Offering 1: Conducting and Using Success Stories for Capacity Building

Level: Intermediate

Description: In order to build program capacity, a programs "success" must be told at many levels. In addition, impacts of prevention programs may not be able to be demonstrated for several years therefore communicating success during the various life stages of a program is important for long term sustainability. The presenter will use her experience with various public health programs to demonstrate how to use success stories to build both program and evaluation capacity. The session will be a practical and hands on session enabling attendees to begin writing their own success stories. This session includes time for practicing practical applications for use in your own practice. Attendees will receive the workbook: Impact and Value: Telling Your Program’s Story for use during the class and to take home for reference.

Audience: Attendees working in any context with a working knowledge of both evaluation and qualitative inquiry

René Lavinghouze is a Senior Evaluation Scientist in the Office of Smoking and Health at CDC where she is the lead on the evaluation of recovery act programs. Rene has over 21 years experience with CDC and in the private sector. She is Chair of AEA’s TIG for Cluster, Multi-site/level evaluations.

Offered:

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

Offering 3: Strengthening Evaluation Through Cultural Relevance and Cultural Competence

Level: Intermediate

Description: This skill-building session addresses the centrality of culture in evaluation. It is organized in two segments. The opening segment addresses the relevance of culture to all stages of the evaluation process, to the fundamental validity of our work as evaluators, and to ethical standards and guidelines of our profession. Presenters will use an FAQ format to raise questions and address common misconceptions that marginalize discussions of culture within the evaluation community (e.g., Is “culture” really just a code-word for “race”? How does culture apply to me as a white evaluator working within predominantly white populations? What is the “value added” of culture in evaluation? Why should I care?) The second segment extends cultural relevance to present strategies for building cultural competence through experience, education and self-awareness. Theoretical frameworks that situate culture in evaluation (e.g., Frierson, Hood & Hughes, 2002; Hall & Hood, 2005; Kirkhart, 2005) are presented as advance organizers for practice and application purposes. Presenters use case scenarios and participants’ own examples to integrate workshop content with participants’ field experience, interests, and concerns. They rely on various theoretical frameworks to guide the two segments in tangible and practical ways. Additional resources are provided to extend and reinforce participant learning.

Audience: Attendees working in any context with a basic background in evaluation.

Rodney K. Hopson has undergraduate and graduate degrees in English Literature, Educational Evaluation, and Linguistics from the University of Virginia, and he is Hillman Distinguished Professor, Department of Educational Foundations and Leadership and faculty member in the Center for Interpretive and Qualitative Research at Duquesne University. Karen and Rodney have served in positions of leadership within the American Evaluation Association, and both are actively involved in education and scholarship on culture, diversity, and social justice in evaluation. Rodney serves as Co-Project Director for the Robert Wood Johnson Foundation Evaluation Fellowship Program. Karen is a member of the AEA Multicultural Task Force and the Diversity Committee task force charged with developing a public statement on the subject of cultural competence and evaluation. Karen E. Kirkhart holds a Ph.D. in Social Work and Psychology from The University of Michigan and is currently Professor, School of Social Work, College of Human Ecology, Syracuse University.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

Offering 4: Focus Group Research: Understanding, Designing and Implementing

Level: All

Description: As a qualitative research method, focus groups are an important tool to help researchers understand the motivators and determinants of a given behavior. This course provides a practical introduction to focus group research. At the completion of this course, participants will be able to 1) identify and discuss critical decisions in designing a focus group study, 2) understand how research or study questions influence decisions regarding segmentation, recruitment, and screening; and, 3) identify and discuss different types of analytical strategies and focus group reports.

Audience: Attendees working in any context who are new to focus group facilitation

Michelle Revels is a technical director at ORC Macro specializing in focus group research and program evaluation. Ms. Revels attended Hampshire College in Amherst, Massachusetts and the Hubert H. Humphrey Institute of Public Affairs at the University of Minnesota.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

Offering 5: Enhanced Group Facilitation: Techniques and Process

Level: All

Description: This popular and well-received workshop will familiarize participants with the management of the group facilitation process and a variety of group facilitation techniques. Participants will learn best practices for navigating group dynamics and capitalizing on challenging attendees. We will discuss facilitator’s different roles and responsibilities in group facilitation and how these roles intersect with the tasks inherent in planning and managing a group facilitation experience. Participants will also learn how to choose a facilitation technique based on goals and objectives, anticipated outcome, type and number of participants, and logistics. Two to three facilitation techniques for generating ideas, focusing thoughts, and drawing conclusions and next steps will be explored in greater detail. We will also cover variations on these techniques and how they may be used for your facilitation purposes. Job aides and reference lists will be provided.

Audience: Attendees working in any context who work with, or expect to be working with, client groups of any size.

Level: All

Jennifer Dewey is a Senior Research Associate with James Bell Associates, Inc. She is the Project Director for the Family Connection Discretionary Grants evaluation, funded by the Administration for Children and Families (ACF), Children’s Bureau, where she leads a team of JBA staff in designing and implementing a cross-site evaluation protocol, along with providing technical assistance to local evaluation activities. Prior to joining JBA, Dr. Dewey was a Technical Director at ICF Macro, where in addition to business and organizational development, she directed three core studies and oversaw ongoing training and technical assistance to 60+ local evaluation teams for a multi-site evaluation of a SAMHSA-funded systems of care program. Her technical expertise encompasses: project management; proposal development; evaluation design and budgeting; needs assessment; field and survey research; telephone and in-person interviews; group facilitation and training; quantitative and qualitative data collection and analysis; and evaluation reporting.

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within)

NEW! Offering 6: Building Evaluation Capacity: Tips and Strategies (Taylor-Powell)

Level: Intermediate

Description: Are you trying to build individual, team, program, and/or organizational capacity in evaluation yet are uncertain or stymied with what to do and how to do it?  This workshop will draw from the US and international literature on evaluation capacity building as well as the presenter’s own experience in public sector organizations and work with community-based nonprofits.  Key components of evaluation capacity building will be discussed with tips and strategies for each component that promote individual and organizational change.  Bring your own ideas and strategies to share as we explore what works and what doesn’t in a participatory, interactive format.

Ellen Taylor-Powell has over 25 years’ experience in evaluation and program development with a focus on capacity building in the public and nonprofit sectors. She trains widely on logic modeling and all aspects of evaluation pertinent to community-based program performance. As Distinguished Evaluation Specialist at the University of Wisconsin-Extension, she authored or co-authored numerous practical evaluation resources including the online logic model course (Enhancing Program Performance with Logic Models) and the manuals, Evaluating Collaboratives and Building Capacity in Evaluating Outcomes. Ellen spent over a decade working in international settings and currently is an independent consultant. She served on the American Evaluation Association Board and past chair of the Extension Evaluator's TIG. Ellen holds a M.S. and Ph.D. from Cornell University (program evaluation focus).

Offered:

• Monday, June 13, 9:25 – 12:45 (20 minute break within) FULL

Offering 7: Every Picture Tells a Story: Flow Charts, Logic Models, LogFrames, Etc. What They Are and When to Use Them

Level: Advanced Beginner

Description: A host of visual aids are in use in planning and evaluation. This session will introduce you to some of the most popular ones—with an emphasis on flow charts, logic models, project network diagrams, and logframes. We’ll review the content and format of each one and then compare and contrast their uses so that you can better match specific tools to specific program needs. We’ll review simple ways to construct each type of tool and work through some simple cases both as illustrations and as a way for you to practice the principles presented in the session.

Audience: Assumes prior familiarity with evaluation terminology and some experience in constructing logic models.

Thomas Chapel is a Senior Evaluation Scientist in the Office of Workforce and Career Development, at the Centers for Disease Control and Prevention. He serves as a central resource on strategic planning and program evaluation for CDC programs and their partners. Before joining CDC, Tom was Vice-President of the Atlanta office of Macro International where he directed and managed projects in program evaluation, strategic planning, and evaluation design for public and nonprofit organizations. He is a frequent presenter at national meetings, a frequent contributor to edited volumes and monographs on evaluation, and has facilitated or served on numerous expert panels on public health and evaluation topics.

Offered:

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 8: Project Management Fundamentals – The Key to Implementing Evaluation Projects

Level: Intermediate

Description: This highly experiential session is focused on how to implement an evaluation effort through a structured project management methodology. The workshop will focus on initiating, planning, executing, and monitoring and controlling an evaluation project. Participants will gain an understanding of how the project management process can be used effectively to add value to evaluation projects, and will also gain an understanding of how to generate a project plan and manage an evaluation project. At the conclusion of the session, participants will be able to:

• Describe the documents that define an evaluation “project plan”

• Write a project charter (mission statement) for an evaluation

• Develop a Project scope statement for an evaluation project

• Interpret a graphic picture of a project via a network diagram and Gantt chart

• Generate a project schedule and describe the project critical path

Audience: Participants with responsibilities for evaluation implementation, as well as individuals with oversight responsibilities for evaluation projects. Participants already participating in program evaluation will benefit most from the session.

Richard H. Deane has over 30 years experience in the application of structured project management techniques in both the private and public health sectors. He has taught and won numerous teaching awards at Purdue University, Georgia Tech and Georgia State University. Richard has been a highly acclaimed instructor at the CDC University for many years, and his project management consulting assignments over the past 25 years have included work with various agencies within t?he CDC, state public health departments, private health organizations, federal agencies, and numerous private sector corporations.

Offered:

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

Offering 9: Improving Survey Quality: Assessing and Increasing Survey Reliability

Level: Intermediate

Description: Develop higher-quality surveys! This workshop is designed to teach participants how to improve survey reliability, thus increasing their utility of and confidence in the data they collect. We will look at surveys to elicit factual information as well as ones that ask about subjective and abstract concepts. Through the use of hands-on activities, mini-lectures, and demonstrations participants will understand what is meant by reliability with respect to surveys and will learn ways to improve it during the survey design phase for both types of surveys. Next, using a case example and SPSS we will explore ways to use pilot test responses to assess the reliability of subjective / abstract survey constructs by conducting confirmatory factor analysis and calculating Cronbach’s alpha. We will work together to understand what our findings tell us as well as what they don’t tell us. You will receive a workbook and SPSS screenshots to help you remember how to perform many of the computations we will perform. Participants will be surprised by how easy it is to improve survey quality through a few easy to implement steps!

Audience: Attendees working in any context with a basic background in survey development and an understanding of factor analysis.

Amy A. Germuth earned her PhD in Evaluation, Measurement and Psychology from UNC Chapel Hill and a Certificate in Survey Methodology via a joint program through UNC-CH, UMD, Westat and RTI. She is a founder of EvalWorks, LLC a survey and evaluation consulting firm that conducts evaluations at the local, state, and national levels. As part of her work she has evaluated numerous initiatives, including health prevention and outreach programs, Math Science Partnerships, K-12 science outreach programs, and workforce development initiatives, and has worked with a variety of organizations including federal and state government, independent firms, and universities. Amy teaches evaluation and instrument development as part of Duke University’s Certificate Program in Nonprofit Management

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 10: Theory-Driven Evaluation for Assessing Planning, Implementation, and Effectiveness

Level: Intermediate

Description: Learn the theory-driven approach for assessing and improving program planning, implementation and effectiveness. You will explore the conceptual framework of program theory and its structure, which facilitates precise communication between evaluators and stakeholders regarding evaluation needs and approaches to address those needs. Mini-lectures, group exercises and case studies will be used to illustrate the use of program theory and theory-driven evaluation for program planning, initial implementation, mature implementation and outcomes. In addition, the participants will learn principles and strategies for using the theory-driven approach to deal with the following cutting edge issues: how to go beyond traditional methodology for designing a real world evaluation, how to achieve both internal and external validity in an evaluation, and how to use program theory for guiding the application of mixed methods in an evaluation.

Audience: Attendees with a basic background in logic models and/or program theory.

Huey Chen is a senior evaluation scientist at the CDC. He was a professor at the School of Public Health at the University of Alabama at Birmingham until January 2008. Dr. Chen has contributed to the development of evaluation theory and methodology, especially in the areas of program theory, theory-driven evaluations, and evaluation taxonomy. His book Theory-Driven Evaluations has been recognized as one of the landmarks in program evaluation and his newest text, Practical Program Evaluation, offers an accessible approach to evaluation for those working in any context. In 1993 he received the AEA Paul F. Lazarsfeld Award for contributions to Evaluation Theory and in 1998 he received the CDC Senior Biomedical Research Service Award.

Offered:

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

Offering 11: Case Study Methods for Evaluators

Level: Beginner

Description: Case Study Methods allow evaluators to approach program assessment from a powerful and flexible design palette. While often heavily steeped in the use of qualitative methods, case studies also may include the use of quantitative data. The approach is particularly rich for tinting and shading the effects of programs as well as investigating important program questions in depth.

This interactive, three-hour session will provide participants with an overview and examples of case study research methods as they apply to evaluation settings. Through the development and expansion of sample case studies, by the end of the session, participants will:

• Comprehend the role of case study methods within the context of other evaluation approaches

• Be able to describe the elements of case study research and identify the major strengths and weaknesses of case study methods;

• Understand the sequential, operational guidelines for implementing case study research

• Review techniques for establishing the validity and reliability of case study data

• Strengthen data gathering and analysis skills through use of techniques common to case study research

Audience: Attendees working in any context

Rita O’Sullivan is Associate Professor of Evaluation and Assessment at the University of North Carolina at Chapel Hill where she teaches graduate courses in Educational Program Evaluation, Case Study Methods, Research Design, Measurement, and Statistics. She is also Executive Director of Evaluation, Assessment, and Policy Connections (EvAP), a unit she founded within the UNC School of Education that conducts local, state, and national evaluations. Dr. O’Sullivan has specialized in developing collaborative evaluation techniques that enhance evaluation capacity and utilization among educators and public service providers. She is senior author of Programs for At-Risk Student: A Guide to Evaluation (Corwin Press, 1993) and wrote Practicing Evaluation: A Collaborative Approach (Sage) in 2004.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

Offering 12: An Introduction to Economic Evaluation

Level: Intermediate

Description: Economic evaluation refers to applied analytic methods used to identify, measure, value, and compare the costs and consequences of programs and interventions. This course provides an overview of these methods, including cost analysis, cost-effectiveness analysis (CEA), and cost-benefit analysis (CBA) with an opportunity for hands-on application of each. You will leave understanding when and how to apply each method appropriately in a range of evaluation contexts.

Audience: Attendees with a basic background in statistical methods and understanding of evaluation.

Phaedra S. Corso is currently an Associate Professor in the Department of Health Policy and Management at the University of Georgia College of Public Health. Previously, she served as the senior health economist in the National Center for Injury Prevention and Control at CDC where she worked for over a decade in the areas of economic evaluation and decision analysis, publishing numerous articles on the cost-effectiveness of prevention interventions and co-editing a book on prevention effectiveness methods in public health. She holds a Master’s degree in public finance from the University of Georgia and a Ph.D. in health policy and decision sciences from Harvard University.

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within) FULL

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 13: (Dawkins) Evaluability Assessments: Achieving Better Evaluations

Level: Beginner

Description: Though rigorous evaluation is a valuable method, in practice it also proves costly and time-consuming. Further, rigorous evaluation is not an appropriate fit for every initiative. Evaluability assessments (EAs) offer a cost-effective technique to help guide evaluation choices. Developed by Joseph Wholey and colleagues three decades ago (1979), EA received significant attention at the time but subsequently displayed diminished activity (Rog 1985; Trevisan 2007). But EAs help answer critical questions evaluators continue to face in practice: Is a program or policy ready for rigorous evaluation? What are viable options for evaluating a particular initiative? EAs involve clarifying program goals and design, finding out stakeholders’ views on important issues, and exploring the reality of a given initiative. In short, EAs are a valuable and important tool to have in an evaluator’s toolbox. This workshop will provide participants an understanding of EAs and how they can be applied in their own practice.

Audience: Those working in any context.

Nicola Dawkins received her PhD in sociology and Masters in Public Health from Emory University. She serves as a Principal at ICF Macro where she designs and implements numerous research and evaluation studies. Among these is the Coordinating Center for the Early Assessment of Programs and Policies to Prevent Childhood Obesity—a project that employed a cluster evaluability assessment methodology to examine multiple initiatives. Nicola oversees other individual evaluability assessment and evaluation projects as well.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

Offering 14: Handling Data: From Logic Model to Final Report

Description: Collect, analyze, and present data from complex evaluation studies in ways that are feasible for the evaluator and meaningful to the client. Explore lessons learned through over twenty years in evaluation consulting to ask the right questions, collect the right data and analyze and present findings in simple yet comprehensive ways.

We will use actual data samples along with examples of analysis techniques. You will have an opportunity to work in small groups with sample data and will explore various analysis techniques. Throughout the workshop, the presenter will respond to individual questions and facilitate group discussion on data handling topics. At the end of the workshop, you will take away fresh ideas to tackle your data handling challenges.

You will learn:

• To develop and link a program theory, a holistic logic model, a data collection matrix, and evaluation tools,

• To ask the right questions and get the answers you need,

• To develop a data summary that triangulates the information collected from different sources,

• To extract and map themes, prepare an evidence table, and report findings in a comprehensive but user-friendly way.

Audience: Attendees with a basic background in evaluation.

Gail V. Barrington is an independent consultant who started her consulting firm, Barrington Research Group, Inc. in 1985. She has conducted over 100 program evaluation studies and has made a significant contribution to the field of evaluation through her practice, writing, teaching, training, mentoring and service. In 2008, she won the Canadian Evaluation Society award for her Contribution to Evaluation in Canada.

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 15: Dissemination and Implementation Research

Level: Intermediate

Description: We will be focusing on the evaluating efforts associated with the process of moving from pilot or demonstration projects into widespread practice.   Issues associated with this process include the following:

• Packaging the program for potential users
• Marketing/disseminating  the program to the potential users
• Building capacity to select and use effective interventions
• Understanding the decisions involved in adopting a new program
• Maintaining quality control over multiple implementation sites
• Facilitating and measuring implementation
• Considering the balance between fidelity and adaptation
• Scaling up efforts to achieve public health impact

Each of these issues is a possible focus for evaluation research questions.  Among these are the following questions are the following:

• What characteristics of the message, its delivery agent, modality, and its recipient lead to the most successful dissemination efforts?
• What characteristics of programs, organizations, their mutual contexts, and resources affect the decision to adopt a program?
• What characteristics of programs, organizations, their mutual context and resources affect effective implementation (and related constructs such as fidelity and adaptation)?
• How do the processes and level of implementation act as moderators and/or mediators in understanding program outcomes

Audience: Attendees with a basic background in evaluation.

James Emshoff is an Associate Professor Emeritus and former Director of the Community Psychology Program at Georgia State University. He also founded and serves as Director of Research at EMSTAR Research, Inc., an evaluation and organizational services firm. Dr. Emshoff has conducted evaluation research in multiple contexts at the local, state, and national levels. Rita K. Noonan is a team leader in CDC’s Division of Unintentional Injury Prevention. Dr. Noonan has published on a variety of topics including: quality of life outcomes for women and children; the international debt crisis; dating and sexual violence prevention; empowerment evaluation, older adult fall prevention; and translational research. Richard W. Puddy joined CDC’s Division of Violence Prevention in March 2007. Richard has over 20 years experience working in the field of prevention across all levels of the social ecology to prevent child maltreatment, suicide, youth violence, intimate partner violence, and sexual violence. He is the Branch Chief of the Program Implementation and Dissemination Branch (PIDB) at CDC.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

Offering 16: Systems Level Evaluation of Communities of Practice

Level: Intermediate

Description: In an environment of increasing social participation and transparency, communities of practice are one means to unite a variety of partners to address common issues, share resources, and learn new information. When asked to design an evaluation of this type of complex social initiative, evaluators increasingly turn to system level evaluation. One means to frame a system level evaluation is the use of social and behavioral science theory. Both implicit and explicit use of theory will be covered, including theory of change as an explicit approach that links activities, outcomes, and contexts in a way that maximizes the attribution of interventions to outcomes. This workshop will use lecture, exercises and discussion to improve attendees’ ability to understand the application of a systems level evaluation to communities of practice as well as how to design evaluations of such complex social initiatives.

You will learn:

• What communities of practice are and how they are used

• How to view communities of practice within a systems-level framework

• How to use theory, including theory of change to evaluate communities of practice and their impacts

Audience: Attendees with a basic understanding of systems approaches who would like to improve their evaluation of large-scale social programs

Andrea M. Hegedus, is a Health Research Analyst for Northrop Grumman Corporation. She has over 25 years of experience including evaluation of behavioral healthcare programs and other complex social initiatives. Jan C. Jernigan, is a Senior Evaluator in the Division of Nutrition, Physical Activity and Obesity at CDC. She has taught program evaluation courses and conducted program evaluation and evaluation research for over 15 years. Drs. Hegedus and Jernigan both received their doctorates from the Graduate School of Public and International Affairs at the University of Pittsburgh with a specialty in public policy research and analysis.

Offered:

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within) FULL

Offering 17: Transformative Mixed Methods Evaluations

Level: Beginner

This workshop focuses on the methodological and contextual considerations in designing and conducting transformative mixed methods evaluation. It is geared to meet the needs of evaluators working in communities that reflect diversity in terms of culture, race/ethnicity, religion, language, gender, and disability. Deficit perspectives that are taken as common wisdom can have a deleterious effect on both the design of a program and the outcomes of that program. A transformative mixed methods approach enhances an evaluator's ability to design and conduct evaluations with a greater potential for social change.

Theoretical perspectives and their methodological implications will be presented through mini-lectures, small- and large-group discussions, and opportunities to apply the transformative principles to your own evaluations. Alternative strategies based on transformative mixed methods are illustrated through reference to the presenters' own work, the work of others, and the challenges that participants bring to the workshop.

You will learn:

• To critically examine the transformative paradigm's assumptions in culturally diverse communities,

• To identify different methodological approaches within a transformative mixed methods model,

• To apply critical skills associated with selecting the design and use of transformative mixed methods evaluation.

Donna Mertens is a Past President of the American Evaluation Association, conducts national and international evaluations, and teaches program evaluation to graduate students at Gallaudet University in Washington, DC. Donna recently authored Transformative Research and Evaluation (Guilford). Katrina L Bledsoe is a current member of the AEA board, and a research scientist/evaluation specialist at the Education Development Center in Washington, DC. Katrina conducts evaluations in culturally complex communities domestically and internationally.

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 18: Essential Competencies for Program Evaluators

Level: All

Description: This workshop is designed to teach participants the Essential Competencies for Program Evaluators, a set of knowledge, skills, and attitudes in six categories. The session will begin with the analysis of program evaluation vignettes representing diverse areas of practice to show both the common competencies across settings and those unique to specific contents or contexts. Following a brief history of how the competencies were developed, the session will then examine the competencies in all six categories: professional practice, systematic inquiry, situational analysis, project management, reflective practice, and interpersonal skills. This discussion, which builds on the continuum of interpersonal evaluation practice, will ground participants in the competencies’ content and allow people to ask questions as they think about their own evaluation practice. After a short break, participants will develop concept maps to explore how the competencies make sense in their roles or content areas. Comparative discussion will further illuminate the competencies, and then participants will complete a self-assessment tool and discuss how to set priorities and action steps for professional development. Most of the session will consist of interactive exercises with just enough lecture to frame the discussion.
Audience: All evaluators, and those thinking about entering the field of evaluation, working in any context

Jean King is Professor and Director of Graduate Studies in the Department of Organizational Leadership, Policy and Development at the University of Minnesota where she also serves as Director of the Minnesota Evaluation Studies Institute.  A sought-after presenter and long-time writer on evaluation topics, she is the author of numerous articles, chapters, and reviews. Jean has received numerous awards for her work, including AEA’s Myrdahl Award for Evaluation Practice and Ingle Award for Extraordinary Service, three teaching awards, and three community service awards.  From 2004-2009 she served as Co-PI with Frances Lawrenz on “Beyond Evaluation Use: Determining the Effect of Individual Project Participation on the Influence of STEM Overall Program Evaluations,” an NSF-funded research project. 

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

Offering 19: Evaluating and Improving Organizational Collaboration

Level: Intermediate

Description: Program personnel and evaluators have a unique responsibility and opportunity to engage in the systematic assessment and improvement of organizational collaboration, which has become the most widely championed vehicle for precipitating efficiency, innovation and essential organizational outcomes in the 21st century. This workshop aims to increase participants’ capacity to quantitatively and qualitatively examine the development of strategic alliances and inter-professional collaboration using the Collaboration Evaluation Improvement Framework (CEIF). Together, participants will apply CEIF associated data collection, analysis, and reporting techniques to their own contexts; such methods and approaches are currently being used in the formative and developmental evaluation of grant-funded health care coalitions, medical and dental programming, and other collaborative endeavors including the federally sponsored K-16 Safe Schools/Healthy Students initiative.

Audience: Attendees with a basic understanding of organizational change theory/systems theory and familiarity with mixed methodological designs

Rebecca Woodland has been a facilitator of workshops and courses for adult learners for more than 10 years. She was a top-10 workshop presenter at Evaluation 2008 & 2009, lauded for her hands-on, accessible, and immediately useful content. As a principal investigator, she is responsible for the evaluation and development of organizational collaboration for an array of small and large-scale, grant-funded organizational improvement initiatives. Her most recent publications on the topic of organizational collaboration have been published in the American Journal of Evaluation (Gajda, 2004; Gajda & Koliba, 2008), Journal of Nursing Regulation (Anderson, et. al. in press) and the International Journal of Public Administration (Koliba & Gajda, 2009). Rebecca is currently an Associate Professor of Educational Leadership at the University of Massachusetts Amherst

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 20: Logic Models as a Platform for Program Evaluation Planning, Implementation, and Use of Findings

Level: Beginner

Description: Practitioners use logic models to describe important components of a program; make visible a theory of change; and link activities to intended outcomes. For the purposes of evaluation practice, a well-constructed logic model provides a program-specific foundation for gaining clarity with stakeholders, identifying key evaluation questions; prioritizing data needs; and translating findings into recommendations for ongoing program improvement. Despite these obvious benefits, many program implementers and evaluators have been turned off to logic models because of overly complex approaches that get mired in complicated formatting rules or semantic wrangling. Aimed directly at recapturing the utility of even basic logic models, the workshop will show how to construct simple logic models, how to extract important insights about your program from a simple logic model, introduce the many terms in play in logic models, but with attention to the added-value each of these adds to your understanding of the program, and illustrated how these models make it easier to tackle key tasks in planning and evaluation.

Workshop Objectives:

• Demystify the process of developing logic models

• Demonstrate the added value of logic models for evaluation and planning

• Present a simple approach for creating basic logic models

• Identify the added-value of elaborating a basic logic model and demonstrate how to choose the appropriate format and level of complexity.

• Demonstrate use of logic models in making evaluation and planning choices.

Audience: Attendees who implement or evaluate programs and who are either new to logic modeling or have been traumatized by prior experiences with logic modeling!

Thomas Chapel is a Senior Evaluation Scientist in the Office of Workforce and Career Development, at the Centers for Disease Control and Prevention. He serves as a central resource on strategic planning and program evaluation for CDC programs and their partners. Before joining CDC, Tom was Vice-President of the Atlanta office of Macro International where he directed and managed projects in program evaluation, strategic planning, and evaluation design for public and nonprofit organizations. He is a frequent presenter at national meetings, a frequent contributor to edited volumes and monographs on evaluation, and has facilitated or served on numerous expert panels on public health and evaluation topics.

Offered:

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

Offering 21: Impact Evaluation Design
 

Level: Intermediate

Description: Impact is the positive and negative long-term effects produced by an intervention. Measuring impact requires attention to culture, context, contributors, and causal mechanisms, and the ways in which their effects manifest over time. Appropriate impact evaluation design requires situational responsiveness - matching the design to the needs, constraints and opportunities of the particular case. The design must reflect the nature of the intervention and the purposes of the impact evaluation. In particular, impact evaluation needs to address simple, complicated and complex aspects of the intervention. Different designs are recommended for each case, including randomized controlled trials, regression discontinuity, unstructured community interviews, Participatory Performance Story Reporting, and developmental evaluation.

 

Audience: Evaluators practicing in any context who have an understanding of basic evaluation principles in practice

Patricia Rogers has worked in public sector evaluation and research for more than 25 years, across a wide range of programs, including health, early childhood, education, community development, Indigenous housing, criminal justice, international development, and agriculture. She is a contributor to the International Initiative for Impact Evaluation's working paper series, and is currently working on the development of 'BetterEvaluation' - an online resource and community of practice on evaluation methods. A sought-after and experienced facilitator, Patricia has offered workshops for the American Evaluation Association, the Evaluators Institute, and the Australasian Evaluation Society

 

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within) FULL

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 22: Ensuring Evaluation Use
 

Level: Beginner

Description: Many evaluators and program staff are concerned with designing evaluations that are intended to inform more immediate decision-making and promote organizational change. A goal, then, is to offer the most useful information to answer the evaluation questions given the program context and the resources available to conduct the investigation. This session will focus on developing participants’ understanding of, and methods for increasing, evaluation use. We will begin by providing a theoretical framework for understanding and promoting evaluation use. Employing interactive and small group exercises, we will examine strategies and techniques for increasing the use of both the evaluation process and findings. Upon completion of the session, you will: understand use as a primary purpose of evaluation, understand the difference between evaluation process and evaluation findings use, become familiar with a framework for promoting use, and identify strategies for promoting and increasing evaluation use.
 

Audience: Those who are new to evaluation, including stakeholders in the evaluation design process, who are working in any context.

Christina A. Christie is an Associate Professor at the UCLA Graduate School of Education & Information Studies. Her research and practical work has three main foci: applied evaluation research studies, research on evaluation practice, and theoretical analysis. Tina is co-founded the Southern California Evaluation Association, a local affiliate of the American Evaluation Association, and is on the AEA Board of Directors. In 2004, she received the American Evaluation Association’s Marcia Guttentag Early Career Achievement Award.

 

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 23: Using Theory to Improve Evaluation Practice
 

Level: Beginner

Description: This workshop is designed to provide evaluators with an opportunity to improve their understanding of how to use theory to improve evaluation practice. We'll examine social science theory and stakeholder theories, including theories of change and their application to making real improvements in how evaluations are framed and conducted. Lecture, exercises, and discussions will help participants learn how to apply evaluation theories, social science theories, and stakeholder theories of change to improve the accuracy and usefulness of evaluations. A wide range of examples from evaluation practice will be provided to illustrate main points and key take-home messages.

You will learn:

• To define and describe evaluation theory, social science theory, and program theory;

• How evaluation theory can be used to improve evaluation practice;

• How implicit and explicit social science theories can be used to guide evaluation decisions;

• The components and processes of several commonly used social science theories that have been used to develop and evaluate interventions;

• How developing stakeholder theories of change can be used to improve evaluation practice.

Stewart Donaldson is Dean of the School of Behavioral and Organizational Sciences at Claremont Graduate University. He has published widely on the topic of applying program theory, developed one of the largest university-based evaluation training programs, and has conducted theory-driven evaluations for more than 100 organizations during the past decade.
Audience: Evaluators working in any context.

 

Offered:

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 24: Qualitative Interviewing: Asking the Right Questions in the Right Way
 

Level: Intermediate

Description: Preparing a proper interview guide is only a first step to proper Q & A in a qualitative data collection episode. This session outlines key sections to include in an interview guide and offers suggestions for how to conduct a qualitative interview and/or focus group. The face-to-face interaction in this case is critical. The interviewer must balance attention to the questions designed for the interaction and the emergent topics in the interview. Core skills that focus attention on the audience for the study, the topics of the project, important lines of questioning and goals for ensuring quality interaction in this relationship improve the quality of data collection.

Audience: Researchers in any discipline with a basic knowledge of qualitative analysis who are interested in using conversational techniques in the form of interviews or focus groups
 

Raymond C. Maietta  is president of ResearchTalk Inc., a qualitative research consulting company. A sociologist from the State University of New York at Stony Brook, Ray's interests in the art of qualitative research methods motivated him to start ResearchTalk in 1996. ResearchTalk Inc. provides advice on all phases of qualitative analysis to university, government, not-for-profit and corporate researchers. Work with ResearchTalk clients using qualitative software informs recent book chapters: Systematic Procedures of Inquiry and Computer Data Analysis Software for Qualitative Research (with John Creswell) and State of the Art: Integrating Software with Qualitative Analysis. More than 15 years of consultation with qualitative researchers informs the methods book Ray is writing. Sort and Sift, Think and Shift will be completed in 2012.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 14, 9:25 – 12:45 (20 minute break within)

Offering 25: Developmental Evaluation
 

Level: Beginner

Description: Developmental evaluation (DE) is especially appropriate for innovative initiatives or organizations in dynamic and complex environments where participants, conditions, interventions, and context are turbulent, pathways for achieving desired outcomes are uncertain, and conflicts about what to do are high. DE supports reality-testing, innovation, and adaptation in complex dynamic systems where relationships among critical elements are nonlinear and emergent. Evaluation use in such environments focuses on continuous and ongoing adaptation, intensive reflective practice, and rapid, real-time feedback. The purpose of DE is to help develop and adapt the intervention (different from improving a model). This evaluation approach involves partnering relationships between social innovators and evaluators in which the evaluator’s role focuses on helping innovators embed evaluative thinking into their decision-making processes as part of their ongoing design and implementation initiatives. DE can apply to any complex change effort anywhere in the world. Through lecture, discussion, and small-group practice exercises, this workshop will explore the basics of DE and position DE as an important option for evaluation in contrast to formative and summative evaluations as well as other approaches to evaluation.
 

Audience: Evaluators working in any context.

 

Michael Quinn Patton is an independent consultant and professor at the Union Institute. An internationally known expert on Utilization-focused Evaluation, this workshop is based on his just published new book, Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use (Guilford, 2010).

Offered:

• Wednesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 26: Unraveling the Puzzle: Performance Management, Quality Improvement, and Evaluation
 

Level: Intermediate

Description: Recently, there has been a keen focus on public health organizational performance and accountability. Performance Management, Quality Improvement, and Evaluation sometimes seem like interchangeable terms that need to be mastered as part of these efforts. How are these concepts related, how are they are different, how can each be used for its intended purpose, and how can use of these concepts be complimentary? Using case studies, mini-lectures, and application opportunities, the facilitator will define and illustrate how these concepts are related and can be thoughtfully used to improve organizational and program performance. While the focus of mini-lectures and case studies will be on public health agencies, personnel in all types of organizations may find this session useful.
 

By the end of the session, participants will be able to:

• Define performance management, quality improvement, and evaluation

• Describe how these concepts are related, similar, and different

• Describe examples of how these concepts have been used to improve performance

• Apply these concepts in their own settings

Audience: This course assumes a basic understanding of evaluation, some experience using the CDC Evaluation Framework or similar framework, and some involvement and interest in improving organizational or program performance

 

Mary V Davis is Director of Evaluation Services at the North Carolina Institute for Public Health and Adjunct Faculty in the University of North Carolina School of Public Health where she teaches several advanced evaluation courses. Since 2001, Evaluation Services has provided program planning and evaluation services to more than 40 programs at NCIPH, the School of Public Health, and national efforts. Mary conducts public health systems and services research using participatory approaches; specialty areas are accreditation, quality improvement, and public health partnerships. She earned a DrPH from Johns Hopkins University School of Public Health and an MSPH from the University of Alabama at Birmingham School of Public Health.

 

Offered:

• Monday, June 13, 9:25 – 12:45 (20 minute break within)

Offering 50: The Bottom-Up Approach for Assessing Viability, Effectuality, and Transferability

Level: Intermediate

Description: The popular top-down approach (a sequential route from efficacy evaluation to effectiveness evaluation) has made a profound contribution in advancing biomedical interventions, but its application to health promotion/social betterment programs has not been as fruitful as anticipated. Efficacy evaluations, for example, are usually not followed by effectiveness evaluations. Evaluators and researchers have increasingly recognized that in an evaluation, the over-emphasis on internal validity reduces that evaluation’s usefulness and contributes to the gulf between academic and practical communities regarding interventions. This workshop will introduce the bottom-up approach (a sequential route from viability evaluation, effectiveness evaluation, to efficacy evaluation) as an alternative perspective for program evaluation. 

Participants will leave with an understanding of the new approach and its advantages in enabling evaluators to meet scientific and service requirements, facilitating in advancing external validity, gaining a new perspective on research methods, and furnishing a balanced view of credible evidence.

The workshop will provide an opportunity for participants to engage in discussion, share ideas, and learn from each other in a supportive environment.  

Audience: Attendees with a basic background in logic models and/or program theory.

Huey Chen is a senior evaluation scientist at the CDC. He was a professor at the School of Public Health at the University of Alabama at Birmingham until January 2008. Dr. Chen has contributed to the development of evaluation theory and methodology, especially in the areas of program theory, theory-driven evaluations, and evaluation taxonomy. His book Theory-Driven Evaluations has been recognized as one of the landmarks in program evaluation and his newest text, Practical Program Evaluation, offers an accessible approach to evaluation for those working in any context. In 1993 he received the AEA Paul F. Lazarsfeld Award for contributions to Evaluation Theory and in 1998 he received the CDC Senior Biomedical Research Service Award.

Offered:

• Monday, June 13, 2:15 – 4:15 PM

Offering 51: Evaluating Community Engagement in Translational Health Promotion Activities: Tools, Tips and Lessons Learned

Level: Beginners

Description: National public health disparities often require local, community-based approaches to influence behaviors and policies. While community-based organizations serve as catalysts for prevention and health promotion activities, many do not consistently practice program evaluation due to the challenges of limited time, staff and measurement skills. This session will describe how to 1) develop an audience targeted evaluation plan, 2) choose appropriate tools and mediums for community driven evaluations, 3) engage in bi-directional learning through employing both community engagement and evaluation approaches and 4) navigate challenges in community-campus evaluation partnership.

Audience: Attendees working with community-based public health initiatives to conduct evaluations, who offer evaluation capacity building or technical assistance in program assessment.

Tabia Henry Akintobi is a Research Assistant Professor within the Department of Community Health and Prevention at the Morehouse School of Medicine and also serves as Director of its Prevention Research Center. Tabia’s work is built upon establishment of partnerships incorporating her expertise in the areas of community-based participatory and translational research and evaluation. Donoria Evans, is the Interim Assistant Director of Evaluation at Morehouse School of Medicine’s Prevention Research Center. Currently, she serves as the evaluation project coordinator for several multi-site evaluation. Her behavioral research and program evaluation experience has focused on health disparities, community engagement, capacity building, and community-based evaluation.

Offered

• Tuesday, June 14, 2:15 – 4:15 PM

Offering 52: 25+ Low-cost/No-cost Tools for Evaluators

Level: All

Join us for a review of over 25 low-cost/no-cost tools that are useful, used, and user-friendly. Who isn't short on time, short on funds, and short on the patience needed to decide which tools are worth investing the time needed to access and use? Drawing on contributions from over 20 colleagues working in different evaluation contexts, we'll show examples of tools that are used by practicing evaluators to conduct background research; create and document evaluation plans and logic models; facilitate data cleaning, exploration and analysis; listen to and learn from online exchanges; and promote and enhance collaboration.

We’ll give mini-demonstrations of four tools, followed by rapid-fire exchange highlighting over 20 others with examples of each in the context of evaluation, and ending with a call to the audience to share their favorite tools that we have not mentioned (and perhaps win a little something for doing so!). Each attendee will leave with a handout identifying the tool, its uses, the time needed to learn to use it, and any special considerations. The session will be supplemented by a website with live links to each tool and where and how to learn more.

Audience: All

Susan Kistler is the Executive Director of the American Evaluation Association and owner of iMeasure Media. She has taught statistics, research methods, and evaluation at the university level and is an experienced trainer and speaker for local, regional, and national audiences. LaMarcus Bolton is the Technology Director for the American Evaluation Association and a doctoral student within Saint Louis University's Industrial/Organizational Psychology program.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 2:15 – 4:15 PM

• Tuesday, June 14, 2:15 – 4:15 PM FULL

Offering 53: Developing an Evaluation Plan

Level: Beginner

Use of evaluation results is not something that can be hoped or wished for but must be planned, directed, and intentional (Patton, 2008). A written plan is one of your most effective tools in your evaluation tool box. The process of developing an evaluation plan in cooperation with an evaluation workgroup of stakeholders will foster collaboration and a sense of shared purpose. Having a written evaluation plan will facilitate transparency and ensure that stakeholders are on the same page with regards to the purpose, use, and users of the evaluation results.

The purpose of this workshop is to help participants develop an understanding of what constitutes an evaluation plan, why it is important, and how to develop an effective evaluation plan in context of the planning process. This session includes time for practicing practical applications of portions of the evaluation plan process and participants will receive a packet of materials that will facilitate the development of a complete evaluation plan.

Audience: Attendees working in any context with a working knowledge of evaluation

René Lavinghouze is a Senior Evaluation Scientist in the Office of Smoking and Health at CDC. Rene has over 22 years experience with CDC and in the private sector. She is Chair of AEA’s TIG for Cluster, Multi-site/level evaluations. Jan Jernigan is a Senior Evaluation Scientist in the Division of Nutrition, Physical Activity, and Obesity at CDC. Prior to her work at the CDC, Jan was on the faculty at the Graduate School of Public Health (GSPH), University of Pittsburgh, for 15 years where she conducted numerous evaluation studies locally, statewide and nationally, and taught courses in program evaluation for eight years. Kim Snyder is an Analyst at ICF Macro. She is currently working as the liaison to the Office on Smoking and Health at CDC providing evaluation technical assistance to recovery act programs. She holds a Master’s degree in Public Health from the University at Buffalo

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 2:15 – 4:15 PM

• Tuesday, June 14, 2:15 – 4:15 PM

Offering 54: Using the Guiding Principles to Improve Your Evaluation Practice

Level: Beginner to Intermediate

Description: The Guiding Principles for Evaluators focus on five areas of evaluation practice: systematic inquiry, competence, integrity and honesty, respect for people, and responsibilities for general and public welfare. The Principles guide the professional practice of evaluators, and inform evaluation clients and the general public about the principles they can expect to be upheld by professional practitioners. This session will share ways to use the Principles to improve the ways in which you plan for and conduct evaluations and work with stakeholders and clients. After a brief presentation that introduces the Principles, participants will work together in small groups to discuss the Principles as they relate to a topical case study. Through case explorations, lecture and small and large group discussions, you will gain a deeper understanding of the practical applications of the Principles. The workshop will also introduce resources—print, web-based and collegial networks—that evaluators can consult to handle professional dilemmas that arise in their practice. You will receive copies of the workshop presentation, the case study, the Principles in full and abbreviated brochure format, and a list of resources for more information and consultation.

Audience: Evaluators and commissioners of evaluation working in any context

Leslie Goodyear is Program Officer in the Division of Research on Learning at the National Science Foundation. In addition to grant making, she coordinates evaluation for the division’s programs. As a program evaluator and researcher, Dr. Goodyear has worked with programs focused on HIV/AIDS Prevention; Out-of-School Time; Youth Engagement and Youth Media; Educational Research; and STEM Education. Leslie is a past Chair of the AEA Ethics Committee, past AEA Board member and current section editor for the Ethics Section of the American Journal of Evaluation. She earned her M.S. and Ph.D. in Human Service Studies, with focus on Program Evaluation, from Cornell University.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 2:15 – 4:15 PM

• Tuesday, June 14, 2:15 – 4:15 PM

Offering 55: Facing Evaluation Challenges in the Real World: A Case-Based Approach

Level: Beginner

Description: This session will use case study analysis to highlight strategic, ethical, and methodological challenges that evaluation practitioners encounter in the real world and explore effective strategies for meeting those challenges. As a result of class discussion, case analysis and small group activities, participants will be able to:

• Discuss ethical, strategic and methodological challenges associated with evaluation practice

• Propose strategies for meeting those challenges

• Discuss proactive strategies for ensuring an effective and useful evaluation study.

Audience: Novice evaluators working in any context.

Iris Smith holds a doctorate in Community Psychology from Georgia State University and a Master’s Degree in Public Health from Emory University. She is currently an Associate Professor and Director of the Career Master of Public Health Program (CMPH) at Emory University’s Rollins School of Public Health, where she also teaches a graduate level online course in Evaluation Research. Her current evaluation projects include the evaluation of the Atlanta Clinical Translational Science Institute, an interdisciplinary collaboration between Emory University, Georgia Institute of Technology and Morehouse School of Medicine. She is also Co- Director of the Evaluation Core for the Emory Prevention Research Center. Previously, Iris was the Director of National Evaluation Services for the American Cancer Society, and has also served as a Deputy Commissioner for the Georgia Department of Juvenile Justice.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 2:15 – 4:15 PM

• Tuesday, June 14, 2:15 – 4:15 PM

Offering 56: Using GIS in Evaluation

Level: Beginner

Description: At a broad level this workshop will provide an overview of what a geographic information system (GIS) is and ways in which one can be used for evaluative purposes. Specifically, workshop participants will be engaged in an interactive presentation and discussion about how these systems can be used in multiple settings (organizations and schools, communities, states). Learning objectives for participants include:

• Understanding how GIS can be used to develop and answer evaluation questions.

• Learning how GIS data can be used in conjunction with other types of data to approach evaluation questions through multiple methods.

• Becoming acquainted with GIS resources (e.g., websites, software, hardware, archival and pre-existing data sources) which are available to assist practitioners and researchers in obtaining and analyzing GIS data.

• Discussing a “real-world” case example which utilized a multi-method approach (including GIS methods) to evaluate a collaborative community-based project which aims to increase physical activity in high-crime low-income neighborhoods.

This workshop has been developed for participants who have minimal knowledge of GIS and its uses; however, it is assumed that novice GIS users who attend this workshop will be able to broaden their knowledge-base through the interactive discussion and completion of the learning objectives.

Duncan Meyers is a doctoral candidate in the Clinical/Community Psychology program at the University of South Carolina. He got his start in evaluation as the Evaluation Coordinator for a pilot System of Care project which sought to transform the community-based child mental health services of a large metropolitan area. Since that time, Duncan has been an evaluator for a University-based Center for Public Health Preparedness which conducted large scale trainings for local public health practitioners, has been en evaluator of an after-school program which aimed to increase physical activity for middle school students, and has been an evaluator for a community-based project which aims to increase physical activity for adults. As a project director for an NIH funded environmentally-based intervention which seeks to increase physical activity in high-crime low-income neighborhoods, Duncan has used GIS to enhance the methods through which outcomes of the intervention are evaluated.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 2:15 – 4:15 PM

• Tuesday, June 14, 2:15 – 4:15 PM

Offering 57: You + Graphic Design = Great Evaluation Presentations

Level: Beginner

Description: "Death by PowerPoint" won't literally kill your audience. But it will cause them to check their phone messages, flip ahead in the handout, and fall asleep. In this demonstration, attendees will learn the science behind good design for evaluation materials and will leave with direct, pointed changes that can be administered to their own evaluation presentations. The demonstration will focus on evidence-based principles of presentation design that support legibility, comprehension, and retention of our evaluation work in the minds of our clients. Grounded in visual processing theory, the principles will enhance attendees' ability to communicate more effectively with peers, colleagues, and clients through a focus on the proper use of color, placement, and type. We’ll look at before and after examples and incorporate group critique of samples to gain a concrete understanding of what works, what doesn’t, and why. You’ll leave with an understanding of how to improve your own presentations as well as information about tools that anyone can use to make their work cleaner, clearer, and more visually appealing.

Audience: Anyone who has to give a presentation or create handouts and materials to convey evaluation information.

Stephanie Evergreen is the owner of Evergreen Evaluation. Her dissertation work explores the role of graphic design in evaluation communication. Stephanie has applied design principles to ramp up her own communications as well as in an advisory role to stakeholders.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 2:15 – 4:15 PM

• Tuesday, June 14, 2:15 – 4:15 PM

Offering 58: Analyzing Qualitative Data: Using Qualitative Data Analysis Software to Balance the Expected and Unexpected

Level: Intermediate

Description: Ray Maietta's ‘Sort and Sift, Think and Shift’ qualitative method informs the content of this session. Qualitative evaluations are often defined by pre-determined goals and questions to pursue in analysis. However, issues emerge in initial document reviews that both confirm and challenge these goals. This session addresses ways to use qualitative data analysis software to facilitate serendipitous discovery and to balance new ideas with pre-existing questions for the study. We will discuss ways to ensure that software, with ATLAS.ti and MAXQDA as examples is always a tool that supports your exploration rather than it being a driver that defines where you are to go.

Audience: Researchers in any discipline who have collected qualitative data in the form of interviews, focus groups or fieldnotes

Raymond C. Maietta  is president of ResearchTalk Inc., a qualitative research consulting company. A sociologist from the State University of New York at Stony Brook, Ray's interests in the art of qualitative research methods motivated him to start ResearchTalk in 1996. ResearchTalk Inc. provides advice on all phases of qualitative analysis to university, government, not-for-profit and corporate researchers. Work with ResearchTalk clients using qualitative software informs recent book chapters: Systematic Procedures of Inquiry and Computer Data Analysis Software for Qualitative Research (with John Creswell) and State of the Art: Integrating Software with Qualitative Analysis. More than 15 years of consultation with qualitative researchers informs the methods book Ray is writing. Sort and Sift, Think and Shift will be completed in 2012.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 2:15 – 4:15 PM

• Tuesday, June 14, 2:15 – 4:15 PM

Offering 59: Collaborative Evaluations: A Step-by-Step Model for the Evaluator

Level: Beginner

Description: Do you want to engage and succeed in collaborative evaluations? Using clear and simple language, Dr. Liliana Rodríguez will outline key concepts and effective tools to help master the mechanics of collaboration in the evaluation environment. Specifically, you will explore how to apply the Model for Collaborative Evaluations (MCE) to real-life evaluations, with a special emphasis on those factors that facilitate and inhibit stakeholders' participation. The presenter shares her experience and insights regarding this subject in a precise and easy to understand fashion, so that participants can use the information learned from this workshop immediately. Using discussion, demonstration, hands-on exercises and small group work, participants will apply the learned techniques to specific situations. In addition, participants are encouraged to bring actual evaluation examples, present scenarios and/or specific problem areas for discussion.

You will learn to:

• Understand the factors that influence the success of collaboration in evaluations,

• Capitalize on others' strengths to encourage feedback, clarify interpretations, and resolve misunderstandings,

• Select the methods and tools to facilitate collaborative evaluations and build collaborative relationships.

Audience: Evaluators working in any context.

Liliana Rodriguez is an assistant professor in the educational measurement and research department at the University of South Florida’s College of Education. She has received with the American Evaluation Association's Marcia Guttentag Promising New Evaluator Award and served as Program Chair for AEA's Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group. She is the author of Collaborative Evaluations (Lumina Press), a highly comprehensive and easy-to-follow book for those evaluators who want to engage and succeed in collaborative evaluations.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 13, 2:15 – 4:15 PM

• Tuesday, June 14, 2:15 – 4:15 PM

Offering 60: Unraveling the Participant Observer Interaction: "Being there" more than a data collection method for evaluators?

Level: Beginner

Description: Participant observation, as opposed to non-participant or direct observation, is a method in which the evaluator takes part, more or less, in the everyday activities of the practice or program under study. Participants in this workshop will share their experiences with participant observation, engage in some participant observing, and trouble the distinction made between an explicit culture, as enacted or reported by people involved in the evaluand, and the evaluator’s tacit experiences of the culture so enacted. An engaged conception of participant observation is put forward delineating an interactional, interventionist role for the evaluator. Questions to be addressed include: What are the advantages of “being there” and how can those advantages be maximized? How does the engaged nature of responsive evaluation alter evaluator-stakeholder relationships as well as the kinds of reporting practices evaluators might use?

Audience: Evaluators working in any context

Jori N. Hall is an assistant professor at the University of Georgia. As a member of the Qualitative Research Program, she teaches graduate courses in Qualitative Research Design, Program Evaluation, and Mixed Methods Research. Dr. Hall also works with the Program Evaluation Group (PEG), a unit within the College of Education that conducts local, state, and national level evaluations. Dr. Hall specializes in researching responsive approaches to evaluation, focusing on strategies for engaging stakeholder values. She has made scholarly contributions to journals such as Qualitative Inquiry (2011), New Directions for Evaluation (2006), and the second edition of the Handbook of Mixed Methods in Social and Behavioral Research (Sage, 2010).

Offered:

• Tuesday, June 14, 2:15 – 4:15 PM

Offering 61: Engaging Stakeholders in Data Issues: Creating a Conversation

Level: Beginner

Description: 200 words Sharing data with stakeholders can feel like a tension-filled “big reveal.” This session will share strategies to change the dynamic to a conversation about what the data say and how evaluators and stakeholders can jointly determine what the data mean. The presenter will use stories from the field and share tips to facilitate participant ability to create a shared conversation about data and how to ensure that results will be used.
By the end of the session, participants will be able to:

• Identify common misconceptions about sharing data with stakeholders

• Describe strategies to create a conversation with stakeholders about data

• Describe strategies to engage stakeholders in ensuring that results are used

Audience: This is an introductory level session designed to introduce concepts of data sharing with stakeholders.

Mary V Davis is Director of Evaluation Services at the North Carolina Institute for Public Health and Adjunct Faculty in the University of North Carolina School of Public Health where she teaches several advanced evaluation courses. Since 2001, Evaluation Services has provided program planning and evaluation services to more than 40 programs at NCIPH, the School of Public Health, and national efforts. Mary conducts public health systems and services research using participatory approaches; specialty areas are accreditation, quality improvement, and public health partnerships. She earned a DrPH from Johns Hopkins University School of Public Health and an MSPH from the University of Alabama at Birmingham School of Public Health.

Offered:

• Monday, June 13, 2:15 – 4:15 PM

Offering 62: Design Principles of Dashboards and Interactive Applications

Level: Intermediate

Description: Building on this morning’s keynote, we will explore the depths of best practices for applying visualization principles when designing web-based dashboards and other analytical tools. We will describe how to gradually reveal information as users express interest, how to make the tough decisions about what information should be shown, and offer guidance on creating workflows that support making your data more impactful. We will demonstrate the process for constructing information presentation displays including: 1) understanding specific people needs and data requirements; 2) determining key features and functionality for the interface; 3) styling choices to create an information presentation that encourages engagement and exploration.

Audience: Those attending should be comfortable with identifying basic measures and indicators, as well as working in an online environment

Ken Hilburn joined Juice in 2007 with over 20 years of technology development experience focused on software product development and implementation. Ken works with Juice’s community and customers to help them better understand how to incorporate information presentation best practices into their world. Ken holds an undergraduate degree from the Georgia Institute of Technology and lives right here in Atlanta. Zach Gemignani is co-founder and CEO of Juice Analytics. His focus on making client’s data more usable and valuable has helped Juice provide solutions with real benefit. His writings and speaking engagements have earned him a tremendous reputation in the Data Visualization arena. Before Zach helped found Juice, he led efforts to bring analytical rigor to AOL and was a consultant with Diamond Consultants and Booz Allen.

Offered:

• Monday, June 13, 2:15 – 4:15 PM