SCHEDULE OVERVIEW ¨ WORKSHOPS ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS

PRE-INSTITUTE WORKSHOPS

• PI1: (McKnight) Quantitative Methods for Evaluation

• PI2: (Apt) Introduction to Evaluation

Note that these pre-institute workshops are not included in standard Institute registration, and require an additional payment, but may be registered for on the same form. We also welcome registrants who wish to attend only a pre-institute workshop and that, too, may be done using the regular registration form.

PI1: Quantitative Methods for Evaluation

Level: Beginner
 
Description: Brush up on your statistics! We will focus primarily on descriptive statistics: measures of central tendency (e.g., the mean) and measures of dispersion (e.g., standard deviation), defining what these terms mean, when and how to use them, and how to present them to evaluation stakeholders. From there, we will move to a basic discussion of inferential statistics, the difference between a statistic and a parameter, and the use of population parameters. The workshop will involve demonstrations by the presenter using SPSS software as well as hands-on exercises for you to calculate some of the descriptive statistics we cover. This workshop is not designed to teach you how to use SPSS or any other statistical software package, but rather to introduce you to the process of statistical analysis and interpreting statistical output. We will focus on interpreting statistical indices (e.g., means and standard deviations) as well as graphical output, such as histograms and stem-and-leaf plots. The understanding you gain regarding the concepts presented in this course, coupled with the chance to interpret statistical output, should ready you to employ basic descriptive statistics as well as to interpret the work of others.

Audience: Attendees who have never taken a statistics course before, or who feel they need a refresher workshop on basic statistics. Those who attend this workshop need no prior background in statistics

Katherine McKnight has taught statistics workshops at AEA's annual conferences for many years. An outstanding facilitator, she is known for making difficult concepts accessible and interesting.

Offered:

• Sunday, June 13, 9:00 - 4:00 (1 hour break for lunch within)

PI2: Introduction to Evaluation

Level: Advanced Beginner

Description: This workshop will provide an overview of program evaluation for Institute participants with some, but not extensive, prior background in program evaluation. The session will be organized around the Centers for Disease Control and Prevention’s (CDC) six-step Framework for Program Evaluation in Public Health as well as the four sets of evaluation standards from the Joint Commission on Evaluation Standards. The six steps constitute a comprehensive approach to evaluation. While its origins are in the public health sector, the Framework approach can guide any evaluation. The course will touch on all six steps, but particular emphasis will be put on the early steps, including identification and engagement of stakeholders, creation of logic models, and selecting/focusing evaluation questions. Several case studies will be used both as illustrations and as an opportunity for participants to apply the content of the course and work through some of the trade-offs and challenges inherent in program evaluation in public health and human services.

Audience: Attendees with some background in evaluation, but who desire an overview and an opportunity to examine challenges and approaches. Cases will be from public health but general enough to yield information applicable to any other setting or sector.

Betty Apt has worked in evaluation for over 20 years. She is recently retired as an evaluation specialist with the Division of Sexually Transmitted Disease Prevention at the Centers for Disease Control and Prevention (CDC) where she provided evaluation technical assistance to the Division; and planned, designed, and oversaw evaluation projects related to the prevention of STDs. Prior to her work at CDC, Betty served as a team leader for national evaluations of a variety of programs administered by the Department of Health and Human Services, including Medicare, Medicaid, and public assistance. Currently, Betty works as an independent consultant on a range of evaluation projects.

Offered:

• Sunday, June 13, 9:00 - 4:00 (1 hour break for lunch within)

SCHEDULE OVERVIEW ¨ WORKSHOPS ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS

SCHEDULE OVERVIEW ¨ WORKSHOPS ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS

Monday, June 14, Keynote 1:

(Collins) Program Evaluation and Program Improvement: Challenges, Opportunities, and New Directions at the Centers for Disease Control and Prevention

Description: In 1999, with the publication of its Framework for Program Evaluation in Public Health, CDC signaled its desire to increase the quality of program evaluation and use of evaluation findings for program improvement. Great strides have been made in program evaluation of public health in the decade that followed. And recently, with the creation of the new Associate Director for Program office, CDC has a focal point for extending and improving performance measurement and evaluation of its efforts. Still, challenges remain. Dr. Collins will talk about the challenges of program evaluation in public health and at CDC, some of the opportunities for increasing the leverage and impact of program evaluation and performance measurement, and some key current initiatives and new directions being pursued at CDC as examples of how one large organization can effectively employ performance measurement and program evaluation for program improvement.

Janet Collins is Associate Director for Program in the Office of the Director, CDC. She joined CDC in 1990 as Chief of the Surveillance and Evaluation Research Branch in the Division of Adolescent and School Health, and, for the past four years, served as Director of CDC’s National Center for Chronic Disease Prevention and Health Promotion, directing a diverse portfolio of programmatic and scientific initiatives across ten Divisions. Dr. Collins is a respected voice in program evaluation with the public health community; she earned her Ph.D. in Educational Psychology from Stanford University.

Offered as Keynote Address:

• Monday, June 14, 8:30 - 9:15 AM

Tuesday, June 15, Keynote 2:

(Fine) Challenges and Opportunities for Measuring Social Media Efforts in a Networked World

Description: In a world powered by social media, a new kind of organizational form has developed, the Networked Nonprofit. Allison Fine will describe this new entity and discuss the particular challenges and exciting new opportunities for measuring the use of social media by organizations of all types. We will follow the bouncing ball of new relationships and connections created through social media in order to better understand the ways that digital media are shaping social change efforts.

Allison H. Fine is the founder of Innovation Network, a nonprofit dedicated to transforming evaluation for social change. She is the author of the award-winning book Momentum: Igniting Social Change in the Connected Age (Jossey - Bass/John Wiley & Sons, 2006) and a senior fellow on the democracy team at Demos: A Network for Change and Action in New York City. In 2008, she published a paper on young people and activism commissioned by The Case Foundation called “ Social Citizens beta ,” and co-edited a collection of essays, Rebooting America: Ideas for Redesigning American Democracy for the Internet Age, published by the Personal Democracy Forum (2008), about transformative ways to reinvent twentyfirst-century democracy using new media tools. Allison hosts a monthly podcast for The Chronicle of Philanthropy called “Social Good ” and writes her own blog, A. Fine Blog. Her newest book, The Networked Nonprofit, with Beth Kanter, is due out in June of 2010 from John Wiley and Sons.

Offered as Keynote Address:

• Tuesday, June 15, 8:30 - 9:15 AM

Wednesday, June 16, Keynote 3:

(Newcomer): Performance Management in the Obama Administration

Description: Professor Newcomer will describe the context and challenges the Obama Administration’s approach to performance management confronts. She will describe the legacy of performance measurement assets, logistical challenges and skepticism the Obama Administration inherited. Then she will lay out the initial actions taken by the new Administration, and commitments articulated by Obama’s Chief Performance Officer and OMB staff. And finally, she will outline some challenges facing the Obama Administration as the new OMB and agency leaders try to move performance improvement efforts forward.

Kathryn Newcomer is a the Director of the Trachtenberg School of Public Policy and Public Administration at the George Washington University where she is also the Co-Director of the Midge Smith Center for Evaluation Effectiveness, home of The Evaluators’ Institute (TEI), She teaches and routinely conducts research and training for federal and local government agencies and nonprofit organizations on performance measurement and program evaluation. Dr. Newcomer has published numerous journal articles as well as five books, including The Handbook of Practical Program Evaluation (with Joe Wholey and Harry Hatry and the 3rd Edition forthcoming in 2010), Getting Results: A Guide for Federal Leaders and Managers (2005), and Transformational Leadership: Leading Change in Public and Nonprofit Agencies, (2008). She is a Fellow of the National Academy of Public Administration, and currently serves on the Comptroller General’s Educators’ Advisory Panel. Dr. Newcomer has received two Fulbright awards, the Elmer Staats Award for Achievements in Government Accountability, as well as two awards for outstanding teaching.  

Offered as Keynote Address:

• Tuesday, June 15, 8:30 - 9:15 AM

SCHEDULE OVERVIEW ¨ WORKSHOPS ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS

CONCURRENT SESSION DESCRIPTIONS

Offering 1: Handling Data: From Logic Model to Final Report (new!)

Collect, analyze, and present data from complex evaluation studies in ways that are feasible for the evaluator and meaningful to the client. Explore lessons learned through over twenty years in evaluation consulting to ask the right questions, collect the right data and analyze and present findings in simple yet comprehensive ways.

We will use actual data samples along with examples of analysis techniques. You will have an opportunity to work in small groups with sample data and will explore various analysis techniques. Throughout the workshop, the presenter will respond to individual questions and facilitate group discussion on data handling topics. At the end of the workshop, you will take away fresh ideas to tackle your data handling challenges.

You will learn:

• To develop and link a program theory, a holistic logic model, a data collection matrix, and evaluation tools,

• To ask the right questions and get the answers you need,

• To develop a data summary that triangulates the information collected from different sources,

• To extract and map themes, prepare an evidence table, and report findings in a comprehensive but user-friendly way.

Gail V. Barrington is an independent consultant who started her consulting firm, Barrington Research Group, Inc. in 1985. She has conducted over 100 program evaluation studies and has made a significant contribution to the field of evaluation through her practice, writing, teaching, training, mentoring and service. In 2008, she won the Canadian Evaluation Society award for her Contribution to Evaluation in Canada.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 2: Lessons From the Field: Ways to Improve Your Evaluation Practice (new!)

Explore lessons learned in structuring evaluation projects, improving client relationships, responding to ethical issues, managing project resources and enhancing personal survival skills in stressful work environments. The presenter will share her hard-won lessons from the field, which have allowed her to bootstrap her evaluation practice up from one level to the next over the past two decades.

Participants will have an opportunity to work in small groups to develop solutions to a scenario and will compare strategies with their colleagues. At the end of the workshop, you will take away some fresh ideas to tackle your most pressing evaluation challenges.

You will learn:

• A conceptual framework around which to organize evaluation projects,

• Ways to improve client relationships,

• Standards for ethical conduct,

• Tools for project management,

• Tips for personal survival.

Gail V. Barrington is an independent consultant who started her consulting firm, Barrington Research Group, Inc. in 1985. She has conducted over 100 program evaluation studies and has made a significant contribution to the field of evaluation through her practice, writing, teaching, training, mentoring and service. In 2008, she won the Canadian Evaluation Society award for her Contribution to Evaluation in Canada.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 2:30 – 4:00 PM

Offering 3: Leading Concepts in Community Health Evaluation

Level: Intermediate

In this session, participants will learn different evaluation methodologies for communityassessments and be able to apply multiple approaches to the evaluation of community-based health promotion programs. Among other topics, the course will address:

• Dr. Goodman’s FORECAST method of evaluating complex community programs, including how to develop models for complex programs and then use them to develop markers, measures, and standards (meanings) for measuring program adequacy.

• Social ecological perspectives including assessing and determining adequacy of interventions from a social ecology perspective

• Community capacity, including expanding our perspectives of community capacity within the construct of health promotion, identifying linkages that interconnect to strengthen community capacity, and developing an evaluation approach that assesses the development of community capacity

• Uncovering “deeper structural meanings” in community responses to evaluation

Audience: Attendees working with community health programs with experience conducting evaluations.

Robert M. Goodman is a Professor and Dean of the School of Health, Physical Education and Recreation at Indiana University. Dr. Goodman has written extensively on issues concerning community health development, community capacity, community coalitions, evaluation methods, organizational development, and the institutionalization of health programs. He has been the principal investigator and evaluator on projects for CDC, The National Cancer Institute, The Centers for Substance Abuse Prevention, The Children’s Defense Fund, and several state health departments. In 2004, Dr. Goodman received the Distinguished Fellow Award from the Society for Public Health Education, the highest honor it bestows. Currently, Dr. Goodman is consulting on community-based public health practices and empowerment evaluation with the Diabetes Translation and Injury Prevention Branches at CDC. Also, he is leading an evaluation of community-based approaches to increasing interest in cancer clinical trials.

Offered:

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

Offering 4: Systems Level Evaluation of Communities of Practice

Level: Intermediate

Description: In an environment of increasing social participation and transparency, communities of practice are one means to unite a variety of partners to address common issues, share resources, and learn new information. When asked to design an evaluation of this type of complex social initiative, evaluators increasingly turn to system level evaluation. One means to frame a system level evaluation is the use of social and behavioral science theory. Both implicit and explicit use of theory will be covered, including theory of change as an explicit approach that links activities, outcomes, and contexts in a way that maximizes the attribution of interventions to outcomes. This workshop will use lecture, exercises and discussion to improve attendees’ ability to understand the application of a systems level evaluation to communities of practice as well as how to design evaluations of such complex social initiatives.

You will learn:

• What communities of practice are and how they are used

• How to view communities of practice within a systems-level framework

• How to use theory, including theory of change to evaluate communities of practice and their impacts

Audience: Attendees with a basic understanding of systems approaches who would like to improve their evaluation of large-scale social programs

Andrea M. Hegedus, is a Health Research Analyst for Northrop Grumman Corporation. She has over 25 years of experience including evaluation of behavioral healthcare programs and other complex social initiatives. Jan C. Jernigan, is a Senior Evaluator in the Division of Nutrition, Physical Activity and Obesity at CDC. She has taught program evaluation courses and conducted program evaluation and evaluation research for over 15 years. Drs. Hegedus and Jernigan both received their doctorates from the Graduate School of Public and International Affairs at the University of Pittsburgh with a specialty in public policy research and analysis.

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 16, 9:25 – 12:45 (20 minute break within)

Offering 5: "PIP Dreams": Using Evaluation Findings to Develop Program Improvement Plans (PIP) (new!)

Description: This workshop will provide participants with a structure and tools that can be used to turn evaluation findings into program improvements. The workshop will be centered on a diagram that directs participants in how to use their evaluation findings and other data to determine if a program improvement plan (PIP) is needed, and a template they can use to guide the steps needed to implement the improvements. These tools can also be used to plan programs, as well as improve existing programs.

Audience: Attendees with some background in evaluation who wish to learn how to develop a plan to improve their program, based on evaluation findings. 

Betty Apt has worked in evaluation for over 20 years. She is recently retired as an evaluation specialist with the Division of Sexually Transmitted Disease Prevention at the Centers for Disease Control and Prevention (CDC) where she provided evaluation technical assistance to the Division; and planned, designed, and oversaw evaluation projects related to the prevention of STDs. Prior to her work at CDC, Betty served as a team leader for national evaluations of a variety of programs administered by the Department of Health and Human Services, including Medicare, Medicaid, and public assistance. Currently, Betty works as an independent consultant on a range of evaluation projects.

Offered:

• Monday, June 14, 2:30 – 4:00 PM

Offering 7: Using Program Evaluation to Improve Nonprofit Outcomes

Description: Funders, communities, and service recipients themselves ask that nonprofits show results and demonstrate progress towards service goals. Yet, identifying, measuring, and reporting on service outcomes in ways that attend to the multiple stakeholders involved can be a challenging task. This workshop will help evaluators, program officers, and program administrators working in the nonprofit sector by providing an overview of the practice of outcomes assessment tailored to the nonprofit context. Attendees at the session will:

• Learn what/who drives program evaluation and outcomes assessment in nonprofit service providers,

• Explore uses of performance data in the nonprofit sector,

• Discuss challenges to measurement of social service program outcomes, and

• Develop strategies for making measurement decisions.

Audience: Evaluators, program officers, and program administrators working in the nonprofit sector.

Kathryn Newcomer is a the Director of the Trachtenberg School of Public Policy and Public Administration at the George Washington University where she is also the Co-Director of the Midge Smith Center for Evaluation Effectiveness, home of The Evaluators’ Institute (TEI), She teaches and routinely conducts research and training for federal and local government agencies and nonprofit organizations on performance measurement and program evaluation. Dr. Newcomer has published numerous journal articles as well as five books, including The Handbook of Practical Program Evaluation (with Joe Wholey and Harry Hatry and the 3rd Edition forthcoming in 2010), Getting Results: A Guide for Federal Leaders and Managers (2005), and Transformational Leadership: Leading Change in Public and Nonprofit Agencies, (2008). She is a Fellow of the National Academy of Public Administration, and currently serves on the Comptroller General’s Educators’ Advisory Panel. Dr. Newcomer has received two Fulbright awards, the Elmer Staats Award for Achievements in Government Accountability, as well as two awards for outstanding teaching.

Offered:

• Wednesday, June 16, 9:25 – 12:45 (20 minute break within)

Offering 8: RealWorld Evaluation

Level: Intermediate

Description: What do you do when asked to perform an evaluation on a program that is well underway? When your questions about baseline data and control groups are met with blank stares? When time and resources are few, yet clients expect “rigorous impact evaluation”? When there are political expectations and pressures to deal with? This workshop presents a seven-step approach that seeks to ensure the best quality evaluation under real-life constraints. Through presentations and discussion, with real-world examples drawn from multiple contexts, participants in this workshop will be introduced to the RealWorld Evaluation approach. RealWorld evaluation techniques are applicable to evaluators working in any context with budget, time, data and political constraints. The fundamental framework for the session is an exploration off the RealWorld Evaluation approach, including ways to negotiate RealWorld solutions with clients. Finally, we’ll identify a number of evaluation designs appropriate for RealWorld evaluations and discuss their appropriate context-dependent applications.

Audience: Evaluators working in any context with previous experience in many of the common evaluation approaches and methods.

Jim Rugh brings over 46 years of experience in international development, including 30 years specializing in program evaluation. For 12 years he was the Coordinator of program Design, Monitoring and Evaluation for CARE, a large international NGO headquartered in Atlanta with programs in 70 countries. Under Jim’s leadership CARE adopted an evaluation policy, strategies, standards and guidelines to promote learning for improved program quality and accountability for effectiveness. An active member of AEA for almost 25 years, he currently serves as the AEA Representative to the IOCE (International Organization for Cooperation in Evaluation). Jim has facilitated evaluation workshops for many years at AEA’s annual conferences as well as many other countries including South Africa, Niger, Egypt, Sri Lanka, Nepal, India, Thailand, Indonesia, Ukraine, Portugal, the UK and elsewhere. He was co-author with Michael Bamberger and Linda Mabry of the book RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints published by SAGE in 2006..

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 9: Focus Group Research: Understanding, Designing and Implementing

Level: All

Description: As a qualitative research method, focus groups are an important tool to help researchers understand the motivators and determinants of a given behavior. This course provides a practical introduction to focus group research. At the completion of this course, participants will be able to 1) identify and discuss critical decisions in designing a focus group study, 2) understand how research or study questions influence decisions regarding segmentation, recruitment, and screening; and, 3) identify and discuss different types of analytical strategies and focus group reports.

Audience: Attendees working in any context who are new to focus group facilitation

Michelle Revels is a technical director at ORC Macro specializing in focus group research and program evaluation. Ms. Revels attended Hampshire College in Amherst, Massachusetts and the Hubert H. Humphrey Institute of Public Affairs at the University of Minnesota.

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 16, 9:25 – 12:45 (20 minute break within)

Offering 10: Theory-Driven Evaluation for Assessing and Improving Program Planning, Implementation, and Effectiveness

Level: Intermediate

Description: Learn the theory-driven approach for assessing and improving program planning, implementation and effectiveness. You will explore the conceptual framework of program theory and its structure, which facilitates precise communication between evaluators and stakeholders regarding evaluation needs and approaches to address those needs. Mini-lectures, group exercises and case studies will be used to illustrate the use of program theory and theory-driven evaluation for program planning, initial implementation, mature implementation and outcomes. In addition, the participants will learn principles and strategies for using the theory-driven approach to deal with the following cutting edge issues: how to go beyond traditional methodology for designing a real world evaluation, how to achieve both internal and external validity in an evaluation, and how to use program theory for guiding the application of mixed methods in an evaluation.

Audience: Attendees with a basic background in logic models and/or program theory.

Huey Chen is a senior evaluation scientist at the CDC. He was a professor at the School of Public Health at the University of Alabama at Birmingham until January 2008. Dr. Chen has contributed to the development of evaluation theory and methodology, especially in the areas of program theory, theory-driven evaluations, and evaluation taxonomy. His book Theory-Driven Evaluations has been recognized as one of the landmarks in program evaluation and his newest text, Practical Program Evaluation, offers an accessible approach to evaluation for those working in any context. In 1993 he received the AEA Paul F. Lazarsfeld Award for contributions to Evaluation Theory and in 1998 he received the CDC Senior Biomedical Research Service Award.

Offered:

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

Offering 11: How Policy Is Made and How Evaluators Can Affect It

Level: All

Description: This session will explain how public policy processes work and how evaluators can get their evaluations noticed and used by policy makers. It will guide evaluators through the maze of policy processes, such as legislation, regulations, administrative procedures, budgets, re-organizations, and goal setting. It will also explain how policy makers can be identified and reached and describe their preferred writing and other communication styles, what they value in terms policy advice, and when they are open to it. The session will show how to effectively present evaluation findings and recommendations to decision makers of congressional and executive branches of Federal, state, and local governments as well as boards and administrative offices of foundations and not-for-profit public service organizations. It will also explain how to influence policies that affect the evaluation profession.

Audience: This session is designed for evaluators of any experience level who wish to have impact on public policies through their evaluations.

George F. Grob is President of the Center for Public Program Evaluation. He is an independent consultant focusing on evaluation of public programs and related fields of policy development and performance management. He currently serves as consultant to the AEA Evaluation Policy Task Force. Other recent projects include work for the Robert Wood Johnson Foundation, National Aquarium at Baltimore, NOAA’s National Marine Fisheries Administration, and the Agency for Families and Children. Before establishing this consultancy, he served as Executive Director of the Citizens’ Health Care Working Group and Deputy Inspector General for Evaluation and Inspections of the U.S. Department of Health and Human Services. He has testified before Congress more than two dozen times.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 2:30 – 4:00 PM

• Tuesday, June 15, 2:30 – 4:00 PM

Offering 12: Minding Your Mind

Level: All

Description: Evaluators and other professionals often overlook the importance of understanding and developing the one tool that they use in all facets of their work—their own brains. They seldom pay attention to how their brains work day by day; how memories are created, organized, and accessed; under what physical circumstances creative inspirations arise; how different ideas get connected to one another; how food choices, sleep, exercise, unstructured time, personal life problems, and other factors affect what they think about; and how easily or stressfully they handle various thinking chores. This course addresses these factors and explains how they can be controlled and corralled to make thinking easier, less stressful, and more productive. This is not about “gimmicks” but is based on the physiological aspects of brain functioning, research on food and the mind, and the working habits and experiences of successful people involved in intellectual activities.

Audience: This session is designed for evaluators and other professionals of any experience level who feel overwhelmed by having to keep too many balls in the air at one time, are exasperated from deadlines and paperwork, and wish to become more efficient and effective and derive greater enjoyment and success in their intellectual pursuits.

George F. Grob is President of the Center for Public Program Evaluation. He is an independent consultant focusing on evaluation of public programs and related fields of policy development and performance management. He currently serves as consultant to the AEA Evaluation Policy Task Force. Other recent projects include work for the Robert Wood Johnson Foundation, National Aquarium at Baltimore, and several Federal agencies. Previously, he served as Deputy Inspector General for Evaluation and Inspections of the U.S. Department of Health and Human Services where he and his staff systematically looked for ways to enjoy and succeed in their work. He has lectured on this and other psychological aspects of evaluation and other professional work.

Offered:

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 13: Strengthening Evaluation Through Cultural Relevance and Cultural Competence

Level: Intermediate

Description: This skill-building session addresses the centrality of culture in evaluation. It is organized in two segments. The opening segment addresses the relevance of culture to all stages of the evaluation process, to the fundamental validity of our work as evaluators, and to ethical standards and guidelines of our profession. Presenters will use an FAQ format to raise questions and address common misconceptions that marginalize discussions of culture within the evaluation community (e.g., Is “culture” really just a code-word for “race”? How does culture apply to me as a white evaluator working within predominantly white populations? What is the “value added” of culture in evaluation? Why should I care?) The second segment extends cultural relevance to present strategies for building cultural competence through experience, education and self-awareness. Theoretical frameworks that situate culture in evaluation (e.g., Frierson, Hood & Hughes, 2002; Hall & Hood, 2005; Kirkhart, 2005) are presented as advance organizers for practice and application purposes. Presenters use case scenarios and participants’ own examples to integrate workshop content with participants’ field experience, interests, and concerns. They rely on various theoretical frameworks to guide the two segments in tangible and practical ways. Additional resources are provided to extend and reinforce participant learning.

Audience: Attendees working in any context with a basic background in evaluation.

Karen E. Kirkhart holds a Ph.D. in Social Work and Psychology from The University of Michigan and is currently Professor, School of Social Work, College of Human Ecology, Syracuse University. Rodney K. Hopson has undergraduate and graduate degrees in English Literature, Educational Evaluation, and Linguistics from the University of Virginia, and he is Hillman Distinguished Professor, Department of Educational Foundations and Leadership and faculty member in the Center for Interpretive and Qualitative Research at Duquesne University. Karen and Rodney have served in positions of leadership within the American Evaluation Association, and both are actively involved in education and scholarship on culture, diversity, and social justice in evaluation. Rodney serves as Co-Project Director for the Robert Wood Johnson Foundation Evaluation Fellowship Program. Karen is a member of the AEA Multicultural Task Force and the Diversity Committee task force charged with developing a public statement on the subject of cultural competence and evaluation.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 14: Dissemination and Implementation Research (new!)

Level: Intermediate

Description: We will be focusing on the evaluating efforts associated with the process of moving from pilot or demonstration projects into widespread practice.   Issues associated with this process include the following:

• Packaging the program for potential users
• Marketing/disseminating  the program to the potential users
• Building capacity to select and use effective interventions
• Understanding the decisions involved in adopting a new program
• Maintaining quality control over multiple implementation sites
• Facilitating and measuring implementation
• Considering the balance between fidelity and adaptation
• Scaling up efforts to achieve public health impact

Each of these issues is a possible focus for evaluation research questions.  Among these are the following questions are the following:

• What characteristics of the message, its delivery agent, modality, and its recipient lead to the most successful dissemination efforts?
• What characteristics of programs, organizations, their mutual contexts, and resources affect the decision to adopt a program?
• What characteristics of programs, organizations, their mutual context and resources affect effective implementation (and related constructs such as fidelity and adaptation)?
• How do the processes and level of implementation act as moderators and/or mediators in understanding program outcomes

James Emshoff is an Associate Professor Emeritus of Psychology and former Director of the Community Psychology Program at Georgia State University. He also founded and serves as Director of Research at EMSTAR Research, Inc., an evaluation and organizational services firm.  Dr. Emshoff has conducted evaluation research focused on substance abuse, violence, HIV/AIDS, child abuse, community collaboratives, mentoring, delinquency, health promotion programs and issues of dissemination and implementation at the local, state, and national levels. Rita K. Noonan is a team leader in CDC’s Division of Unintentional Injury Prevention. Dr. Noonan has published on a variety of topics including: quality of life outcomes for women and children; the international debt crisis; women’s movements in Latin America; dating and sexual violence prevention; empowerment evaluation, older adult fall prevention; and “translational research.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 16, 9:25 – 12:45 (20 minute break within)

Offering 15: Managing Evaluation (new!)

Level: Intermediate

Description: Increasingly, funders are requiring nonprofits to evaluate their programs and demonstrate results. Program managers, however, are often overwhelmed or intimidated by the process of engaging in program evaluation. In order for evaluations to be successful, program directors must be involved. This workshop will help clarify where an overworked program manager should begin. We define program evaluation as another essential management tool. This workshop will help you understand the process of evaluation and your role in that process. We will explain the steps of an evaluation, how to assemble an internal evaluation team (and why you should), the roles of the evaluator and the program manager at each step in the evaluation, how to hire an evaluation consultant, what to ask for in a scope of work, the cost of evaluation, and what to expect in a final report.

Audience: This session is appropriate for program managers and program staff responsible for evaluation.

Ann Price is president of Community Evaluation Solutions, Inc and has over 20 years experience in both treatment and prevention. She has conducted evaluations in many areas including intimate partner violence, mental health, substance abuse, tobacco prevention and oral health. Prior to CES, Dr. Price was a Senior Data Analyst at ORC Macro on a multi-site national child mental health evaluation.

Offered:

• Monday, June 14, 2:30 – 4:00 PM

Offering 16: Logic Models as a Platform for Program Evaluation Planning, Implementation, and Use of Findings

Level: Beginner

Description: Practitioners use logic models to describe important components of a program; make visible a theory of change; and link activities to intended outcomes. For the purposes of evaluation practice, a well-constructed logic model provides a program-specific foundation for identifying evaluation questions; prioritizing data needs; and translating findings into recommendations for ongoing program improvement. Aimed directly at improving the utility of logic models and quality of evaluation practice in your setting, the workshop addresses two questions:

1. What are the hallmarks of a well-constructed, scientifically-sound and useful logic model?

2. How do we maximize the use of logic models for program evaluation planning, implementation and use of findings?

Workshop Objectives:

• Demystify and define the logic model as a starting point for everyday evaluation practice

• Identify the hallmarks of a well-constructed, scientifically-sound logic model

• Clarify the relevance of process and outcome evaluation to preparing and using program logic models

• Demonstrate the use of logic models to identify and prioritize evaluation questions and data needs

• Examine the use of logic models to identify opportunities/options for demonstrating accountability for scarce resources

• Demonstrate use of a logic model to guide preparation of findings/recommendations aimed at ongoing program improvement

• Pinpoint additional resources for continued study/application

Audience: Attendees working in any context who are new to logic modeling.

Sally Honeycutt joined the Emory Prevention Research Center in February 2007 as an Evaluation Specialist. Before coming to Emory, Sally was a member of the Surveillance and Evaluation Team for the Steps to a HealthierUS Program at CDC. She has served as a Maternal and Child Health Educator with the Peace Corps and has experience coordinating health promotion programs both domestically and internationally.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 17: The Bottom-Up Approach to Integrative Validity: A New Perspective for Program Evaluation (new!)

Level: Intermediate

Description: The popular top-down approach (a sequential route from efficacy evaluation to effectiveness evaluation) has made a profound contribution in advancing biomedical interventions, but its application to health promotion/social betterment programs has not been as fruitful as anticipated. Efficacy evaluations, for example, are usually not followed by effectiveness evaluations. Evaluators and researchers have increasingly recognized that in an evaluation, the over-emphasis on internal validity reduces that evaluation’s usefulness and contributes to the gulf between academic and practical communities regarding interventions. This workshop will introduce the bottom-up approach (a sequential route from viability evaluation, effectiveness evaluation, to efficacy evaluation) as an alternative perspective for program evaluation. 

Participants will leave with an understanding of the new approach and its advantages in enabling evaluators to meet scientific and service requirements, facilitating in advancing external validity, gaining a new perspective on research methods, and furnishing a balanced view of credible evidence.

The workshop will provide an opportunity for participants to engage in discussion, share ideas, and learn from each other in a supportive environment.  

Audience: Attendees with a basic background in logic models and/or program theory.

Huey Chen is a senior evaluation scientist at the CDC. He was a professor at the School of Public Health at the University of Alabama at Birmingham until January 2008. Dr. Chen has contributed to the development of evaluation theory and methodology, especially in the areas of program theory, theory-driven evaluations, and evaluation taxonomy. His book Theory-Driven Evaluations has been recognized as one of the landmarks in program evaluation and his newest text, Practical Program Evaluation, offers an accessible approach to evaluation for those working in any context. In 1993 he received the AEA Paul F. Lazarsfeld Award for contributions to Evaluation Theory and in 1998 he received the CDC Senior Biomedical Research Service Award.

Offered:

• Monday, June 14, 2:30 – 4:00 PM

Offering 18: Qualitative Evaluation Approaches

Level: Beginner

Practical qualitative evaluation methods will be shared for community health programs. The workshop will be an overview of qualitative methods that include the use of logic models, social ecology principles, participatory-based strategies, and basic data analysis methods. The session is designed to foster creativity when planning, implementing and evaluating a community-based program. Simulation activities will be used to dramatize many of the concepts.

Audience: Attendees working in any context seeking an introduction to working with qualitative data for evaluation.

Robert M. Goodman is a Professor and Dean of the School of Health, Physical Education and Recreation at Indiana University. Dr. Goodman has written extensively on issues concerning community health development, community capacity, community coalitions, evaluation methods, organizational development, and the institutionalization of health programs. He has been the principal investigator and evaluator on projects for CDC, The National Cancer Institute, The Centers for Substance Abuse Prevention, The Children’s Defense Fund, and several state health departments. In 2004, Dr. Goodman received the Distinguished Fellow Award from the Society for Public Health Education, the highest honor it bestows. Currently, Dr. Goodman is consulting on community-based public health practices and empowerment evaluation with the Diabetes Translation and Injury Prevention Branches at CDC. Also, he is leading an evaluation of community-based approaches to increasing interest in cancer clinical trials.

Offered:

• Monday, June 14, 2:30 – 4:00 PM

Offering 19: Grappling With the Unexpected From Firefighting to Systematic Action (new!)

Level: Intermediate

Description: All evaluators deal with unintended events that foul their evaluation plans. Either the program does not work as planned, or the evaluation does not work as planned, or both. Usually we treat these situations as fires, i.e. we exercise our skills to meet the crisis. This workshop steps back from crisis mode and presents a systematic treatment of why these situations pop up, the continuum from “unforeseen” to “unforeseeable”, tactics that can be used along the continuum, and why caution is needed because anything we do to minimize the effect of surprise may be the cause of yet other difficulties. The intent of the workshop is twofold. First, to provide individual attendees with skills and knowledge they can employ in their own practice. Second, to further a community of interest among evaluators dedicated to developing systematic understanding of the phenomenon of unanticipated events that threaten the integrity of evaluation.

Jonathan A. Morell (Jonny) spends his professional life integrating practitioner and theoretical interests. As a practitioner he evaluates organizational change, R&D, and safety programs. His theoretical interests include unexpected behavior in programs and their evaluations, logic models, continuous process improvement methodologies in evaluation, complex systems, and the nature of practical action. Lately he has been playing with two pet curiosities, the relationship between agent-based modeling and evaluation, and the relationship between the visual forms of logic models and unexpected program behavior. He is the editor of Evaluation and Program Planning and is active in the American Evaluation Association, where he has played a role in establishing the Business and Industry, and the Systems TIGS. He is a recipient of AEA’s Marcus Ingle Distinguished Service Award. Jonny is a Senior Policy Analyst at the Vector Research Center, a division of TechTeam Government Solutions. His PhD in Psychology is from Northwestern University.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

Offering 20: Every Picture Tells a Story: Flow Charts, Logic Models, LogFrames, Etc. What They Are and When to Use Them

Level: Advanced Beginner

Description: A host of visual aids are in use in planning and evaluation. This session will introduce you to some of the most popular ones—with an emphasis on flow charts, logic models, project network diagrams, and logframes. We’ll review the content and format of each one and then compare and contrast their uses so that you can better match specific tools to specific program needs. We’ll review simple ways to construct each type of tool and work through some simple cases both as illustrations and as a way for you to practice the principles presented in the session.

Audience: Assumes prior familiarity with evaluation terminology and some experience in constructing logic models.

Thomas Chapel is a Senior Evaluation Scientist in the Office of Workforce and Career Development, at the Centers for Disease Control and Prevention. He serves as a central resource on strategic planning and program evaluation for CDC programs and their partners. Before joining CDC, Mr. Chapel was Vice-President of the Atlanta office of Macro International where he directed and managed projects in program evaluation, strategic planning, and evaluation design for public and nonprofit organizations. He is a frequent presenter at national meetings, a frequent contributor to edited volumes and monographs on evaluation, and has facilitated or served on numerous expert panels on public health and evaluation topics. Mr. Chapel is active nationally and locally in the American Evaluation Association (AEA), currently as past-chair of the Membership Committee and convener of AEA’s Local Affiliate Collaborative. Mr. Chapel holds a BA degree from Johns Hopkins University and MA in public policy and MBA degrees from the University of Minnesota.

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 16, 9:25 – 12:45 (20 minute break within)

Offering 21: Facing Evaluation Challenges in the Real World: A Case-Based Approach

Level: Beginner

Description: This session will use case study analysis to highlight strategic, ethical, and methodological challenges that evaluation practitioners encounter in the real world and explore effective strategies for meeting those challenges. As a result of class discussion, case analysis and small group activities, participants will be able to:

• Discuss ethical, strategic and methodological challenges associated with evaluation practice

• Propose strategies for meeting those challenges

• Discuss proactive strategies for ensuring an effective and useful evaluation study.

Audience: Novice evaluators working in any context.

Iris Smith holds a doctorate in Community Psychology from Georgia State University and a Master’s Degree in Public Health from Emory University. She is currently an Associate Professor and Director of the Career Master of Public Health Program (CMPH) at Emory University’s Rollins School of Public Health, where she also teaches a graduate level online course in Evaluation Research. Her current evaluation projects include the evaluation of the Atlanta Clinical Translational Science Institute, an interdisciplinary collaboration between Emory University, Georgia Institute of Technology and Morehouse School of Medicine. She is also Co- Director of the Evaluation Core for the Emory Prevention Research Center. Previously, Iris was the Director of National Evaluation Services for the American Cancer Society, and has also served as a Deputy Commissioner for the Georgia Department of Juvenile Justice.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 2:30 – 4:00 PM

• Tuesday, June 15, 2:30 – 4:00 PM

Offering 22: Enhanced Group Facilitation: Techniques and Process

Level: All

Description: This popular and well-received workshop will familiarize participants with the management of the group facilitation process and a variety of group facilitation techniques. Participants will learn best practices for navigating group dynamics and capitalizing on challenging attendees. We will discuss facilitator’s different roles and responsibilities in group facilitation and how these roles intersect with the tasks inherent in planning and managing a group facilitation experience. Participants will also learn how to choose a facilitation technique based on goals and objectives, anticipated outcome, type and number of participants, and logistics. Two to three facilitation techniques for generating ideas, focusing thoughts, and drawing conclusions and next steps will be explored in greater detail. We will also cover variations on these techniques and how they may be used for your facilitation purposes. Job aides and reference lists will be provided.

Audience: Attendees working in any context who work with, or expect to be working with, client groups of any size.

Level: All

Jennifer Dewey is a Senior Research Associate with James Bell Associates, Inc. She is the Project Director for the Family Connection Discretionary Grants evaluation, funded by the Administration for Children and Families (ACF), Children’s Bureau, where she leads a team of JBA staff in designing and implementing a cross-site evaluation protocol, along with providing technical assistance to local evaluation activities. Prior to joining JBA, Dr. Dewey was a Technical Director at ICF Macro, where in addition to business and organizational development, she directed three core studies and oversaw ongoing training and technical assistance to 60+ local evaluation teams for a multi-site evaluation of a SAMHSA-funded systems of care program. Her technical expertise encompasses: project management; proposal development; evaluation design and budgeting; needs assessment; field and survey research; telephone and in-person interviews; group facilitation and training; quantitative and qualitative data collection and analysis; and evaluation reporting.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 23: Using the Guiding Principles to Improve Your Evaluation Practice

Level: Beginner to Intermediate

Description: The Guiding Principles for Evaluators focus on five areas of evaluation practice: systematic inquiry, competence, integrity and honesty, respect for people, and responsibilities for general and public welfare. The Principles guide the professional practice of evaluators, and inform evaluation clients and the general public about the principles they can expect to be upheld by professional practitioners. This session will share ways to use the Principles to improve the ways in which you plan for and conduct evaluations and work with stakeholders and clients. After a brief presentation that introduces the Principles, participants will work together in small groups to discuss the Principles as they relate to a topical case study. Through case explorations, lecture and small and large group discussions, you will gain a deeper understanding of the practical applications of the Principles. The workshop will also introduce resources—print, web-based and collegial networks—that evaluators can consult to handle professional dilemmas that arise in their practice. You will receive copies of the workshop presentation, the case study, the Principles in full and abbreviated brochure format, and a list of resources for more information and consultation.

Audience: Evaluators and commissioners of evaluation working in any context

Leslie Goodyear is Program Officer in the Division of Research on Learning at the National Science Foundation. In addition to grant making, she coordinates evaluation for the division’s programs. As a program evaluator and researcher, Dr. Goodyear has worked with programs focused on HIV/AIDS Prevention; Out-of-School Time; Youth Engagement and Youth Media; Educational Research; and STEM Education. Leslie is a past Chair of the AEA Ethics Committee, past AEA Board member and current section editor for the Ethics Section of the American Journal of Evaluation. She earned her M.S. and Ph.D. in Human Service Studies, with focus on Program Evaluation, from Cornell University.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 2:30 – 4:00 PM

• Tuesday, June 15, 2:30 – 4:00 PM

Offering 24: Transformative Mixed Methods Evaluations (new!)

Level: Beginner

This workshop focuses on the methodological and contextual considerations in designing and conducting transformative mixed methods evaluation. It is geared to meet the needs of evaluators working in communities that reflect diversity in terms of culture, race/ethnicity, religion, language, gender, and disability. Deficit perspectives that are taken as common wisdom can have a deleterious effect on both the design of a program and the outcomes of that program. A transformative mixed methods approach enhances an evaluator's ability to accurately represent how this can happen.

Interactive activities based upon case studies will give you an opportunity to apply theoretical guidance that will be provided in a plenary session, a mini-lecture and small- and large-group discussions. Alternative strategies based on transformative mixed methods are illustrated through reference to the presenters' own work, the work of others, and the challenges that participants bring to the workshop.

You will learn:

• To critically examine the transformative paradigm's assumptions in culturally diverse communities,

• To identify different methodological approaches within a transformative mixed methods model,

• To apply critical skills associated with selecting the design and use of transformative mixed methods evaluation.

Donna Mertens is a Past President of the American Evaluation Association who teaches evaluation methods and program evaluation to deaf and hearing graduate students at Gallaudet University in Washington, D.C. Mertens recently authored Transformative Research and Evaluation (Guilford). Katrina L Bledsoe is a senior research associate at Walter R. McDonald & Associates, conducting and managing evaluations in culturally complex communities nationally. 

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 16, 9:25 – 12:45 (20 minute break within)

Offering 25: Offering 25: Using GIS in Evaluation (new!)

Level: Beginner

Description: At a broad level this workshop will provide an overview of what a geographic information system (GIS) is and ways in which one can be used for evaluative purposes. Specifically, workshop participants will be engaged in an interactive presentation and discussion about how these systems can be used in multiple settings (organizations and schools, communities, states). Learning objectives for participants include:

• Understanding how GIS can be used to develop and answer evaluation questions.

• Learning how GIS data can be used in conjunction with other types of data to approach evaluation questions through multiple methods.

• Becoming acquainted with GIS resources (e.g., websites, software, hardware, archival and pre-existing data sources) which are available to assist practitioners and researchers in obtaining and analyzing GIS data.

• Discussing a “real-world” case example which utilized a multi-method approach (including GIS methods) to evaluate a collaborative community-based project which aims to increase physical activity in high-crime low-income neighborhoods.

This workshop has been developed for participants who have minimal knowledge of GIS and its uses; however, it is assumed that novice GIS users who attend this workshop will be able to broaden their knowledge-base through the interactive discussion and completion of the learning objectives.

Duncan Meyers is a doctoral candidate in the Clinical/Community Psychology program at the University of South Carolina. He got his start in evaluation as the Evaluation Coordinator for a pilot System of Care project which sought to transform the community-based child mental health services of a large metropolitan area. Since that time, Duncan has been an evaluator for a University-based Center for Public Health Preparedness which conducted large scale trainings for local public health practitioners, has been en evaluator of an after-school program which aimed to increase physical activity for middle school students, and has been an evaluator for a community-based project which aims to increase physical activity for adults. Currently, Duncan is the project director for an NIH funded environmentally-based intervention which seeks to increase physical activity in high-crime low-income neighborhoods.

Offered:

• Tuesday, June 15, 2:30 – 4:00 PM

Offering 26: Case Study Methods for Evaluators

Level: Beginner

Description: Case Study Methods allow evaluators to approach program assessment from a powerful and flexible design palette. While often heavily steeped in the use of qualitative methods, case studies also may include the use of quantitative data. The approach is particularly rich for tinting and shading the effects of programs as well as investigating important program questions in depth.

This interactive, three-hour session will provide participants with an overview and examples of case study research methods as they apply to evaluation settings. Through the development and expansion of sample case studies, by the end of the session, participants will:

• Comprehend the role of case study methods within the context of other evaluation approaches

• Be able to describe the elements of case study research and identify the major strengths and weaknesses of case study methods;

• Understand the sequential, operational guidelines for implementing case study research

• Review techniques for establishing the validity and reliability of case study data

• Strengthen data gathering and analysis skills through use of techniques common to case study research

Audience: Attendees working in any context

Rita O’Sullivan is Associate Professor of Evaluation and Assessment at the University of North Carolina at Chapel Hill where she teaches graduate courses in Educational Program Evaluation, Case Study Methods, Research Design, Measurement, and Statistics. She is also Executive Director of Evaluation, Assessment, and Policy Connections (EvAP), a unit she founded within the UNC School of Education that conducts local, state, and national evaluations. Dr. O’Sullivan has specialized in developing collaborative evaluation techniques that enhance evaluation capacity and utilization among educators and public service providers. She is senior author of Programs for At-Risk Student: A Guide to Evaluation (Corwin Press, 1993) and wrote Practicing Evaluation: A Collaborative Approach (Sage) in 2004.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 27: Translation of Evaluation Capacity Building Strategies to Community BasedOrganizations Conducting Health Promotion Activities: Tools, Tips and Lessons Learned

Level: Beginners

Description: National public health disparities often require local, community-based approaches to influence behaviors and policies. While community-based organizations serve as catalysts for prevention and health promotion activities, many do not consistently practice program evaluation due to the challenges of limited time, staff and measurement skills. This session will describe how to 1) develop an audience targeted evaluation capacity building plan, 2) choose appropriate tools and mediums for evaluation capacity building, 3) engage in bi-directional learning on how to couple community engagement and evaluation approaches and 4) navigate challenges in evaluation capacity building partnerships.

Audience: Attendees working with community-based public health initiatives to conduct evaluations, who offer evaluation capacity building or technical assistance in program assessment.

Tabia Henry Akintobi is a Research Assistant Professor at the Morehouse School of Medicine and Director of Evaluation for its Prevention Research Center. She evaluates The Atlanta Clinical and Translational Service Institute Community Engagement and Research Program designed, in part, to engage academicians and community is collaborative research. She provides evaluation or capacity building for programs addressing infrastructure development, health outcomes and service delivery in the areas of maternal and child health, substance abuse, mental health, HIV/AIDS and teen pregnancy. She led assessment of the Pfizer Foundation Southern HIV/AIDS Prevention Initiative and the Southeastern Collaborative Center of Excellence for the Elimination of Health Disparities. She is Chairperson for The National Prevention Research Center Evaluation Advisory Committee.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 2:30 – 4:00 PM

• Tuesday, June 15, 2:30 – 4:00 PM

Offering 28: The Democratization of Data Insight: 20 Free Tools for Data Access, Analysis, and Reporting and Considerations for Their Use (new!)

Level: Beginner

Description: Join us for a look at the range of free tools available for accessing, analyzing, and reporting on data of use to evaluators in a range of contexts. We'll examine options for exploring large public data sets such as census and government databases; for understanding data from social media produced by RSS feeds from sources such as blogs and twitter accounts; for analyzing and understanding website traffic and communications; and for analyzing and sharing your own qualitative and quantitative data sets. The workshop will be broken into two parts: first, we'll look at the range of tools available, including reviewing examples from the interface, input, and output of each; then we'll get the participants involved and discuss issues related to the democratization of data analysis - from considerations of privacy and transparency to examination of increasing the capacity of end-users to understand analyses. You'll leave with a basic understanding of 20+ tools and information on how to access and learn more about each as well as a knowledge of issues to consider before their use. Plus, we'll explore two tools in a bit more depth that can help you to empower stakeholders to perform their own analyses.

Audience: Attendees working in any context

Susan Kistler is the Executive Director of the American Evaluation Association and owner of iMeasure Media. She has taught statistics, research methods, and evaluation at the university level and is an experienced trainer and speaker for local, regional, and national audiences. For this workshop, she has built upon her traditional evaluation and qualitative and quantitative analysis background, as well as a passion for empowering stakeholders to understand and explore data.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 2:30 – 4:00 PM

• Tuesday, June 15, 2:30 – 4:00 PM

Offering 29: Conducting and Using Success Stories for Capacity Building

Level: Intermediate

Description: In order to build program capacity, a programs “success” must be told at many levels. In addition, impacts of prevention programs may not be able to be demonstrated for several years therefore communicating success during the various life stages of a program is important for long term sustainability. The presenters will use their experience with various public health programs to demonstrate how to use success stories to build both program and evaluation capacity. The session will be a practical and hands on session enabling attendees to begin writing their own success stories. This session includes time for practicing practical applications for use in your own practice. Attendees will receive the newly developed workbook: Impact and Value: Telling Your Program’s Story for use during the class and to take home for reference.

Audience: Attendees working in any context with a working knowledge of both evaluation and qualitative inquiry

René Lavinghouze is a Senior Evaluation Scientist in the Office of Smoking and Health at CDC where she is the lead on the evaluation of recovery act programs. Rene has over 21 years experience with CDC and in the private sector. She is Chair of AEA’s TIG for Cluster, Multi-site/level evaluations. Ann Price is president of Community Evaluation Solutions, Inc and has over 20 years experience in both treatment and prevention. She has conducted evaluations in many areas including intimate partner violence, mental health, substance abuse, tobacco prevention and oral health. Prior to CES, Dr. Price was a Senior Data Analyst at ORC Macro on a multi-site national child mental health evaluation.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 16, 9:25 – 12:45 (20 minute break within)

Offering 30: An Introduction to Economic Evaluation

Level: Intermediate

Description: Economic evaluation refers to applied analytic methods used to identify, measure, value, and compare the costs and consequences of programs and interventions. This course provides an overview of these methods, including cost analysis, cost-effectiveness analysis (CEA), and cost-benefit analysis (CBA) with an opportunity for hands-on application of each. You will leave understanding when and how to apply each method appropriately in a range of evaluation contexts.

Audience: Attendees with a basic background in statistical methods and understanding of evaluation.

Phaedra S. Corso is currently an Associate Professor in the Department of Health Policy and Management at the University of Georgia College of Public Health. Previously, she served as the senior health economist in the National Center for Injury Prevention and Control at CDC where she worked for over a decade in the areas of economic evaluation and decision analysis, publishing numerous articles on the cost-effectiveness of prevention interventions and co-editing a book on prevention effectiveness methods in public health. She holds a Master’s degree in public finance from the University of Georgia and a Ph.D. in health policy and decision sciences from Harvard University.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 31: Methods for Analyzing Change Over Time

Level: Intermediate

Description: We will focus on a variety of methods for analyzing change in outcomes over time, including the traditional fixed effects methods of pre/posttest ANCOVA and Repeated Measures ANOVA; the slopes-as-outcomes individual regression analysis approach; and multilevel modeling and random coefficients models. The purpose of the workshop is to explore the conceptual underpinnings of these different approaches to assessing change, and to compare the kinds of statistical information one is able to glean from these types of analyses when addressing questions of change. We will discuss what it means to measure change, how each method attacks that task, and how to determine which measure to use in a given situation since each method has its strengths and weaknesses with respect to its conceptual approach, parameter estimation, precision of estimates and handling missing data. Due to the nature of the topic, the majority of the workshop will involve presentations. Conceptual information, statistical output and graphs will be shared in a give-and-take format, where participants bring their own questions and concerns about analyzing change over time. Demonstration of how to set up longitudinal data for the different analytical methods will be included as well as interpreting statistical output.

Audience: Attendees with a good understanding of General Linear Models (including the ANOVA family and MRC) and some basic experience with longitudinal analysis.

Katherine McKnight has taught statistics workshops at AEA's annual conferences for many years and began with the Institute in 2007. An outstanding facilitator, she is known for making difficult concepts accessible and interesting.

Offered:

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

Offering 32: Improving Survey Quality: Assessing and Increasing Survey Reliability and Validity

Level: Intermediate

Description: Develop higher-quality surveys! This workshop is designed to teach participants how to improve survey quality, thus increasing their utility of and confidence in the data they collect. We will look at surveys to elicit factual information as well as ones that ask about subjective and abstract concepts. Through the use of hands-on activities, mini-lectures, and demonstrations participants will understand what is meant by reliability and validity with respect to surveys and will learn ways to improve each during the survey design phase for both types of surveys. Next, using a case example and SPSS we will explore ways to use pilot test responses to assess the reliability of subjective / abstract survey constructs by conducting confirmatory factor analysis and calculating Cronbach’s alpha. We will work together to understand what our findings tell us as well as what they don’t tell us, and consider other ways to assess survey quality. Last we will explore the types of validity associated with surveys and ways to assess the various forms of validity again using our case example. You will receive a workbook and SPSS screenshots to help you remember how to perform many of the computations we will perform. Participants will be surprised by how easy it is to improve survey quality through a few easy to implement steps!

Audience: Attendees working in any context with a basic background in survey development and an understanding of factor analysis.

Amy A. Germuth earned her PhD in Evaluation, Measurement and Psychology from UNC Chapel Hill and a certificate in survey methodology via a joint program through UNC-CH, UMD, Westat and RTI. She is a founding partner of Compass Consulting Group, LLC, a private evaluation consulting firm that conducts evaluations at the local, state, and national levels. As part of Compass she has evaluated numerous initiatives, including health prevention and outreach programs, Math Science Partnerships, K-12 science outreach programs, and workforce development initiatives, and has worked with a variety of organizations including the US Education Department, Department of Health and Human Services, Westat, Georgia Tech., Virginia Tech., University of North Carolina, the New York State Education Department, multiple NC Childhood Education Partnerships, and Hawaii’s Kamehameha Schools. As part of her evaluation work, she has developed and guided large-scale survey initiatives. Dr. Germuth teaches evaluation and instrument development as part of Duke University’s Certificate Program in Non-Profit Management.

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 16, 9:25 – 12:45 (20 minute break within)

Offering 33: An Executive Summary is not Enough: Effective Reporting Techniques for Evaluators (new!)

Level: Beginner

Description: As an evaluator you are conscientious about conducting the best evaluation possible, but how much thought do you give to communicating your results effectively? Do you consider your job complete after submitting a lengthy final report? Reporting is an important skill for evaluators who care about seeing their results disseminated widely and recommendations actually implemented. This interactive workshop will present an overview of three key principles of effective reporting and engage participants in a discussion of its role in effective evaluation. Participants will leave with an expanded repertoire of innovative reporting techniques and will have the opportunity to work on a real example in groups.

You will learn:

• The role of communication and reporting in good evaluation practice

• Three key principles for communicating results effectively

• Four innovative reporting techniques

Audience: Evaluation practitioners of all levels in all sectors.

Kylie Hutchinson is the principal of Community Solutions Planning & Evaluation, a consulting firm specializing in evaluation, strategic planning, and program sustainability. She has over twenty years experience in the field of evaluation and has conducted numerous evaluations in the areas of health, human services, and others. In addition to her direct consulting practice, Kylie is a popular trainer known for her engaging and informative workshops on evaluation. Since 2005 she has delivered the Canadian Evaluation Society’s Essential Skills Workshop Series in British Columbia. In 2008 and 2009 she was a pre-conference workshop presenter and poster judge for the American Evaluation Association’s annual conference. She is also a contract instructor with the Justice Institute of British Columbia’s Centre for Leadership. In 1990 she received an award from the Canadian Evaluation Society for her work with the Heart and Stroke Foundation of Ontario.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 9:25 – 12:45 (20 minute break within)

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 34: Project Management Fundamentals – The Key to Implementing Evaluation Projects

Level: Intermediate

Description: This highly experiential session is focused on how to implement an evaluation effort through a structured project management methodology. The workshop will focus on initiating, planning, executing, and monitoring and controlling an evaluation project. Participants will gain an understanding of how the project management process can be used effectively to add value to evaluation projects, and will also gain an understanding of how to generate a project plan and manage an evaluation project. At the conclusion of the session, participants will be able to:

• Describe the documents that define an evaluation “project plan”

• Write a project charter (mission statement) for an evaluation

• Develop a Project scope statement for an evaluation project

• Interpret a graphic picture of a project via a network diagram and Gantt chart

• Generate a project schedule and describe the project critical path

Audience: Participants with responsibilities for evaluation implementation, as well as individuals with oversight responsibilities for evaluation projects. Participants already participating in program evaluation will benefit most from the session.

Richard H. Deane has over 30 years experience in the application of structured project management techniques in both the private and public health sectors. He has taught and won numerous teaching awards at Purdue University, Georgia Tech and Georgia State University. Richard has been a highly acclaimed instructor at the CDC University for many years, and his project management consulting assignments over the past 25 years have included work with various agencies within t?he CDC, state public health departments, private health organizations, federal agencies, and numerous private sector corporations.

Offered:

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

Offering 35: Idea Exchange: Evaluation of Social Media Interventions (new!)

Level: Beginner

Description: This is not your traditional workshop, rather it is an opportunity to engage in discussion, share ideas, and learn from each other in a collaborative and supportive environment. We’ll frame 3-4 big picture issues, and then turn to the audience to ask how you are responding to the challenge of measuring social media initiatives – What are you measuring? How are you measuring it? How do you define success? How do you measure not just inputs and click-throughs but impact and change? Today’s keynote speaker Allison Fine and AEA Executive Director Susan Kistler will act as facilitators and discussants, and we’ll walk the talk by recording and sharing our lessons learned via social media channels.

Audience: Those working in public health contexts.

Allison H. Fine is the founder of Innovation Network, a nonprofit dedicated to transforming evaluation for social change. She is the author of the award-winning book Momentum: Igniting Social Change in the Connected Age (Jossey - Bass/John Wiley & Sons, 2006) and a senior fellow on the democracy team at Demos: A Network for Change and Action in New York City. In 2008, she published a paper on young people and activism commissioned by The Case Foundation called “ Social Citizens beta ,” and co-edited a collection of essays, Rebooting America: Ideas for Redesigning American Democracy for the Internet Age, published by the Personal Democracy Forum (2008), about transformative ways to reinvent twenty first-century democracy using new media tools. Allison hosts a monthly podcast for The Chronicle of Philanthropy called “Social Good ” and writes her own blog, A. Fine Blog. Her newest book, The Networked Nonprofit, with Beth Kanter, is due out in June of 2010 from John Wiley and Sons.

Offered:

• Monday, June 14, 2:30 – 4:00 PM

Offering 36: (Dawkins) Evaluability Assessments: Achieving Better Evaluations

Level: Beginner

Description: Though rigorous evaluation is a valuable method, in practice it also proves costly and time-consuming. Further, rigorous evaluation is not an appropriate fit for every initiative. Evaluability assessments (EAs) offer a cost-effective technique to help guide evaluation choices. Developed by Joseph Wholey and colleagues three decades ago (1979), EA received significant attention at the time but subsequently displayed diminished activity (Rog 1985; Trevisan 2007). But EAs help answer critical questions evaluators continue to face in practice: Is a program or policy ready for rigorous evaluation? What are viable options for evaluating a particular initiative? EAs involve clarifying program goals and design, finding out stakeholders’ views on important issues, and exploring the reality of a given initiative. In short, EAs are a valuable and important tool to have in an evaluator’s toolbox. This workshop will provide participants an understanding of EAs and how they can be applied in their own practice.

Audience: Those working in any context.

Nicola Dawkins received her PhD in sociology and Masters in Public Health from Emory University. She serves as a Principal at ICF Macro where she designs and implements numerous research and evaluation studies. Among these is the Coordinating Center for the Early Assessment of Programs and Policies to Prevent Childhood Obesity—a project that employed a cluster evaluability assessment methodology to examine multiple initiatives. Nicola oversees other individual evaluability assessment and evaluation projects as well.

Offered (Two Rotations of the Same Content - Do not register for both):

• Tuesday, June 15, 9:25 – 12:45 (20 minute break within)

• Wednesday, June 16, 9:25 – 12:45 (20 minute break within)

Offering 38: Bringing Funders, Researchers/Evaluators, and Practitioners Together to Achieve Outcomes: The Interactive Systems Framework (ISF) for Dissemination and Implementation (new!)

Level: All

Description: If we are to achieve better outcomes in public health, education, and other social programs, we need new models and frameworks for planning, implementing, and evaluating evidence-based approaches. It is essential to: 1) integrate research to practice models with community-centered models (practice-centered models); and 2) bring funders, researchers/evaluators, and practitioners together in a partnership for results-based accountability. In collaboration with Divisions at CDC, the Interactive Systems Framework for Dissemination and Implementation (ISF) was developed and is being utilized to bridge the gap between research and practice and to support evidence-based practices (Wandersman et al, 2008; Lesesne et al, 2008). The ISF has as its foundation three interacting systems that together offer a framework for bridging research and practice. The systems are: synthesis and translation of programs, policies, and principles; prevention support; and prevention delivery. The ISF is designed to illuminate the perspectives and contributions of funders, practitioners, and researchers/evaluators. A CDC example of the use of the ISF to promote science-based approaches to teen pregnancy prevention using Getting To Outcomes® will be described. Additional examples of ways to use the ISF in planning, implementation, evaluation, and overall program design will also be described.

Audience: Attendees working in any context of prevention, treatment, or education

Abraham Wandersman is a Professor of Psychology at the University of South Carolina-Columbia. He received his doctorate from Cornell University. Dr. Wandersman performs research and program evaluation on citizen participation in community organizations and coalitions and on interagency collaboration. He is a co-author of Getting To Outcomes (winner of the 2008 AEA Outstanding Publication Award) and a co-editor of Empowerment Evaluation: Principles in Practice. He is a co-author of the article that established the Interactive Systems Framework for Dissemination and Implementation.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 2:30 – 4:00 PM

• Tuesday, June 15, 2:30 – 4:00 PM

Offering 39: Writing Questions that Elicit What You’re Looking For

Level: All

Description: There is a lot to know when constructing a productive survey. Participants will be reminded of what they already know and will learn many non-obvious but critical considerations as they draft a survey relating to their field of interest. The principles underlying valid, useful, and reliable open-ended and fixed-choice questions will be discussed as well as additional considerations involved in aggregating questions on a survey. We will explore (a) when to use open- versus close-ended questions, including issues of feasibility of analysis; (b) the range of question types, such as yes/no, multiple choice, scales, ranking, short answer, and factors that influence selection; (c) question ordering and its impact on response; and (d) careful wording to avoid common question development pitfalls.

Audience: Attendees working in any context.

Angelika Pohl is founder and President of Atlanta-based Better Testing & Evaluations. Dr. Pohl has extensive professional experience in education and in all aspects of K-12 educational testing and evaluation. After a career of college and graduate-level teaching and research, she was Project Director at National Evaluation Systems (now a part of Pearson), the leading developer of customized teacher certification tests. Before launching her consulting firm, in which she works mostly with K-12 teachers, public school systems, and related educational organizations, she worked for the Georgia Dept. of Education where she was responsible for several testing programs. Her work with Better Testing & Evaluations focuses on producing information that is useful to all constituencies and developing data-driven blueprints for strengthening programs.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 2:30 – 4:00 PM

• Tuesday, June 15, 2:30 – 4:00 PM

Offering 40: Evaluating Environmental Change Strategies…Beyond Counting the Number of Laws Passed

Level: All

Description: The purpose of this session is to explore the challenges associated with evaluating environmental change strategies and identify methods to assess the effectiveness of these strategies. As a point of clarification, for purposes of this session, environmental change strategies refer to population-based interventions that aim to change the environment or context in which individuals make decisions and not issues related to the evaluation of natural resources, biodiversity, and conservation. With the increase in the translation of research to policy and other environmental change strategies, it is imperative that we understand the most appropriate methodological tools available to analyze the effects, and outcomes of these strategies.

During this session participants will develop a working definition of environmental change strategies, discuss examples of environmental change strategies, identify challenges associated with evaluating environmental change strategies and build a tool-kit for addressing these challenges. The presenter will use a mix of lecture, group discussion and hands-on activities in conducting the session.

Kristi Pettibone is Director of The MayaTech Corporation’s Center for Community Prevention and Treatment Research. She has over 10 years experience managing, directing, and guiding public health policy research and evaluation projects. Dr. Pettibone evaluated community-based environmental change strategies as part of a cross-site evaluation of the Center for Substance Abuse Prevention’s (CSAP) Strategic Prevention Framework-State Incentive Grant initiative. Dr. Pettibone has also worked on several evaluations for CDC’s National Center for Injury Prevention and Control and provided technical assistance in evaluating environmental change strategies to CDC’s Heart Disease and Stroke Prevention grantees. She holds a Ph.D. in Policy Sciences (health policy concentration) from the University of Maryland, Baltimore County.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 2:30 – 4:00 PM

• Tuesday, June 15, 2:30 – 4:00 PM

Offering 41: Gaining Support for Evaluation from Stakeholders

Level: Beginner

Description: Stakeholder involvement and support is critical in conducting program evaluations, and is the first step in CDC’s Framework for Evaluating Public Health Programs. And yet, many stakeholders are resistant to or ambivalent about evaluations. During this session participants will explore reasons for stakeholder resistance to evaluation and will identify some of the misconceptions stakeholders may have regarding evaluation. Participants will develop strategies for gaining support of stakeholders and will brainstorm approaches for involving stakeholders in the evaluation process, since a participatory approach ultimately increases acceptance and relevancy of the evaluation. The presenter will use a mix of lecture, group discussion and hands-on activities in conducting the session.

Suzanne Randolph is Chief Science Officer at The MayaTech Corporation. She has more than 25 years of expertise in survey, experimental, and evaluation research designs applied to the study of educational, social and human services, and public health programs and policies. Dr. Randolph works with non-profits and foundations as well as with large, Federal and State clients and is experienced in gaining stakeholder support for evaluations. Dr. Randolph is a retired, tenured Associate Professor of Family Science at University of Maryland, College Park's School of Public Health. She was research coordinator and faculty member of the Howard University School of Medicine, director of evaluation for HIV prevention programs at the American National Red Cross, and a Fellow in the W.K. Kellogg National Fellowship Program. Her B.S. in Psychology is from Howard University, and her Ph.D. in psychology is from the University of Michigan, Ann Arbor.

Offered:

• Tuesday, June 15, 2:30 – 4:00 PM

Offering 42: How to Measure What Matters: A Practical Introduction to Performance Measure Development

Level: Beginner

Audience: Social science researchers and evaluation practitioners

Description: Recent years have witnessed an increasing emphasis on accountability and transparency for public health programs, resulting in a renewed focus on performance measurement. While many public health programs have a strong foundation grounded in empirical research potentially facilitating measurement development, newer programs such as public health preparedness present unique challenges in identifying and developing meaningful and reliable measures. This course provides a practical introduction to the application of CDC’s Program Evaluation Framework and other social science methodologies to the development of a standardized measurement system for CDC’s public health emergency preparedness program. At the completion of this course, participants will be able to 1) discuss the key steps in a systematic approach to performance measure development, including the limitations and benefits of each step, 2) identify appropriate stakeholders for engagement throughout the development process, and 3) apply evaluation tools and methods to develop new performance measures.

Craig Thomas is Chief of the Outcome Monitoring and Evaluation Branch in the Office of Public Health Preparedness and Response at the Centers for Disease Control and Prevention. He has the primary responsibility for determining the broad direction of health services research and program evaluation activities related to the Public Health Emergency Preparedness Program. Anita McLees is Acting Science and Evaluation Team Lead in the Branch. She works on several measurement and evaluation projects, including leading the Public Health Emergency Preparedness (PHEP) Measurement Development Project, collaborating with partners at all levels to develop and implement a standardized national measurement system. Karen Mumford is Acting Data Analysis and Research Team Lead in the Branch. She assists in a variety of evaluation projects including the development and implementation of the performance measures for the PHEP Cooperative Agreement.

Offered (Two Rotations of the Same Content - Do not register for both):

• Monday, June 14, 2:30 – 4:00 PM

• Tuesday, June 15, 2:30 – 4:00 PM