KEYNOTES

Keynote: Attributing “Cause” in Complex Programs: If We Want More Evidence-Based Practice, We Need More Practice-Based Evidence

The evidence-based medicine movement since the 1970s has insinuated its way into American public health thinking, first via the Canadian Task Force on the Periodic Health Examination and the U.S. Preventive Services Task Force, then through the Cochran Collaboration and most recently the US Task Force for Community Preventive Services. A tenet of these systematic reviews leading to the identification and recommendation of “best practices” is that they must give greatest weight to evidence that attributes the causal strength of interventions based on randomized controlled trials (RCTs). The pursuit of randomization and other controls in RCTs has produced evidence that is strong on internal validity, but often weak on external validity. The traditions of evaluation of actual programs in vivo and in real time, rather than in situ under artificially controlled circumstances, promises greater relevance of the findings for practice and greater external validity in generalizing the findings to other “real world” practices and populations. This presentation will examine ways to supplement “best practices” from strong evidence with “best processes” of applying the always-limited evidence in real world settings and diverse populations, and ways that systems modeling of outcomes (“in sylico”) might help bridge the gaps.

Prof Lawrence W. Green co-leads the Social and Behavioral Sciences Program at the University of California at San Francisco Comprehensive Cancer Center. He recently retired from CDC as Distinguished Fellow-Visiting Scientist and Director of the Office of Science and Extramural Research. He has served on the medical and public health faculties at Berkeley, Johns Hopkins, Harvard, Texas, UBC, Emory, and Maryland Universities. He is a past President and Distinguished Fellow of the Society for Public Health Education, recipient of the American Public Health Association's highest awards, the Distinguished Career Award and Award of Excellence, and the American Academy of Health Behavior Research Laureate Medal.

Click here for the handouts! 

Attendees might also be interested in the handouts from Larry Green's Training Session.

Keynote: How to Work With FORECAST and Logic Models as an Ecological Empowerment Evaluation Strategy 

The Formative Evaluation, Consultation, and Systems Technique (FORECAST) is a participatory approach to the evaluation of complex community initiatives.  FORECAST is an empowerment evaluation method because community stakeholders are part of the design, planning, implementation, and reflection of findings in the evaluation process.  Attendees at the plenary session will receive an overview of the FORECAST method by illustration of its application to an actual community-based project to prevent first time male sexual violence against adolescent women.  The evaluation was funded by the Centers for Disease Control and Prevention Division of Violence Prevention.  The program example will illustrate how FORECAST uses program logic models that are developed locally as a means for developing evaluation markers and measures that are employed in real time in order to provide the program with timely feedback for continuous quality improvement.  The FORECAST approach will be introduced by a discussion that illustrates why alternative evaluation approaches to randomized, controlled and quasi-experimental designs often are more appropriate for the evaluation of complex community programs.

Robert M. Goodman, Ph.D., M.P.H., M.A., is a Professor and Chair, Department of Behavioral and Community Health Sciences, Graduate School of Public Health, University of Pittsburgh. Dr. Goodman has written extensively on issues concerning community health development, community capacity, community coalitions, evaluation methods, organizational development, and the institutionalization of health programs. He has been the principal investigator and evaluator on projects for CDC, The National Cancer Institute, The Centers for Substance Abuse Prevention, The Children’s Defense Fund, and several state health departments. In 2004, Dr. Goodman received the Distinguished Fellow Award from the Society for Public Health Education, the highest honor it bestows. Currently, Dr. Goodman is consulting on community-based public health practices and empowerment evaluation with the Diabetes Translation and Injury Prevention Branches at CDC. Also, he is leading an evaluation of community-based approaches to increasing interest in cancer clinical trials.

Click here for the handouts!

Keynote: Exploratory Evaluation: Efficient Allocation of Evaluation Resources

Exploratory evaluation, also known as evaluability assessment, is a pre-evaluation technique that can improve the chances of obtaining useful information from evaluation. It has been extensively used in health and education programs and is becoming increasingly common in recent years. This presentation will outline the technique and illustrate with applications at federal, state and local levels. The participant will gain a better understanding of the advantages and limitations of this technique.

Laura C. Leviton, Ph.D. is a Senior Program Officer of the Robert Wood Johnson Foundation.  She has overseen evaluations in most of the areas of focus for the Foundation, and now works primarily on initiatives in preventing childhood obesity and in improving the quality of medical care.  Before joining the Foundation she was a Professor of Public Health at University of Alabama at Birmingham (UAB) and before that, on faculty of the University of Pittsburgh School of Public Health.  Dr. Leviton is a leading writer on evaluation methods and practice, in particular for disease prevention.  She was President of the American Evaluation Association in the year 2000, coauthored a leading evaluation text, and serves on several editorial boards for evaluation journals.  She received her Ph.D. in social psychology from University of Kansas and postdoctoral training in evaluation research from Northwestern University.

Click here for the handouts!

MONDAY  TRAININGS

Using the Balanced Scorecard: A Tool in Performance Monitoring

Level: All

The emphases on accountability and continuous quality improvement in private, public, and non-profit sectors has created renewed interest  in performance measurement and systems of “dashboard metrics.”  Yet, unless these are linked organically to an organization’s mission and strategy, time and effort put into measurement will not pay off in program improvement.  This session will showcase the Balanced Scorecard as an approach to translating organizational mission and strategy into a strategic measurement and management system. The session will provide an overview of the Balanced Scorecard approach, which assesses functions across four “balanced” perspectives: financial, customers, internal business processes, and learning and growth, and discuss the necessary modifications to the approach to make it most useful for public and non-profit organizations. The presentation will allow the participants to better understand and implement the Balanced Scorecard process, with specific emphasis on: effectively planning the Balanced Scorecard process and exercising flexibility in adapting the process to your performance management framework;  identifying the critical performance measures, targets and initiatives,  and creating and sustaining overall Balanced Scorecard success.

Harry E. Chambers is President of Trinity Solutions, Inc., an Atlanta-area consulting and training firm specializing in organizational development, leadership, and team coaching and building. He is a nationally-known presenter and trainer, and an award-winning author on topics in leadership and management, most recently, No Fear Management:  Rebuilding Trust, Performance, and Commitment in the New American Workplace and The Bad Attitude Survival Guide. Harry has trained, facilitated and consulted with leaders of America’s top corporations as well as public sector and non-profit agency leadership. He is a frequent presenter at Inc. Magazine’s Leadership Development Conferences and his work has been featured on the web sites of CNN FN and Business Week OnLine. His articles, excerpts, and direct quotes appear in many publications and journals, including Executive Excellence, Investors Business Daily, Success and Cosmopolitan magazines. He has been interviewed on NBC’s Today Show, ABC’s Morning News and National Public Radio’s Morning Edition.

Click here for the handouts!

Evaluation 101: An Overview for New Evaluation Practitioners

This training session is full. Registration is no longer available for this session.

Level: Beginner
 
This course will provide an introduction to program evaluation and evaluation terminology for Institute participants with little or no prior background in program evaluation. The teaching framework for the course will be the Centers for Disease Control and Prevention’s (CDC) six-step Framework for Program Evaluation in Public Health as well as the four sets of evaluation standards from the Joint Commission on Evaluation Standards. The course will touch on all six steps of the CDC Framework but will emphasize the early steps, including identification and engagement of stakeholders, creation of logic models, and selecting/focusing evaluation questions. Several public health case studies will be used both as illustrations and as an opportunity for participants to apply the content of the course.

Thomas Chapel, M.A., M.B.A., is a Senior Health Scientist in the Office of the Director, Office of Program Planning and Evaluation at the Centers for Disease Control and Prevention. He is responsible for helping CDC programs build evaluation capacity by providing training and technical assistance on strategic planning and evaluation design; building networks on evaluation and planning across CDC; and providing a range of training and facilitation in strategic planning and evaluation design. Before joining CDC, Mr. Chapel was Vice-President of the Atlanta office of ORC Macro where he directed and managed projects in program evaluation, strategic planning, and evaluation design for public and non-profit organizations. He is a frequent presenter at national meetings and facilitated and serves on expert panels on a host of public health and evaluation topics. He is active in the local affiliate of the American Evaluation Association (AEA), currently as president and formerly as chair of the Professional Development Committee. Mr. Chapel holds a BA degree from Johns Hopkins University and MA in public policy and MBA degrees from the University of Minnesota.

Click here for the handouts!

Enhanced Group Facilitation: Techniques and Process

This training session is full. Registration is no longer available for this session.

Level: All

This workshop will familiarize participants with a variety of group facilitation techniques as well as the management of the facilitation process. Participants will learn how to choose a facilitation technique based on goals and objectives, anticipated outcome, type and number of participants, and logistics. Two to three facilitation techniques for generating ideas and focusing thoughts on both defined and undefined issues will be explored in greater detail. These techniques, and their variations, will include focus groups, item writing, and nominal group technique. Participants will also learn more about the different roles and responsibilities they may have in group facilitation (there are more than you think!), and how these roles intersect with the tasks inherent in planning and managing a group facilitation experience. Job aides and reference lists will be provided.

Jennifer Dewey, Ph.D., is a Technical Director with the research and evaluation professional services firm of ORC Macro. Jennifer leads Site Management, overseeing ongoing training and technical assistance to local evaluation teams for the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program. Prior positions include Director of Internal Evaluation at Learning Point Associates, Senior Consultant at Andersen, and post-doctoral scholar at the Center for Prevention Research at the University of Kentucky. Jennifer holds a doctorate in Applied Experimental Psychology with a specialization in program evaluation. Her knowledge and skills encompass project management, proposal development, methodological and statistical design, qualitative and quantitative analysis, needs assessment, survey  development, telephone and in-person interviews, and group facilitation. Jennifer has published in the Journal of Primary Prevention, American Journal of Evaluation, Advances in Developing Human Resources, and has made over 40 professional conference presentations.

Click here for the handouts!

Exploring Diverse Approaches to Causal Attribution and Practice-Based Evidence

(Register for: Causal Attribution and Practice-Based Evidence)

Level: Intermediate

Following on the keynote presentation in the previous hour, this interactive discussion session will provide opportunities to explore the experience of evaluators and other participants in constructing evidence and using evidence together with theory, professional experience, and indigenous colloquial knowledge in the planning of programs and the design of interventions. The gaps inevitably found between science and practice, and the fit of scientific findings with the situations in which such findings would seem to recommend specific “best practices,” will be examined to discover ways in which such gaps might be closed and the fit improved. We will challenge the conventions of inferring “best practices” from specific types of systematic literature review criteria, and the peer review mechanisms that support their perpetuation in funding and publication decisions. We will explore some approaches to assuring greater external validity in the decisions leading to the funding and publication of research, giving greater weight to context and to moderating variables to balance the current emphasis on internal validity, context-free universal truths implied by the weight given to controlled trial evidence.

Professor Lawrence W. Green co-leads the Social and Behavioral Sciences Program at the University of California at San Francisco Comprehensive Cancer Center. He recently retired from CDC as Distinguished Fellow-Visiting Scientist and Director of the Office of Science and Extramural Research. He has served on the medical and public health faculties at Berkeley, Johns Hopkins, Harvard, Texas, UBC, Emory, and Maryland Universities. He is a past President and Distinguished Fellow of the Society for Public Health Education, recipient of the American Public Health Association's highest awards, the Distinguished Career Award and Award of Excellence, and the American Academy of Health Behavior Research Laureate Medal.

Click here for the handouts!

Sampling 101: Basics of Probability and Purposeful Sampling

Level: All

Choosing and implementing an appropriate sampling strategy can affect the validity, credibility and cost of an evaluation.  Some studies require sophisticated probability sampling methods to produce accurate estimates of the characteristics of the populations served or of the size of the effects of the program or policy on the target population.  Other studies may appropriately use purposeful samples to support theory development or to do detailed case analysis.  In this workshop, participants will be exposed to alternative sampling strategies that are frequently used in evaluation and social research.  In the workshop, the instructor will address the 14 questions from his book Practical Sampling (Sage, 1990) that should be answered prior to sample design, as a part of sample design, and prior to analysis of the data.  The participants will become acquainted with ways to plan and implement sampling strategies that meet the needs of an evaluation.  Examples will be used to illustrate the designs and issues that arise in implementation.  Participants will have the opportunity to raise specific sampling issues that they have encountered in their own work.

Gary T. Henry is a professor in the Andrew Young School of Policy Studies at Georgia State University.  He previously served as the Director of Evaluation and Learning Services for the David and Lucile Packard Foundation.  Henry has evaluated a variety of public policies and programs and has published extensively in the field of evaluation and education policy.  He received the Evaluation of the Year Award from the American Evaluation Association in 1998 for his work with the Georgia’s Council for School Performance and the Joseph S. Wholey Distinguished Scholarship Award in 2001 from the American Society for Public Administration and the Center for Accountability and Performance.  Dr. Henry currently serves on scientific peer review panels for the Institute of Education Sciences, U.S. Department of Education and on a National Research Council/National Academy of Science panel assessing the effects of “green schools” on the health and productivity of teachers and students.

Click here for the handouts!

Systems Thinking and Public Health

This training session is full. Registration is no longer available for this session.

Level: Intermediate 

The persistent challenges that show up most in public health often share common characteristics. They defy quick fixes, they build and dissipate slowly over time, they are rarely “owned” by any one player in the system but affect all, and their components are tightly coupled. The analytical approaches of systems thinking, which was invented in the 1950s at MIT, was created to address such “systems” problems. Systems thinking uses diagramming and simulation modeling to help groups of people improve their understanding of how to improve the performance of a range of social/physical systems such as a city, an ecosystem, an industry, or, in this context, a population facing a threat to their health. In this interactive session, participants will learn about a systems thinking perspective on public health, learn about causal mapping techniques, learn about insights generated by a computer-based “management flight simulator” created with the Division of Diabetes Translation at CDC, and have the opportunity to reflect on applying systems thinking to the public health challenge they face.

Drew Jones is a project director with Sustainability Institute. His work focuses on applying systems thinking approaches to a wide range of societal concerns, particularly in the public health and environmental areas. Working with CDC’s Division of Diabetes Translation, he has led the development of the Diabetes System Model, which is shaping diabetes strategies at the national and state level. A paper on the work was recently published in the March 2006 issue of AJPH. Mr. Jones received a B.A. in Engineering Sciences and Environmental Studies from Dartmouth College and earned a M.S. from M.I.T., where he studied System Dynamics simulation modeling under John Sterman and Peter Senge.

Click here for the handouts!

Logic Models as a Platform for Program Evaluation Planning, Implementation, and Use of Findings

Level: All

Practitioners use logic models to describe important components of a program; make visible a theory of change; and link activities to intended outcomes.  For the purposes of evaluation practice, a well-constructed logic model provides a program-specific foundation for identifying evaluation questions; prioritizing data needs; and translating findings into recommendations for ongoing program improvement.  Aimed directly at improving the utility of logic models and quality of evaluation practice in your setting, the workshop addresses 2 questions:

(1)   What are the hallmarks of a well-constructed, scientifically-sound and useful logic model?

(2)   How do we maximize the use of logic models for program evaluation planning, implementation and use of findings?

Workshop Objectives:

• Demystify and define the logic model as a starting point for everyday evaluation practice

• Identify the hallmarks of a well-constructed, scientifically-sound logic model

• Clarify the relevance of process and outcome evaluation to preparing and using program logic models

• Demonstrate the use of logic models to identify and prioritize evaluation questions and data needs

• Examine the use of logic models to identify opportunities/options for demonstrating accountability for scarce resources

• Demonstrate use of a logic model to guide preparation of findings/recommendations aimed at ongoing program improvement

• Pinpoint additional resources for continued study/application

Goldie MacDonald, Ph.D. is the National Evaluation Coordinator for the Steps to a HealthierUS Cooperative Agreement Program in CDC’s National Center for Chronic Disease Prevention and Health Promotion. Dr. MacDonald is an experienced facilitator and provides leadership and technical expertise on program evaluation planning, implementation, and the use of findings to inform program improvement at both the national and community levels. Much of her work focuses on identifying appropriate strategies for documenting program processes and outcomes toward increased accountability to diverse program stakeholders. She also provides expertise on participatory approaches to program evaluation, utilizing qualitative methods of data collection, and the use of logic models as a platform for program planning and evaluation. She is lead author of “Introduction to Program Evaluation for Comprehensive Tobacco Control Programs.” For their work on this publication, the authors received the Alva and Gunnar Myrdal Award for Government from AEA in November 2002.

Sorry- Handouts are posted online at the presenters discretion and are not available for this session.

Principled Discovery: Learning From Your Data, with Reason 

Level: Intermediate - one should take the Training or the Breakout with Dr. Mark, but not both

Too often, quantitatively-trained evaluators are taught to test only a priori hypotheses. But evaluation data are too valuable not to try to learn the unanticipated lessons they may have to tell. These can be lessons about how the program operates differently in different contexts or with different subgroups of client.  Or lessons about why the program works (or doesn’t).  Or lessons about different subgroups of clients, or service providers, that we didn’t realize existed in advance.  Some evaluators have learned ways to "snoop" in or “mine” their data. However, without ways to make the snooping/ mining principled, there are serious risks of getting it wrong.

We will explore many ways of learning from one's data, including tests of moderation; graphical methods; post hoc use of clustering methods; exploratory investigation of higher-level variables in multilevel models; and exploratory searching for mediators and countervailing effects. In addition, we will discuss multiple ways of making new discoveries principled, other than the familiar notions of replication.  The session will also provide a conceptual introduction to (or review of) some current quantitative methods such as multilevel models and structural equation modeling.

Melvin M. Mark, Ph.D. is Professor of Psychology at Penn State University , Editor Emeritus of the American Journal of Evaluation and President-elect of the American Evaluation Association. He has conducted federally funded evaluations in the areas of prevention programs for at-risk youth, federal personnel policies, and industrial modernization, and has been involved in evaluations of state and local programs.  An award-winning teacher, he has published numerous papers and chapters on the theory and design of evaluation.  Among his books are Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs (Jossey-Bass, 2000) and a forthcoming Handbook of Evaluation (SAGE London).

Click here for the handouts!

Focus Group Research: Understanding, Designing and Implementing

This training session is full. Registration is no longer available for this session.

Level: All

As a qualitative research method, focus groups are an important tool to help researchers understand the motivators and determinants of a given behavior. This course provides a practical introduction to focus group research. At the completion of this course, participants will be able to 1) identify and discuss critical decisions in designing a focus group study,  2) understand how research or study questions influence decisions regarding segmentation, recruitment, and screening; and, 3) identify and discuss different types of analytical strategies and focus group reports.

Michelle Revels and Bonnie Bates are technical directors at ORC Macro specializing in focus group research and program evaluation.  Ms. Revels attended Hampshire College in Amherst, MA and the Hubert H. Humphrey Institute of Public Affairs at the University of Minnesota. Ms. Bates, also a trained and experienced focus group moderator and meeting facilitator, received her bachelor’s and master’s degree in criminal justice from the University of Maryland.

Click here for the handouts!

WEDNESDAY TRAININGS

Exploring Effect Size and Measures of Association

Level: Intermediate

Answer the call to report effect size and association measures as part of your evaluation results. This workshop will improve your capacity to understand and apply a range of measures including: standardized measures of effect sizes proposed by Cohen, Glass, and Hedges; Eta-squared; Omega-squared; the Intraclass correlation coefficient; and Cramer’s V. Through mini-lecture and demonstration you will improve your understanding of the theoretical foundation and computational procedures for each measure. The session will include: definitions of and procedures for computing a range of effect size and association measures, a presentation that examines the relationships among the common measures, and description of computation of selected confidence intervals for effect sizes and association measures. You will receive SPSS and SAS software program codes for performing many of the computations related to the measures and common confidence intervals.

Jack Barnette, PhD has served as a faculty member at Penn State University , University of Virginia , University of Memphis , University of Alabama at Tuscaloosa , University of Iowa , and is now Senior Associate Dean for Academic Affairs and Professor of Biostatistics at the University of Alabama at Birmingham . He has served as an APHA Statistics Council Member and Section representative to the APHA Action Board. Presently, he is chairing the ASPH biostatistics competency workgroup and is co-chair elect of the ASPH Biostatistics/Epidemiology Section. He has more than 30 years experience in teaching, advising students, and applying research, evaluation, and statistical methods to a wide variety of educational and public health projects. He has conducted evaluations of projects funded by CDC, HRSA, SAMHSA, NHLBI, and NIOSH. He serves on three of the ASPH/CDC Preparedness Exemplar Groups: Education and Evaluation Methods, Certificate Programs, and University-based Student Preparedness. He has been conducting research on the use of effect sizes and measures of association for the past seven years and he has presented pre-sessions on this topic at the last three AEA annual meetings. He holds the PhD in Educational Research and Development from Ohio State (1972).

Click here for the handouts!

Integrating Goals, Performance Measurement, and Evaluation
This training session is full. Registration is no longer available for this session.

Level: Intermediate

While this session will be taught by CDC staff, the focus is on how any large organization integrates evaluation, planning and performance monitoring.  CDC, like many large organizations, is moving to “performance-informed” management, where its program components align their efforts with enterprise-wide goals and objectives, progress is monitored through good performance measurement, and findings are continuously fed back for target evaluation and program improvement.  There are challenges in any organization at each step in this cycle, and then a second set of challenges in integrating the steps so that they are mutually supportive.  In this session,  using some cross-cutting cases, we’ll walk through the development of enterprise goals and objectives, how to align program efforts with goals, how to identify a strong set of performance measures, how to use performance measurement to guide program evaluation, and how to ensure that findings from performance measurement and evaluation feed program improvement.  We’ll focus especially on the key roles evaluators and evaluation skills can play in this continuous quality improvement cycle.

The CDC/OSI Goals Team has responsibility for coordinating and supporting the goals management process at CDC.  Currently, that means identifying, training, and providing analytic and logistical support to more than 20 goal action teams; helping those teams produce quality and timely products; creating knowledge management systems to support their efforts; and doing CDC-wide communication and training to build knowledge and support for the goals process among CDC programs and staff.

Click here for the handouts!

Designing Evaluations for Impact

This training session is full. Registration is no longer available for this session.

Level: Intermediate

Many evaluations have as a goal to impact decision-making or policy formulation by providing systematic empirical feedback.  Information can be intended for a variety of audiences including sponsors, client-groups, administrators, staff, and other relevant constituencies. Most often, evaluation information is considered to have an impact if it aids in decision-making. However, the relationship between an evaluation and its impact is not simple.  Based on empirical knowledge, we know that there are steps that an evaluator can take to help to ensure that an evaluation has impact.  The goal of this workshop is to provide participants with an understanding of, a framework for, and specific strategies that will increase the likelihood than an evaluation will have impact.  In this session, using participants’ own programs and experiences, we will first explore the conditions that can influence an evaluation’s impact.  Participants will then engage in interactive discussions about the specific methods used for increasing evaluation impact.  Small group activities and simulation and role-play exercises will be used to explore the use of specific methods for increasing impact and to foster original and rigorous thinking about how to apply the particular strategies presented to participants’ own contexts and programs.        

Christina A. Christie is an Assistant Professor, Director of the Masters of Arts Program in Psychology and Evaluation, and Associate Director of the Institute of Organizational and Program Evaluation Research at Claremont Graduate University.  Her research, which has been supported by several funders including the Haynes Foundation, focuses on investigating the relationship between evaluation theory and practice and issues related to the development of descriptive theories of evaluation.  She has also received funding from a variety of sources, including the National Science Foundation, the U.S. Department of Education, and the Hewlett-Carnegie Foundation to evaluate social, education, and health behavior programs targeting high-risk and underrepresented populations.  Christie co-founded the Southern California Evaluation Association, a local affiliate of the American Evaluation Association, and is the former Chair of the Theories of Evaluation Division of the American Evaluation Association.  In 2004, Christie received the American Evaluation Association’s Marcia Guttentag Early Career Achievement Award.

Sorry- Handouts are posted online at the presenters discretion and are not available for this session.

Advanced Applications of Program Theory

Level: Intermediate

A top 5 offering at Evaluation 2005! While simple logic models are an adequate way to gain clarity and initial understanding about a program, sound program theory can enhance understanding of the underlying logic of the program by providing a disciplined way to state and test assumptions about how program activities are expected to lead to program outcomes. 

Lecture, exercises, discussion, and peer-critique will help you to develop and use program theory as a basis for decisions about measurement and evaluation methods, to disentangle the success or failure of a program from the validity of its conceptual model, and to facilitate the participation and engagement of diverse stakeholder groups. 

You will learn:

• To employ program theory to understand the logic of a program,

• How program theory can improve evaluation accuracy and use,

• To use program theory as part of participatory evaluation practice.

Stewart I. Donaldson, Ph.D. is Professor and Chair of Psychology, Director of the Institute of Organizational and Program Evaluation Research, and Dean of the School of Behavioral and Organizational Sciences, Claremont Graduate University. He has conducted numerous evaluations, developed one of the largest university-based evaluation training programs, and taught and published widely on applied research and evaluation topics. His recent work includes a book with Michael Scriven about the future of evaluation practice - Evaluating Social Programs and Problems: Visions for the New Millennium (2003), a book Applied Psychology: New Frontiers and Rewarding Careers (2006), and a forthcoming book Program Theory-Driven Evaluation Science: Strategies and Applications. He is co-founder of the Southern California Evaluation Association, has served as Co-Chair of AEA's Theory-Driven Evaluation and Program Theory Topical Interest Group, and is on the Editorial Boards of the American Journal of Evaluation and New Directions for Evaluation.

Click here for the handouts!

Essential Competencies for Program Evaluators

Level: All

This half-day session is designed to teach participants the recently-developed Essential Competencies for Program Evaluators, a set of knowledge, skills, and attitudes in six categories, developed over the past six years.  The session will begin with the analysis of program evaluation vignettes representing diverse areas of practice to show both the common competencies across settings and those unique to specific contents or contexts.  Following a brief history of how the competencies were developed, the group will then collectively examine the competencies in all six categories: professional practice, systematic inquiry, situational analysis, project management, reflective practice, and interpersonal skills.  This activity, which builds on the continuum of interpersonal evaluation practice, will ground participants in the competencies’ content and allow people to ask questions as they think about their own work.  After a short break, participants will divide into role-alike or content-alike groups and develop concept maps to explore how the competencies make sense in their role or content area.  Comparative discussion will further illuminate the competencies, and then participants will complete a self-assessment tool and discuss how to set priorities and action steps for professional development.  Most of the session will consist of interactive exercises with just enough lecture to frame the discussion.

Jean A. King, Ph.D., is a Professor in the Department of Educational Policy and Administration at the University of Minnesota where she serves as the Director of Graduate Studies and Coordinator of the Evaluation Studies Program.  She holds an M.S. and Ph.D. from Cornell University and prior to her graduate study taught middle school English for a number of years.  In 1995, her work using participatory evaluation methods resulted in the Myrdal Award for Evaluation Practice from the American Evaluation Association, and in 1999, she was awarded the Association’s Robert Ingle Award for Extraordinary Service.  Professor King received the University of Minnesota, College of Education and Human Development’s Beck Award for Outstanding Instruction in 1999, the College’s 2002 Distinguished Teaching Award, and the 2005 Community Service Award.  She is the author of numerous articles and chapters and, with Laurie Stevahn, continues writing a book on interactive evaluation practice.

Click here for the handouts!

Strengthening Evaluation Through Cultural Relevance and Cultural Competence
This training session is full. Registration is no longer available for this session.

Level: Intermediate

This skill-building session addresses the centrality of culture in evaluation. It is organized in two segments. The opening segment addresses the relevance of culture to all stages of the evaluation process, to the fundamental validity of our work as evaluators, and to ethical standards and guidelines of our profession. Presenters will use an FAQ format to raise questions and address common misconceptions that marginalize discussions of culture within the evaluation community (e.g., Is “culture” really just a code-word for “race”? How does culture apply to me as a white evaluator working within predominantly white populations? What is the “value added” of culture in evaluation? Why should I care?) The second segment extends cultural relevance to present strategies for building cultural competence through experience, education and self-awareness. Theoretical frameworks that situate culture in evaluation (e.g., Frierson, Hood & Hughes, 2002; Hall & Hood, 2005; Kirkhart, 2005) are presented as advance organizers for practice and application purposes. Presenters use case scenarios and participants’ own examples to integrate workshop content with participants’ field experience, interests, and concerns. They rely on various theoretical frameworks to guide the two segments in tangible and practical ways. Additional resources are provided to extend and reinforce participant learning.

Karen E. Kirkhart holds a Ph.D. in Social Work and Psychology from The University of Michigan and is currently Professor, School of Social Work, College of Human Services and Health Professions, Syracuse University. Rodney K. Hopson has undergraduate and graduate degrees in English Literature, Educational Evaluation, and Linguistics from the University of Virginia, and he is Associate Professor and Chair, Department of Educational Foundations and Leadership and faculty member in the Center for Interpretive and Qualitative Research at Duquesne University. Karen and Rodney have served in positions of leadership within the American Evaluation Association, and both are actively involved in education and scholarship on culture, diversity, and social justice in evaluation. Rodney serves as Project Director for the American Evaluation Association/Duquesne University Graduate Education Diversity Internship Program. Karen is a member of the AEA Diversity Committee task force charged with developing a public interest statement on the subject of cultural competence and evaluation.

Click here for the handouts!

Edward Liebow, PhD, is Senior Research Scientist on the staff of Battelle's Centers for Public Health Research and Evaluation. Liebow has conducted policy-related and evaluation research throughout the western US and in South Australia focusing on applying ethnographic research methods to understand the distinctive response of disadvantaged communities to potential environmental and public health hazards posed by development programs and policies. Dr. Liebow is affiliated with the University of Washington, where he teaches courses in ethnographic research methods, American Indian Studies, and comparative urban politics. He is also a visiting professor of applied anthropology and comparative economics at the Università Carlo Cattaneo in Castellanza, Varese (Italy). He recently completed a term as President of the National Association for the Practice of Anthropology, and he also serves as Book Review Editor for the Journal of Political Ecology.

Click here for the handouts!

RealWorld Evaluation: Overcoming Constraints in International and National Contexts

Level: Intermediate

What do you do when asked to perform an evaluation on a program that is well underway? When your questions about baseline data and control groups are met with blank stares? When time and resources are few, yet clients expect “rigorous impact evaluation”? When there are political expectations and pressures to deal with? This workshop presents a seven-step approach that seeks to ensure the best quality evaluation under real-life constraints.

Through presentations and discussion, with real-world examples drawn from international development evaluation, participants in this workshop will be introduced to the RealWorld Evaluation approach. The workshop focuses on developing country evaluation, but the techniques are applicable to evaluators working in any context with budget, time, data and political constraints.

The fundamental framework for the session is an exploration of the seven steps of the RealWorld Evaluation approach, including ways to work negotiate RealWorld solutions with clients. We’ll also investigate methodological issues applicable to RealWorld Evaluations including approaches to reconstructing baseline data and how to identify and address threats to the validity or adequacy of evaluation methods. Finally, we’ll identify seven evaluation designs appropriate for RealWorld evaluations and discuss their appropriate context-dependent applications.

Jim Rugh brings 42 years of experience in international development, including 26 years specializing in program evaluation. For 11 years he has been the Coordinator of program Design, Monitoring and Evaluation for CARE, a large international NGO with programs in 70 countries reaching over 48 million beneficiaries. Under Jim’s leadership CARE has adopted an evaluation policy, strategies and standards to promote learning for improved program quality and accountability for effectiveness. An active member of AEA, and a founding member of AaEA (the Atlanta-area Evaluation Association), Jim has co-facilitated or facilitated evaluation workshops at AEA’s annual conference as well as in Cape Town South Africa, Kiev Ukraine, Washington DC and Atlanta. He is the co-author with Michael Bamberger and Linda Mabry of the RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints (SAGE Publications).

Click here for the handouts!

BREAKOUTS - Click on a breakout title or scroll down for descriptions

Tools for Navigating the Foundation Paradox

Level: All

Scholar and grant recipient Michael Hooker, in exploring the moral values in philanthropy, described the situation as "a game of rhetorical persuasion where the rules regarding honesty and candor are suspended or subtly altered, just as they are in poker."  His description resonates across government and private funding systems that are increasingly dependent on organizations promising to reach narrow, predetermined targets – without much concern for the context in which the work is occurring. In this session, participants will discuss the forces driving this pathology, which include explanations of Kahneman's "delusional optimism," misaligned incentives and rewards systems, buyer's remorse among decision makers who purchase strategic plans from outside consultants, corrupted indicators and scorecards, and Lee Schorr's description of the "sham" arrangement in which funders ask grant recipients to achieve grant outcomes "with wholly inadequate resources and tools."  Deliverance from these sins requires funders and other decision makers to embrace a paradox - that the only hope for doing well rests on rewarding news of and solutions for whatever it is that they're doing poorly. In the session, participants will learn about nine tools leaders can use to increase the likelihood of navigating this paradox successfully.

Assessing Data Collection Instruments

This breakout is full. Registration is no longer available for this breakout.

Level: Intermediate

When you read reports and articles, do you see indications of psychometric properties reported and do you feel you aren’t quite sure what these represent or how to interpret them? Do you look for instruments you might want to use and see indicators of reliability and validity to help you decide if the instrument has potential use? Do you need to compute reliability and validity measures and report them in reports and publications? If you answered “yes” to any of these, this workshop should interest you. This workshop will focus on commonly used measures of reliability and validity relative to how they are computed, interpreted, and used to help you decide if the research that uses these methods is credible. A new call to include confidence intervals for these measures will be presented as well as a free software program developed by the workshop presenter demonstrated and provided to participants.

Jack Barnette, PhD has served as a faculty member at Penn State University , University of Virginia , University of Memphis , University of Alabama at Tuscaloosa , University of Iowa , and is now Senior Associate Dean for Academic Affairs and Professor of Biostatistics at the University of Alabama at Birmingham . He has served as an APHA Statistics Council Member and Section representative to the APHA Action Board. Presently, he is chairing the ASPH biostatistics competency workgroup and is co-chair elect of the ASPH Biostatistics/Epidemiology Section. He has more than 30 years experience in teaching, advising students, and applying research, evaluation, and statistical methods to a wide variety of educational and public health projects. He has conducted evaluations of projects funded by CDC, HRSA, SAMHSA, NHLBI, and NIOSH. He serves on three of the ASPH/CDC Preparedness Exemplar Groups: Education and Evaluation Methods, Certificate Programs, and University-based Student Preparedness. He has been conducting research on the use of effect sizes and measures of association for the past seven years and he has presented pre-sessions on this topic at the last three AEA annual meetings. He holds the PhD in Educational Research and Development from Ohio State (1972).

Click here for the handouts!

Ensuring Evaluation Use

Level: All

Many evaluators and program staff are concerned with designing evaluations that are intended to inform more immediate decision-making and promote organizational change.  A goal, then, is to offer the most useful information to answer the evaluation questions given the program context and the resources available to conduct the investigation.  This workshop session will focus on developing participants’ understanding of and methods for increasing evaluation use.  This session will provide participants with a theoretical framework for understanding and promoting evaluation use.  Strategies and techniques for increasing the use of both the evaluation process and findings will be examined and demonstrated through interactive and small group exercises.  Upon completion of this course, participants will: understand use as a primary purpose of evaluation, understand the difference between evaluation process and evaluation findings use, become familiar with a framework for promoting use, and identify strategies for promoting and increasing evaluation use. 

Christina A. Christie is an Assistant Professor, Director of the Masters of Arts Program in Psychology and Evaluation, and Associate Director of the Institute of Organizational and Program Evaluation Research at Claremont Graduate University.  Her research, which has been supported by several funders including the Haynes Foundation, focuses on investigating the relationship between evaluation theory and practice and issues related to the development of descriptive theories of evaluation.  She has also received funding from a variety of sources, including the National Science Foundation, the U.S. Department of Education, and the Hewlett-Carnegie Foundation to evaluate social, education, and health behavior programs targeting high-risk and underrepresented populations.  Christie co-founded the Southern California Evaluation Association, a local affiliate of the American Evaluation Association, and is the former Chair of the Theories of Evaluation Division of the American Evaluation Association.  In 2004, Christie received the American Evaluation Association’s Marcia Guttentag Early Career Achievement Award.

Click here for the handouts!

Evaluating Interventions That Have Already Been Deemed to be Evidence-Based
This breakout is full. Registration is no longer available for this breakout.

Level: All

This workshop presents a comprehensive framework for evaluation of diffusion/dissemination and implementation of evidence-based interventions. The frame work will address adoption, adaptation, fidelity, re-invention, processes, quality control, outcomes, and effectiveness. The framework uses a range of evaluation methods across program implementation stages. Distinctions are made between diffusion and dissemination evaluation. The framework allows judgments to be made at multiple diffusion/dissemination and implementation stages. There is an initial focus on use of evaluation methods, including logic modeling to assess the intervention selection process by providers. Agency capacity assessments further improve the intervention selection process. Core element identification allows for establishment of program standards against which subsequent program implementation may be assessed. Fidelity measures, including quality and quantity of delivery follow the identification of core elements. Fidelity scales may be modified to monitor program implementation and quality control.  Adaptation and re-invention assessments are included to make judgments regarding program drift from protocol. Process monitoring will assess service delivery processes and process evaluation addresses obtaining program objectives. Methodological and conceptual issues regarding outcome evaluation of interventions previous demonstrated to be efficacious will be discussed as well as the process of developing a strategy for effectiveness trials.

Charles B Collins, Jr. Ph.D. is the Team Leader for the Science Application Team in the Capacity Building Branch of the Division of HIV/AIDS Prevention at the CDC. His team is responsible for the diffusion of evidence based behavioral interventions into HIV prevention public health practice. He has designed, implemented, and evaluated an efficacious HIV intervention for African American drug users in Birmingham Alabama. He has conducted evaluability assessments and evaluations in multiple community based and other service organizations. Dr. Collins is also a curricula writer and trainer, having trained many CDC grantees and health departments on “Using Evaluation for Program Improvement and Capacity Building” and “Monitoring and Evaluating Outcomes to Improve HIV Prevention Interventions.” He is co-author, with Dr. Belinda Reininger, of the Association of Schools of Public Health Monograph “How to Monitor and Measure Evidence-Based Intervention Effectiveness”.

Click here for the handouts!

Economic Evaluation for Decision Making

Level: Intermediate

Economic evaluation refers to applied analytic methods used to identify, measure, value, and compare the costs and consequences of prevention and treatment strategies. This course provides a brief overview of these methods, including cost analysis, cost-effectiveness analysis (CEA), and cost-benefit analysis (CBA).  Special emphasis will be placed on the conceptual and empirical bases for alternative health-benefit measures − quality-adjusted life years (QALYS) for use in CEAs and the willingness-to-pay methodology for use in CBAs, and the ways in which such measures are used in policymaking.

Phaedra S. Corso, Ph.D., is the senior health economist in the National Center for Injury Prevention and Control at CDC.  Dr. Corso has worked at CDC for over a decade in the areas of economic evaluation and decision analysis, publishing numerous articles on the cost-effectiveness of prevention interventions and co-editing a book on prevention effectiveness methods in public health.  She holds a Master’s degree in public finance from the University of Georgia and Ph.D. in health policy and decision sciences from Harvard University.

Click here for the handouts!

Using Program Theory to Improve Evaluations: A Beginner's Guide

This breakout is full. Registration is no longer available for this breakout.

Level: Beginner

Program theory is an essential tool for contemporary evaluation practice.  It can provide a basis for informed decisions about measurement and method choice, enable evaluators to disentangle the success or failure of program implementation from the validity of a program’s conceptual model, and facilitate a cumulative wisdom about how programs work and how to make them work better.  Program theory-driven evaluation science (consistent with CDC’s Six Step Evaluation Framework) has emerged as one of the most popular evaluation approaches for empowering and promoting the meaningful engagement of diverse stakeholder groups. 

Lecture, exercises, and discussion will introduce you to strategies for developing and using program theory to improve evaluations.  You will learn the basics of: 

• Developing sound program theory,

• Using program theory to design evaluations,

• Participatory program theory-driven evaluation science.

Stewart I. Donaldson, Ph.D. is Professor and Chair of Psychology, Director of the Institute of Organizational and Program Evaluation Research, and Dean of the School of Behavioral and Organizational Sciences, Claremont Graduate University. He has conducted numerous evaluations, developed one of the largest university-based evaluation training programs, and taught and published widely on applied research and evaluation topics. His recent work includes a book with Michael Scriven about the future of evaluation practice - Evaluating Social Programs and Problems: Visions for the New Millennium (2003), a book Applied Psychology: New Frontiers and Rewarding Careers (2006), and a forthcoming book Program Theory-Driven Evaluation Science: Strategies and Applications. He is co-founder of the Southern California Evaluation Association, has served as Co-Chair of AEA's Theory-Driven Evaluation and Program Theory Topical Interest Group, and is on the Editorial Boards of the American Journal of Evaluation and New Directions for Evaluation.

Click here for the handouts!

Identifying and Measuring Outcomes in Health Communication
(Register for: "Outcomes in Health Communication")

This session will provide participants with an overview of evaluation techniques for assessing health communications campaigns, with a particular focus on media-based campaigns. The curriculum will briefly review the role of evaluation in campaign design, ensuring that an explicit, well-grounded logic links the intervention activities to the campaigns goals. Participants will also consider the evaluability of campaign outcomes and discuss the factors that researchers should consider in determining whether an intervention is likely to result in measurable outcomes. The course will focus on techniques that can be used to assess both process and outcome objectives of media-based campaigns. Among the evaluation methods presented will be qualitative assessment of campaign materials, tracking and analysis of media buy patterns, assessment of public affairs/public relations efforts, and techniques for assessment of community-focused campaign components. Particular attention will be paid to measurement of campaign awareness and comprehension of messages as well as behavioral and attitudinal outcomes. Broad analytic techniques will be covered as appropriate to each method. Examples for the class will be drawn from the presenters’ experiences in evaluations of media campaigns prevent smoking and to increase physical activity among youth.

Jennifer Duke, Ph.D. is the Director of Research at the American Legacy Foundation in Washington , D.C. Her work includes outcome evaluations of the TRUTH campaign to prevent smoking among youth as well as a variety of process and outcome evaluations of community-based programs. Previously, Dr. Duke was a study director at Westat, a social science research firm, where she conducted the outcome evaluation of VERB, CDC’s initiative to encourage physical activity among 9-to 13 year olds. She has also evaluated other health communications projects at CDC, the National Cancer Institute, and non-profit health organizations. She holds a PhD in health psychology from Rutgers University.

Click here for the handouts!

Qualitative Evaluation Approaches
This breakout is full. Registration is no longer available for this breakout.

Level: Beginner

Practical qualitative evaluation methods will be shared for community health programs.  The workshop will be an overview of qualitative methods that include the use of logic models, social ecology principles, participatory-based strategies, and basic data analysis methods.  The session is designed to foster creativity when planning, implementing and evaluating a community-based program.  Simulation activities will be used to dramatize many of the concepts.

Robert M. Goodman, Ph.D., M.P.H., M.A., is a Professor and Chair, Department of Behavioral and Community Health Sciences, Graduate School of Public Health, University of Pittsburgh. Dr. Goodman has written extensively on issues concerning community health development, community capacity, community coalitions, evaluation methods, organizational development, and the institutionalization of health programs. He has been the principal investigator and evaluator on projects for CDC, The National Cancer Institute, The Centers for Substance Abuse Prevention, The Children’s Defense Fund, and several state health departments. In 2004, Dr. Goodman received the Distinguished Fellow Award from the Society for Public Health Education, the highest honor it bestows. Currently, Dr. Goodman is consulting on community-based public health practices and empowerment evaluation with the Diabetes Translation and Injury Prevention Branches at CDC. Also, he is leading an evaluation of community-based approaches to increasing interest in cancer clinical trials.

Click here for the handouts!

Using Geographic Information Systems (GIS) in Evaluation: Data, Analysis Techniques, and Cartographic Products

This breakout is full. Registration is no longer available for this breakout.

Level: Intermediate

There is growing interest in using Geographic Information Systems (GIS) as a tool for managing, visualizing, and analyzing data in support of evaluation.  In this session, using examples drawn from the evaluation literature, we will: 

• present the basic concepts of GIS (e.g., scale, resolution, projection),

• discuss data sources (US Census, data clearinghouses),

• describe spatial analysis techniques (buffering, spatial joining), and

• overview software options (ESRI, MapInfo, other).

At the end of this session, participants will have a clear understanding of when GIS might be useful in an analysis and presentation of evaluation data, strengths and pitfalls, and the range of skills and background needed to successfully use a GIS.  This information is important for evaluation professionals wanting to actually use GIS software and for persons who need to know the right questions to ask when contracting for GIS services.  Each participant will be given a cd-rom with an example ArcReader project, links to important websites and articles, presentation slides, and references.

Janet L. Heitgerd is a behavioral scientist in the Geospatial Research, Analysis, and Services (GRASP) in the Division of Health Studies, Agency for Toxic Substances and Disease Registry (ATSDR).  She received her PhD in Sociology at the University of Oklahoma. Her work at ATSDR primarily focuses on advancing the application of social science theory and methods to environmental public health practice.  Mapping technology presents new opportunities for impacting public health through the analysis, visualization, and dissemination of geospatial data and products. Padmaja Vempaty is an Oak Ridge Institute for Science and Education (ORISE) Fellow in GRASP.  She received her MPH at the University of Alabama and MSW at the Tata Institute of Social Sciences in India.  Ms.Vemptay accepted the ORISE fellowship at ATSDR to gain experience in Geographic Information Systems (GIS). Her public health background and interest is in the evaluation and monitoring of HIV/AIDS and reproductive health programs. 

Click here for the handouts!

Planning Influential Evaluations: Linking Evaluation Purpose, Questions, and Design
This breakout is full. Registration is no longer available for this breakout.

Level: All

Plans for evaluations often draw on previous evaluations that an organization has sponsored or the staff’s training or beliefs about evaluation.  Alternatively, evaluation planning can be done by deciding on the specific purpose for the evaluation first and then developing the questions and methods to fit the purpose.  In this workshop, participants will become thoroughly acquainted with the four primary purposes for evaluation: program or organizational improvement; oversight and accountability; assessment of merit and worth; and knowledge development.  During the class, questions guiding each of the evaluations, methods typically used to address each question, and examples of actual evaluations will be presented and discussed.  While the primary training method will be lecturing, workshop participants are encouraged to bring up examples from their own evaluation experiences and raise questions throughout the session.

Gary T. Henry is a professor in the Andrew Young School of Policy Studies at Georgia State University.  He previously served as the Director of Evaluation and Learning Services for the David and Lucile Packard Foundation.  Henry has evaluated a variety of public policies and programs and has published extensively in the field of evaluation and education policy.  He received the Evaluation of the Year Award from the American Evaluation Association in 1998 for his work with the Georgia’s Council for School Performance and the Joseph S. Wholey Distinguished Scholarship Award in 2001 from the American Society for Public Administration and the Center for Accountability and Performance.  Dr. Henry currently serves on scientific peer review panels for the Institute of Education Sciences, U.S. Department of Education and on a National Research Council/National Academy of Science panel assessing the effects of “green schools” on the health and productivity of teachers and students.

Click here for the handouts!

Evaluating Quality Improvement: New Ways of Thinking about the Research-to-Practice Relationship in Health

It takes nearly 20 years on average for new knowledge from medical research to be incorporated into everyday clinical practice, and even then application is highly uneven. Why does this process take seemingly so long? This issue or problem begs fundamental assumptions about how we learn or create new meaning (or knowledge. The commonly accepted paradigm we use to explain the use of research is that as situated knowledge (an “it” or a “thing”), research evidence can be linearly transferred into clinical practice. In this session, we’ll unpack our assumptions about the research-to-practice relationship and offer an alternative interpretation. Informed by the complexity sciences, an alternative paradigm suggests that the evidence-to-practice track record is poor because, in actuality, it is not a mechanistic process.  Innovation is a social process. Meaning is not an inherent property of evidence, not a neutral medium, it’s not simply put or transferred into practice. Evidence is only a part of an ongoing process of research becoming practice. In this session we will see how these paradigms compare and in doing so examine like assumptions about evaluation design.

David M. Introcaso, Ph.D., is the Evaluation Officer for the Agency for Healthcare Research and Quality/AHRQ where he manages an evaluation practice of  projects spanning all areas of  AHRQ's ten portfolios of research. Prior to AHRQ, David worked with the National Hospice and Palliative Care Organization, the National Institute of Nursing Research, Americans for Better Care of the Dying and at DC General Hospital. He teaches bio- and business ethics as an adjunct at the University of Chicago and previously at Georgetown and George Washington University.  Among other awards David was a W. K. Kellogg Foundation National Leadership Fellow and a Johns Hopkins University Change Management Fellow.  He was graduated from Arizona State with a BS, MA and Ph.D.

Click here for the handouts!

Systems Thinking: A Practical Application

Level: Intermediate

Systems thinking is a method for helping a team of people to understand and diagram the inter-relationships in a complex system such as an organization or a public health challenge.  This interactive session will introduce participants to a method for facilitating a diverse group with the “Diabetes Population Flows Map,” a systems thinking diagram created by CDC’s Division of Diabetes Translation to help support conversations about diabetes strategy. Session participants will do hands-on exercises and contribute to group discussions with the purpose of gaining insights into how CDC has benefited from this approach and, more importantly, the application of systems thinking method to any public health challenge, not just diabetes or chronic disease. This session will cover some of the same material as the morning session, “Systems Thinking for Public Health,” and supplement it by providing an experience in using systems thinking as a facilitation method in diverse public health settings. Attendance in the morning session would be helpful but is not required to benefit from this session.

Christopher Benjamin is a public health analyst in the Division of Diabetes Translation. He presently serves as Special Assistant for Project Development in the Program Development Branch. His work includes the coordination and integration of systems thinking into Branch functions and activities. Mr. Benjamin received a B.A. in Public Policy Sciences from Duke University and earned a J.D. and a M.A. in Public Policy from the University of Minnesota. Drew Jones is a project director with Sustainability Institute. His work focuses on applying systems thinking approaches to a wide range of societal concerns, particularly in the public health and environmental areas. Mr. Jones received a B.A. in Engineering Sciences and Environmental Studies from Dartmouth College and earned a M.S. from M.I.T.

Click here for the handouts!

Evaluation on a Shoestring
This breakout is full. Registration is no longer available for this breakout.

Level: All

This course will condense the instructor’s thirty plus years’ experience conducting evaluations in settings that often lack sufficient funding for studying programs even when funders have mandated evaluations. Evaluators must first consider whether or not to conduct an evaluation when resources are limited, so participants will begin by discussing the trade-offs among three categories of the Program Evaluation Standards (utility, feasibility, and accuracy) in deciding if a low-cost evaluation is an appropriate option. The session will then interactively teach participants three techniques they can use with evaluation clients to enable people to collect or process data:

• Techniques for group data processing include dot voting, the corners technique, and cooperative rank ordering;

• Data dialogues are an informal, low-cost alternative to focus groups that borrow from the three-step interview process from social psychology; and

• Carousel or graffiti and concept formation activities encourage people to brainstorm effectively and then to quickly make sense of the group’s ideas.

The session will include enough lecturing to ground the activities conceptually, but will primarily walk participants though the interactive processes, emphasizing the strengths, weaknesses, and appropriate uses of each of them.

Jean A. King, Ph.D., is a Professor in the Department of Educational Policy and Administration at the University of Minnesota where she serves as the Director of Graduate Studies and Coordinator of the Evaluation Studies Program.  She holds an M.S. and Ph.D. from Cornell University and prior to her graduate study taught middle school English for a number of years.  In 1995, her work using participatory evaluation methods resulted in the Myrdal Award for Evaluation Practice from the American Evaluation Association, and in 1999, she was awarded the Association’s Robert Ingle Award for Extraordinary Service.  Professor King received the University of Minnesota, College of Education and Human Development’s Beck Award for Outstanding Instruction in 1999, the College’s 2002 Distinguished Teaching Award, and the 2005 Community Service Award.  She is the author of numerous articles and chapters and, with Laurie Stevahn, continues writing a book on interactive evaluation practice.

Click here for the handouts!

Budget Performance Integration

Level: Intermediate

A core component of continuous quality improvement efforts in large organizations is that budget should be influenced by performance. That is, in allocating resources among organizational components, the most resources should to “programs that work.” This session will review past efforts and current best thinking in budget-performance integration. Focusing especially on public sector efforts, we’ll look at the changing views of budgeting and its relationship to performance, examine standards for success for integrating budget and performance, walk through some simple case illustrations of budget-performance integration, present some myths and realities, and identify critical success factors in implementing a productive budget-performance integration effort.

Joseph L. Kull is a Director in PricewaterhouseCoopers’ Washington Federal Practice, specializing in budget and financial management matters. He spent 30+ years in the Federal government, most recently as Deputy Comptroller for Federal Financial Management at the OMB, providing assistance to Federal agencies in implementing financial systems reform. He was a key architect of the ‘Improving Financial Management’ component of the President’s Management Agenda and the Executive Scorecard. Previously, Mr. Kull served more than 15 years as the National Science Foundation’s (NSF) Chief Financial Officer (CFO) and Budget Director, and led NSF to becoming the first federal agency to obtain 'green' on the Executive Scorecard for financial management. Mr. Kull received a B.S. from Mount Saint Mary’s College and his MBA from George Mason University in Virginia. Mr. Kull has received several awards, including the Presidential Rank of Meritorious Executive Service Award and the JFMIP Donald L. Scantlebury award for distinguished leadership.

Click here for the handouts!

Telling Your Program’s Story: Conducting and Using Success Stories for Capacity Building

Level: Intermediate

In order to build program capacity a programs “success” must be told at many levels. In addition, impacts of prevention programs may not be able to be demonstrated for several years therefore communicating success during the various life stages of a program is important for long term sustainability. The presenters will use their experience with 13 national oral health grantees to demonstrate how to use success stories to build both program and evaluation capacity. The session will be a practical and hands on session enabling attendees to begin writing their own success stories.

René Lavinghouze is with the Division of Oral Health at the Centers for Disease Control and Prevention leading a multi-site, cluster evaluation designed to assess infrastructure development.  Rene has over 15 years experience with the CDC and in the private sector.  She is Chair of the American Evaluation Association's TIG for Cluster, Multi-site/level evaluations and serves on the communications team for the local evaluation affiliate, AaEA. She received her Master’s in Community Psychology at Georgia State University. Ann Price is president of Community Evaluation Solutions, Inc and has over 20 years experience in both treatment and prevention. She has conducted evaluations in many areas including intimate partner violence, mental health, substance abuse, tobacco prevention and oral health.  Prior to CES, Dr. Price was a Senior Data Analyst at ORC Macro on a multi-site national child mental health evaluation. She earned her Doctorate in Community Psychology from Georgia State University.

Click here for the handouts!

Building a Vision for Program Evaluation: A Systematic Approach to Increase Quality, Participation, and Consistency in Multi-Site, Multi-Level Program Evaluation

Level: Intermediate

Most large-scale evaluation efforts require the coordinated action of multiple participants or stakeholders over time.  Often, those tasked with program evaluation quickly identify intended outcomes, indicators and data collection methods; however, they may pay only limited attention to establishing an overarching vision for practice among participating sites.  With regard to multi-site, multi-level program evaluation, how do we balance the need for well-coordinated, strategic action and site autonomy?  This session begins with discussion of the process used to develop a vision, and related checkpoints, for evaluation practice across a multi-site, multi-level public health program—the Steps to a HealthierUS Cooperative Agreement Program.  The speaker presents a set of “foundational elements” to guide program evaluation across the Steps Program, and the payoffs of the vision in terms of increased participation and consistency in evaluation activity across funded communities.  Finally, the speaker makes visible the interplay of practice standards (i.e., accuracy, feasibility, propriety, and utility), and the hard choices required to actually complete the work of program evaluation.

Goldie MacDonald, Ph.D is the National Evaluation Coordinator for the Steps to a HealthierUS Cooperative Agreement Program in CDC’s National Center for Chronic Disease Prevention and Health Promotion. Dr. MacDonald is an experienced facilitator and provides leadership and technical expertise on program evaluation planning, implementation, and the use of findings to inform program improvement at both the national and community levels. Much of her work focuses on identifying appropriate strategies for documenting program processes and outcomes toward increased accountability to diverse program stakeholders. She also provides expertise on participatory approaches to program evaluation, utilizing qualitative methods of data collection, and the use of logic models as a platform for program planning and evaluation. She is lead author of “Introduction to Program Evaluation for Comprehensive Tobacco Control Programs.” For their work on this publication, the authors received the Alva and Gunnar Myrdal Award for Government from AEA in November 2002.

Sorry- Handouts are posted online at the presenters discretion and are not available for this session.

What Works When: Unraveling How Context Affects Program Effectiveness

Level: Intermediate - one should take the Training or the Breakout with Dr. Mark, but not both

How can evaluators move beyond overall tests of program effectiveness and learn what works under what conditions?  And, how can they do this without being misled by chance findings that arise only because one has sliced the data repeatedly?  Most standard statistical training emphasizes the testing of planned, a priori hypotheses, and understates the importance of "discovery."  But if you don't explore your data, you may forego a valuable opportunity to target interventions to those for whom they are effective, avoid giving them to others for whom they are ineffective or even harmful, and perhaps help create better or cheaper programs.  On the other hand, unfettered snooping and data mining creates other serious problems: Paraphrasing Stigler, if you torture your data enough, they'll confess to something, even if "not guilty".  In this session we'll present an approach that (1) allows for uncovering some of the contingencies that limit average intervention effects, but (2) tries, using other data, to keep you from being misled by chance.   Termed "principled discovery" (Mark et al.1998, 2000; Mark, in press), we'll look at some examples of the approach in practice, and as time allows discuss how principled discovery might be applied in the work of those at the session. 

Melvin M. Mark, Ph.D. is Professor of Psychology at Penn State University , Editor Emeritus of the American Journal of Evaluation and President-elect of the American Evaluation Association. He has conducted federally funded evaluations in the areas of prevention programs for at-risk youth, federal personnel policies, and industrial modernization, and has been involved in evaluations of state and local programs.  An award-winning teacher, he has published numerous papers and chapters on the theory and design of evaluation.  Among his books are Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs (Jossey-Bass, 2000) and a forthcoming Handbook of Evaluation (Sage London).

Click here for the handouts!

Using the Guiding Principles to Improve Your Evaluation Practice

Level: Beginner

This workshop focuses on helping evaluators understand and apply the revised AEA Guiding Principles for Evaluators to their professional practice. After a brief presentation that introduces the revised GPFE, participants will work together in small groups to apply the GPFE to a topical case study. Through these case explorations, lecture and small and large group discussions, participants will gain a deeper understanding of the practical applications of the GPFE.  The workshop will also introduce participants to resources—print, web-based and collegial networks—that evaluators can consult to handle professional dilemmas that arise in their practice. Participants will receive copies of the workshop presentation, the case study, the GPFE in full and abbreviated brochure format and a list of resources for more information and consultation.

Jules Marquart, PhD is a Senior Program Evaluator at Centerstone CMHC, Inc., where she supervises several federally-funded mental health evaluation projects. Dr. Marquart has over 25 years of experience in evaluation on the local, state, federal and international levels. Jules has been actively involved in AEA since its inception, and currently Chairs the Ethics Committee. Dr. Marquart received her Ph.D. in Human Service Studies from Cornell University, with a concentration in Program Evaluation and Research Methodology. Leslie Goodyear, PhD is a Senior Research Associate at Education Development Center, where she coordinates research and evaluation for a NSF-funded initiative. As a program evaluator and researcher, Dr. Goodyear has worked with programs focused on HIV/AIDS Prevention; Out-of-School Time; Youth Engagement; Educational Research; and Science Education. Leslie is the recent past Chair of the AEA Ethics Committee. She earned her M.S. and Ph.D. in Human Service Studies, with focus on Program Evaluation, from Cornell University.

Click here for the handouts!

Quality and Outcomes Initiatives: Adapting the Best Approaches and Leaving the "Fads" Behind

This breakout is full. Registration is no longer available for this breakout.

Level: All

Recent decades have seen a plethora of management initiatives, each promising to move organizations to more effective, efficient, and accountable performance.  Balanced Scorecard, Six Sigma, Continuous Quality Improvement, Goals Accountability, and Outcomes Management/Funding are just a few. While many initiatives originated in corporate environments and with management theorists, their insights have made their way into program planning and evaluation in all sectors. And others of these resemble traditional evaluation approaches, perhaps dressed up in new clothes. This session will provide an overview of the most common initiatives, identify where they agree and where they conflict, fit them into an integrating organizational framework, and extract the underlying added-value to evaluation and evaluators. It will strip away the jargon to enable delegates to consider the progress of their programs, or organizations, and to identify the next steps in their quality journeys. Practical tools will be provided for completion during and after the breakout. Free access to the presenter's on-line Toolkits will also be offered. 

John Marsh is President of Marsh & Barnes, Inc., a company which provides facilitation, consulting and training services to a wide range of organizations including healthcare, government agencies and non-profits. John is the author of The Continuous Improvement Toolkit and A Stake in Tomorrow. He has consulted worldwide and represented the UK on International Standards. John was a member of the Hawaii Board of Health and facilitated the creation of the Hawaii Outcomes Institute. Since Hurricane Katrina he has been using his process expertise to John has a Bachelor of Science Degree, with Honors, in Systems Engineering and a Master of Philosophy Degree in Process Management. John is known for his enthusiasm and ability to understand complex systems and issues. He applies the latest theories in practical ways to get real, long-term, benefits.

Click here for the handouts!

Measurement for Evaluators: Key Issues in Reliability, Validity, and Instrument Design

Level: Intermediate

The session will include informative discussion on the some key issues facing evaluators in the area of measurement. We will focus on the importance of testing instruments for reliability and validity prior to data collection, define and explain the various types of reliability and validity testing, and discuss implications of not testing instruments prior to data collection. The audience will also learn about current methods being used to assess reliability and validity. We’ll also touch on other issues in measurement that are critical for today's practicing evaluators, such as questionnaire design and using technology in data collection.  The presenter will bring dual perspectives from experiences in education and public health to this discussion, and will facilitate interactive activities to engage the audience in the presentation.

Frances McCarty, Ph.D. is a Biostatistician in the Behavioral Sciences and Health Education Department in the Rollins School of Public Health. She currently serves as the statistician on several grants that focus on interventions related to HIV prevention, nutrition and exercise, and HIV medication adherence.  She teaches research methods and statistics courses. Prior to her position at Emory, she was an assistant project director for a five-year grant at Georgia State University that focused on Head Start program quality and child outcomes. She holds a B.S. in Health/ Physical Education from Bridgewater College and a M.Ed. in exercise physiology from the University of Virginia.  She earned her Ph.D. in measurement and statistics from Georgia State University (Educational Policy Studies Department). Her research interests include the application of item response theory to instrument development and topics related to hierarchical linear modeling.

Click here for the handouts!

Planning and Evaluating System Change Ventures: A Dialogue on Shifting Concepts, Methods, and Moral Considerations

Level: Intermediate

Ideas about how to plan and evaluate public health work have changed significantly over the past decades, and these shifts are now directing the field to become more systems-oriented, politically engaged, and philosophically conscious. As health protection initiatives become more ambitious, their underlying concepts, methods, and moral considerations are growing more complex, more sensitive to the alignment among diverse actors, and more mindful of the ways that distant forces shape personal experience. Furthermore, because health leaders aspire to assure more equitable and more healthful conditions, which are always politically contested and constantly in flux, the concepts and methods they use for directing change and charting progress must themselves resemble the features of dynamic, democratic systems. Planners and evaluators are increasingly challenged to make sense of complicated situations, while preserving scientific credibility and acting swiftly to confront urgent public problems. Participants in this dialogue will reflect on emerging priorities in their work, while examining alternative ways of thinking about and studying health systems. Informed by CDC's ongoing study of syndemic thinking, this session will open a dialogue about the pressures and possibilities that are currently transforming our view of health system change and public health work.

Bobby Milstein leads the Syndemics Prevention Network and coordinates planning/evaluation activities for emerging investigations and policy initiatives at the Centers for Disease Control and Prevention. With an educational background that combines cultural anthropology, public health, systems science, and social navigation, Mr. Milstein is a participant observer who both studies and leads innovation at the CDC and beyond. The Syndemics Prevention Network explores transformations in public health work that arise in situations where there are multiple interacting epidemics, or syndemics. The project involves a close analysis of the concepts, methods, and moral considerations that guide public health work, along with a synthesis of techniques from other areas of applied science (e.g., system dynamics, democratic organizing, social navigation). Members of the network aim to join the science of epidemiology with the pragmatic action of citizen leaders and thereby expand the range of problem-solving strategies available for protecting the public's health in its widest sense.

Click here for the handouts!

Using Evaluation Theory to Map Evaluation Practice

Level: All

Far from being the mere musing of academics, good theory can be a useful map for practice. Theories may be informal or grand, loosely or tightly constructed, or focused on the individual or the collective. Whatever their form, theories intend to explain why relationships work the way they do. “The fundamental purpose of program evaluation theory is to specify feasible practices that evaluators can use to construct knowledge of the value of social programs that can be used to ameliorate the social problems to which programs are relevant” (Shadish, Cook, and Leviton, 1991). Tucked into this wordy purpose are the components of good evaluation theory: (1) programming, (2) valuing, (3) knowledge, (4) utilization, and (5) practice. An understanding of these core concepts in evaluation cuts across multiple texts, authors, practitioners, content areas, and contexts to an understanding of what makes something evaluation in the first place and the implications for evaluation practice. Drawing on the Foundations of Program Evaluation this session will explore core concepts in evaluation through discussion, lecture, and an unusual group task.  Participants will take away from this session more options for their own evaluation practice and perhaps some thoughts on their own evaluation theory.

Judith Ottoson, Ed.D., M.P.H . is currently an Associate Professor at the Institute of Public Health, Georgia State University, and a lecturer at San Francisco State University .  Her research and teaching focus on how ideas expressed as theory or policy are translated into practice. She explores this relationship through evaluation research in public health, health promotion, and adult education. Substantive research areas have included injury prevention, diabetes programming, school health, and substance abuse prevention. She is the co-author of the text Community and Population Health. Before entering academia, Dr. Ottoson had 20 years of diverse practical experience in the public and private sectors as an administrator, educator, and evaluator.  She holds a doctorate from Harvard's Graduate School of Education and a Master's degree in Public Health Education from the University of Hawaii.

Click here for the handouts!

Identifying Promising Practices for Public Health Program: Evaluation Approaches
This breakout is full. Registration is no longer available for this breakout.

Level: All

Complicated public health problems often are addressed by implementing multi-faceted programs in multiple locations. A key evaluation question is “what works” and what lessons can be learned from individual settings that are worthy of replication by others. T his session will expand participant’s methodological tool kit beyond the traditional expert panel and explore a range of different evaluation methodologies for identifying promising practices. Participants will explore these methods and how best to incorporate them in designing evaluations that support the identification of promising practices. The course will also address:

• Approaches to integrating frontline public health staff in the process.

• Defining promising practices within the context of the project, and recognizing that this is an evaluation task in itself.

• Reporting and disseminating findings from a promising practices investigation.

Anne Powers, PhD is a Senior Health Research Scientist at Battelle’s Centers for Public Health Research and Evaluation. Dr. Powers is a Social Psychologist with research interests in intimate partner violence, adolescent risk behavior, and adolescent and child health. She has conducted evaluation research for many government agencies including CDC, HRSA, NHLBI, NIAID and USAID utilizing mixed-method designs incorporating quantitative, qualitative and participatory methods. Currently, Dr. Powers is leading an evaluation of intimate partner violence prevention programs in family planning clinics, community health centers and county health departments. Using the evaluation findings, she is developing promising practices in intimate partner violence prevention for publicly funded health care centers. Since 2002, Dr. Powers has been a member of Battelle’s Institutional Review Board. She is also an ad hoc reviewer for Qualitative Health Research.

Click here for the handouts!

Applying Appreciative Inquiry in Evaluation Practice

Level: All 

Appreciative Inquiry (AI) has been used in the organizational change and development field for nearly twenty years as a means for creating a future based on positive past experiences. It is a collaborative process that inquires into, identifies, and further develops the best of “what is” in organizations in order to create more successful experiences, outcomes, and results. Appreciative Inquiry conducted within an evaluation context has the capacity to: a) teach organization members about evaluation (build their evaluation capacity), b) affirm that there are many ways to solve a problem or address an issue, c) help organizations move from lethargy to action, and d) provide organization members with hope and a vision of what can be, based on their own authentic successful and positive experiences. An advantage of applying AI to evaluation is its infinite flexibility to various evaluation contexts, purposes, cultures, and stakeholder groups. In this interactive workshop, participants will learn about the underlying principles and processes of Appreciative Inquiry, benefits of using AI in evaluation, and how to apply AI to: 1) focus an evaluation and to develop an evaluation plan, 2) design and conduct interviews and surveys, and 3) design an evaluation system.

Hallie Preskill, Ph.D., is a Professor in the School of Behavioral and Organizational Sciences at Claremont Graduate University. Her books include: Reframing Evaluation Through Appreciative Inquiry (Preskill & Catsambas, 2006), Building Evaluation Capacity: 72 Activities for Teaching and Training (Preskill & Russ-Eft, 2005), Evaluation Strategies for Communicating and Reporting (Torres, Preskill & Piontek, 2005), Evaluation in Organizations (Russ-Eft & Preskill, 2001), and Evaluative Inquiry for Learning in Organizations (Preskill & Torres, 1999). She received the American Evaluation Association’s Alva and Gunnar Myrdal Award for Outstanding Professional Practice in 2002, and the University of Illinois Distinguished Alumni Award in 2004.  For over 20 years she has provided consulting services and workshops in the areas of program evaluation, organizational learning, appreciative inquiry, and training design and design. She has conducted evaluations in schools, healthcare, non-profit, human service, and corporate organizations.  She is currently President-Elect of the American Evaluation Association.

Click here for the handouts!

Communicating and Reporting Evaluation Processes and Results

Level: All

Communicating evaluation processes and results is one of the most critical aspects of evaluation practice. This is especially true when evaluation use is an expected goal of evaluation; that is, intended use by intended users. Yet, evaluators continually express frustration with the number of hours they spend on writing reports that are seldom read, or shared with others. While final reports will continue to be an expectation of many evaluation contracts, there are many other ways in which evaluators can communicate and report on the progress and outcomes of an evaluation. Such alternative methods may increase: a) the evaluation’s credibility, b) stakeholders’ understanding of evaluation processes, c) use of the evaluation’s findings, and d) action on the evaluation’s recommendations. In this session, participants will: a) explore reasons for communicating and reporting throughout an evaluation’s life cycle, consider how stakeholders’ characteristics and information needs influence the choice of communicating approaches, and learn about more than 15 strategies for communicating and reporting evaluation processes and findings.   

Hallie Preskill, Ph.D., is a Professor in the School of Behavioral and Organizational Sciences at Claremont Graduate University. Her books include: Reframing Evaluation Through Appreciative Inquiry (Preskill & Catsambas, 2006), Building Evaluation Capacity: 72 Activities for Teaching and Training (Preskill & Russ-Eft, 2005), Evaluation Strategies for Communicating and Reporting (Torres, Preskill & Piontek, 2005), Evaluation in Organizations (Russ-Eft & Preskill, 2001), and Evaluative Inquiry for Learning in Organizations (Preskill & Torres, 1999). She received the American Evaluation Association’s Alva and Gunnar Myrdal Award for Outstanding Professional Practice in 2002, and the University of Illinois Distinguished Alumni Award in 2004.  For over 20 years she has provided consulting services and workshops in the areas of program evaluation, organizational learning, appreciative inquiry, and training design and design. She has conducted evaluations in schools, healthcare, non-profit, human service, and corporate organizations.  She is currently President-Elect of the American Evaluation Association.

Click here for the handouts!

Building Convincing Evaluation Arguments

The real practice of client-centered evaluation is not about conducting field experiments, it is fundamentally about the construction of convincing evaluation arguments that assist clients in making decisions and taking action. This work requires deliberate problem and context analysis, clear delineation of the evaluand and relevant claims about it, careful selection and implementation of inquiry designs to assess those claims, and the development and critical comparison of subsequent multiple lines of argument. Participants in this workshop will be introduced to a strategy for constructing local, context-sensitive, and case-specific evaluation arguments. Aspects considered include (1) the essential characteristics of this form of evaluation practice, (2) the types of claims comprising evaluation arguments, (3) the levels of evidence associated with various claims, (4) how claims form lines of argument, and (5) the criteria employed in the comparative evaluation of multiple lines of argument.

Nick L. Smith, Ph.D. is a Professor in the School of Education at Syracuse University where he teaches graduate courses in research and evaluation methods. His research interests focus on the critical analysis of evaluation theory, the development of alternative evaluation methods, and descriptive studies of evaluation practice. He has received research and professional service awards from the Evaluation Network, the Association of Teacher Educators, the American Psychological Association, and the American Evaluation Association.  Nick served as president of the American Evaluation Association in 2004. He has also served on numerous editorial boards, including as former editor-in-chief of New Directions for Evaluation. He has published extensively, authoring and editing several volumes in evaluation including Varieties of Investigative Evaluation. Currently, he is co-editing with Paul Brandon a book entitled Fundamental Issues in Evaluation.

Click here for the handouts!

Facing Evaluation Challenges in the Real World: A Case-Based Approach

Level: Beginner

This session will use case study analysis to highlight strategic, ethical, and methodological challenges that evaluation practitioners encounter in the real world and explore effective strategies for meeting those challenges.  As a result of class discussion, case analysis and small group activities, participants will be able to:

• Discuss ethical, strategic and methodological challenges associated with evaluation practice

• Propose strategies for meeting those challenges

• Discuss proactive strategies for ensuring an effective and useful evaluation study.

Iris Smith, PhD, MPH, holds a doctorate in Community Psychology from Georgia State University and a Master’s Degree in Public Health from Emory University. She is currently an Associate Professor and Director of the Career Master of Public Health Program (CMPH) at Emory University’s Rollins School of Public Health, where she also teaches a graduate level online course in Evaluation Research.  Her current evaluation projects include the cross site evaluation of the American Cancer Society’s National Urban School Health Leadership Institute. She also serves as the Co- Director of the Evaluation Core for the Emory Prevention Research Center and  Evaluator for the Emory Center for Public Health Preparedness. Previously, Iris was the Director of National Evaluation Services for the American Cancer Society, and has also served as a Deputy Commissioner for the Georgia Department of Juvenile Justice. 

Click here for the handouts!

Analysis Techniques for Multi-site Interventions

Level: Intermediate

Assessing the impact of multi-level, multi-site programs and initiatives presents many challenges. One consistent challenge for large-scale program evaluations is identifying appropriate methodologies for  SEQ CHAPTER \h \r 1conducting evaluation research across multiple funded sites—applying a systematic and robust assessment approach across sites while acknowledging the variation that naturally occurs among sites. We will discuss the selection of evaluation research methods and analysis strategies that address this challenge.  Specifically, we will discuss issues involved in the selection of quantitative methodologies for use in assessing program impact across multiple levels of analysis, particularly in the case of system change initiatives; and we will address solutions to the analytic challenges these data introduce.

Robert Stephens, Ph.D., M.P.H., is a Technical Director at ORC Macro working primarily on a national multi-site evaluation in children’s mental health. He has 20 years experience as a research psychologist in a variety of settings providing consultation on experimental design and statistical analysis of longitudinal data. He has used hierarchical linear modeling, growth mixture modeling, and other techniques to describe the characteristics of children with serious emotional disturbance and to examine changes in outcomes. Susan Moss, M.S., is a senior data analyst experienced in statistical analysis and data management for public health research studies and programs. She has worked on multi-site studies and longitudinal surveys in HIV prevention, chronic disease, and mental health in the US and Africa. She has analyzed outcomes assessing depression, sexual risk behavior, medication adherence, and health-related quality of life using generalized estimating equations (GEE), weighted GEE, linear and non-linear mixed models, and various multivariate methods.

Click here for the handouts!

Issues in Instrument Development

Level: All

Before program managers can use evaluation information to guide programmatic decision-making, they must be assured that the information obtained is accurate. A major factor in the accuracy of information is the instrument through which it was collected. This is true for qualitative as well as quantitative information. There are a number of issues that evaluation managers and practitioners must address with respect to instrument design. One is the biases inherent in creating an interview guide for a qualitative study. Another is the balance between including sufficient information on an instrument to ensure data quality versus overtaxing respondents or meeting the demands of federal agencies or institutional review boards. A third issue is incorporating the developmental level of the respondents into the creation of the instrument. A fourth issue is how best to assess reliability and validity. An additional issue is pilot testing of the instrument (especially with a limited population) and when and how to include pilot data in subsequent analyses. This course will explore the decision-making process for addressing issues such as these.

Nancy Thompson, Ph.D. has been an Associate Professor of Behavioral Sciences and Health Education, jointly appointed in Epidemiology, in the Rollins School of Public Health of Emory University since 1983. Prior to her appointment at Emory, she spent 15 years with CDC working in areas as diverse as laboratory testing and tuberculosis control. In 1998, under an Interagency Personnel Agreement with the CDC, she authored Demonstrating Your Program’s Worth: A Primer on Evaluation for Programs to Prevent Unintentional Injury. Dr. Thompson has been involved in developing and conducting surveys in a wide range of areas such as consumers’ health insurance information needs, the end-of-life care practices of health care professionals, and knowledge about head and spinal cord injury. Dr. Thompson served as a faculty mentor for the American Cancer Society’s Collaborative Evaluation Fellows Program throughout its existence, and has taught courses in evaluation and research design as well as social behavior, behavioral theory, behavioral epidemiology, basic epidemiology, and statistics.

Click here for the handouts!

Building Evaluation Capacity Among Communities, Partners, and Grantees: Sample Tools and Resources

Level: All

When programs are implemented at the “frontlines,” building consensus approaches to evaluation is crucial to ensure findings turn into program improvement.  This session  profiles examples of tools to help partners and communities do good, consistent evaluation.  The TB Evaluation Tool Kit is a family of resources to help TB programs in all sectors and levels define goals and objectives, set an evaluation focus, and select good performance indicators.  The kit is an inventory of resources that can be used by programs regardless of skill level or budget.  The Intervention Evaluation Map (IEM) helps states in CDC’s Nutrition and Physical Activity Program to Prevent Obesity and Other Chronic Diseases align goals and objectives with national program goals, describe their interventions, and identify evaluation objectives and outcomes that inform key evaluation questions.   Presenters in this session will discuss the history of these tools, their intended use, demonstrate/display them, and talk about lessons learned in using them.

Maureen Wilce is Team Leader and Kai Young is an Evaluation Specialist with the  Program Evaluation Team, Field Services and Evaluation Branch, in CDC’s Division of TB Elimination. Sue Lin Yee, formerly with CDC’s Division of Nutrition and Physical Activity, currently works in CDC’s Coordinating Center for Terrorism Preparedness and Emergency Response. Barri Burrus is a community health psychologist with RTI International.

Click here for the handouts!