|
|
2007
Summer Evaluation Institute |
|
||||||||
Agenda: AEA/CDC 2007 Summer Evaluation Institute This page identifies includes the schedule and session descriptions for the 2007 AEA/CDC Summer Evaluation Institute. |
||||||||||
| Sunday, June 10 | 9:00 - 4:00: Pre-Conference Workshop - Evaluation 101 |
|
Monday, June 11 |
7:30 - 8:30: Check-in, Pick-up Materials (coffee/tea available) 8:30 - 9:15: Keynote: Jean King 9:25 - 12:45: Training Rotation I (light break 10:45 - 11:05) 12:45 - 2:15: Lunch together, included in registration 2:30 - 4:00: Breakout Rotation I |
|
Tuesday, June 12 |
7:30 - 8:30: Continental Breakfast Available 8:30 - 9:15: Keynote: Jennifer Greene 9:25 - 12:45: Training Rotation II (light break 10:45 - 11:05) 12:45 - 2:15: Lunch together, included in registration 2:30 - 4:00: Breakout Rotation II |
|
Wednesday, June 13 |
7:30 - 8:30: Continental Breakfast Available 8:30 - 11:50: Training Rotation III (light break 10:00 - 10:20) 12:00 - 12:45: Keynote: Kathryn Newcomer |
|
SCHEDULE OVERVIEW ¨ WORKSHOP ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS
Evaluation 101: An Overview for New Evaluation Practitioners
We will emphasize the early steps, including identification and engagement of stakeholders, creation of logic models, and selecting/focusing evaluation questions. Several public health case studies will be used both as illustrations and as an opportunity for participants to apply the content of the course.
Audience:
Attendees with little or no background in evaluation and
working in any context although the examples used will be
from public health.
Offered:
SCHEDULE OVERVIEW ¨ WORKSHOP ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS ¨ INDEX OF CONCURRENT SESSIONS BY TIMESLOT SCHEDULE OVERVIEW ¨ WORKSHOP ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS ¨ KEYNOTE SESSION DESCRIPTIONS What’s the Use? Maximizing the Impact of Program Evaluation
Description: Understanding the history of evaluation use and influence can help evaluators take advantage of what the field has learned in the 35 years since Carol Weiss asked two heart-wrenching questions about the role of evaluation in decision making: “Is anybody there? Does anybody care?” By the 1990s, the field accepted a three-faceted concept of evaluation use with instrumental, conceptual, and political/persuasive/symbolic use as categories. Recent discussion has included two new types of use: process use, and imposed or carrot-and-stick use. More importantly, the new millennium has brought about a flurry of theoretical activity that has re-conceptualized the field’s understanding of its impact. What has emerged is an integrated understanding of evaluation’s consequences using the concept of “evaluation influence” as a unifying construct. Mark and Henry map out a logic model for evaluation, focusing solely on evaluation consequences related to the improvement of social conditions and seeking to identify the mechanisms through which evaluations lead to this ultimate goal along differing paths of influence. These ideas have direct implications for practicing evaluators, including the importance of being explicit about evaluative purposes, involving intended users and other stakeholders in evaluation decisions, and having a clear sense of an evaluator’s role in a given project.
Jean A. King, Ph.D., is a Professor in the Department of Educational Policy and Administration at the University of Minnesota where she serves as the Director of Graduate Studies and Coordinator of the Evaluation Studies Program. She holds an M.S. and Ph.D. from Cornell University and prior to her graduate study taught middle school English for a number of years. In 1995, her work using participatory evaluation methods resulted in the Myrdal Award for Evaluation Practice from the American Evaluation Association, and in 1999, she was awarded the Association’s Robert Ingle Award for Extraordinary Service. Professor King received the University of Minnesota, College of Education and Human Development’s Beck Award for Outstanding Instruction in 1999, the College’s 2002 Distinguished Teaching Award, and the 2005 Community Service Award. She is the author of numerous articles and chapters and, with Laurie Stevahn, continues writing a book on interactive evaluation practice.
Offered as Keynote Address:
Evaluation and the Public Good
Description: In whose interests is evaluation conducted? In what ways does and should evaluation serve the public interest? Is an evaluation practice that serves the public good inherently democratic? Should evaluation aspire to democratic aims? This presentation will take up these issues, with a focus on the inherent and inescapable political stances and value commitments of all evaluations and thus the equally inescapable responsibility to position our work in service of the public good.
Jennifer C. Greene received her doctorate in educational psychology from Stanford University in 1976 and has held academic appointments at the University of Rhode Island, Cornell University, and presently the University of Illinois, Urbana-Champaign. Her evaluation scholarship has broadly focused on probing the intersections of social science method with policy discourse and program decision-making, with the intent of making evaluation useful and socially responsible. Dr. Greene has concentrated specifically on advancing qualitative, mixed method, and democratic approaches to evaluation. She has published widely in journals and books on program evaluation, has held leadership positions in AERA and AEA, and was recently co-editor-in-chief of New Directions for Evaluation.
Offered as Keynote Address:
The Current Climate for Public Program Evaluation
Description: Dr. Newcomer will discuss the current environment and challenges for program evaluation and performance measurement in the government and the nonprofit sector in the U.S. She willidentify consequences of programmatic measurement on the behavior of: oversight officials, clients or consumers, and program staff, and focus on how choices are made regarding:
Kathryn Newcomer, Ph.D. is the Director of the doctoral program in Public Policy and Administration, and Associate Director of the School of Public Policy and Public Administration, at the George Washington University where she teaches program evaluation, research design, and applied statistics. She is a sought-after trainer on the topics of performance measurement and program evaluation and has authored or co-authored numerous journal articles as well as five books, among them: The Handbook of Practical Program Evaluation (1994, 2004), and Meeting the Challenges of Performance-Oriented Government (2002). She was identified as one of the top 25 evaluation experts in the country in 2001 by the American Journal of Evaluation, is a Fellow of the National Academy of Public Administration, and currently serves on the Comptroller General’s Educators’ Advisory Panel. Dr. Newcomer earned her Ph.D. in political science from the University of Iowa.
Offered as Keynote Address:
SCHEDULE OVERVIEW ¨ WORKSHOP ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS ¨ CONCURRENT SESSION DESCRIPTIONS
Exploring Effect Size and Measures of Association
Level: Intermediate
Description: Answer the call to report effect size and association measures as part of your evaluation results. This workshop will improve your capacity to understand and apply a range of measures including: standardized measures of effect sizes proposed by Cohen, Glass, and Hedges; Eta-squared; Omega-squared; the Intraclass correlation coefficient; and Cramer’s V. Through mini-lecture and demonstration you will improve your understanding of the theoretical foundation and computational procedures for each measure. The session will include: definitions of and procedures for computing a range of effect size and association measures, a presentation that examines the relationships among the common measures, and description of computation of selected confidence intervals for effect sizes and association measures. You will receive SPSS and SAS software program codes for performing many of the computations related to the measures and common confidence intervals.
Audience: Attendees with an understanding of basic statistics through regression and working in any context
Jack Barnette, PhD has served as a faculty member at Penn State University, University of Virginia, University of Memphis, University of Alabama at Tuscaloosa, University of Iowa, and is now Senior Associate Dean for Academic Affairs and Professor of Biostatistics at the University of Alabama at Birmingham. He has served as an APHA Statistics Council Member and Section representative to the APHA Action Board. Presently, he is chairing the ASPH biostatistics competency workgroup and is co-chair of the ASPH Biostatistics/Epidemiology Section. He has more than 30 years experience in teaching, advising students, and applying research, evaluation, and statistical methods to a wide variety of educational and public health projects. He has conducted evaluations of projects funded by CDC, HRSA, SAMHSA, NHLBI, and NIOSH. He serves on three of the ASPH/CDC Preparedness Exemplar Groups: Education and Evaluation Methods, Certificate Programs, and University-based Student Preparedness. He has been conducting research on the use of effect sizes and measures of association for the past eight years and he has presented pre-sessions on this topic at the last four AEA annual meetings. He holds the PhD in Educational Research and Development from Ohio State (1972).
Offered (Two Rotations of the Same Content - Do not register for both):
Integrating Program Evaluation, Planning, and Performance Measurement
Level: All
In large organizations, those doing evaluation, performance measurement, and strategic planning should be each other’s best customers. Instead, they often operate in hermetically-sealed boxes; as a result, plans are often uninformed by the evidence and evaluation and performance measurement results fall on barren soil because no one has been involved who is in a position to nurture and use results. This session will show how utilization-focused evaluation approaches—CDC’s Framework for Program Evaluation in Public Health being one—help ensure that evaluation results influence program action. The session will emphasize planning and evaluation as an iterative cycle and the steps to creating a strong evaluation plan—including stakeholder engagement, program description/logic modeling, and focusing an evaluation—payoff as well in defining performance measures and identifying areas for strategic action. Also, how taking an intentional iterative approach is not only good for the program, but can increase the “brand” for both evaluation and strategic planning.
Audience: Those new to the field of evaluation and working in any context or those with some experience in the field of evaluation but with interest in how to extend their work to include performance measurement and planning.
Thomas Chapel, MA, MBA, is a Senior Evaluation Scientist in the Office of Workforce and Career Development at the Centers for Disease Control and Prevention. His primary responsibility is to help build the capacity of CDC and its programs by: helping them with strategic planning and evaluation design; building networks on evaluation and planning across CDC’s many programs; and providing a range of training and facilitation services in strategic planning and evaluation design. David Cotton, PhD, is a Senior Vice President in the Applied Research Division at Macro International in Atlanta where he directs a portfolio of evaluation work for federal and non-federal clients. He and his staff provide training and technical assistance to community-based organizations and health departments in applied monitoring and evaluation.
Offered (Two Rotations of the Same Content - Do not register for both):
Level: Intermediate
Description: Learn the theory-driven approach for assessing and improving program planning, implementation and effectiveness. You will explore the conceptual framework of program theory and its structure, which facilitates precise communication between evaluators and stakeholders regarding evaluation needs and approaches to address those needs. Mini-lectures, group exercises and case studies will be used to illustrate the use of program theory and theory-driven evaluation for program planning, initial implementation, mature implementation and outcomes. In addition, the participants will learn principles and strategies for using the theory-driven approach to deal with the following cutting edge issues: how to go beyond traditional methodology for designing a real world evaluation, how to achieve both internal and external validity in an evaluation, and how to use program theory for guiding the application of mixed methods in an evaluation.
Audience: Attendees with a basic background in logic models and/or program theory.
Huey Chen, PhD, is a professor at the School of Public Health at the University of Alabama at Birmingham. Dr. Chen worked at the University of Akron until 1997, when he joined the CDC as chief of an evaluation branch. He took a leadership role in designing and implementing a national evaluation system for evaluating CDC-funded HIV prevention programs based in health departments and community-based organizations. Dr. Chen has contributed to the development of evaluation theory and methodology, especially in the areas of program theory, theory-driven evaluations, and evaluation taxonomy. His book Theory-Driven Evaluations has been recognized as one of the landmarks in program evaluation and his newest text, Practical Program Evaluation, offers an accessible approach to evaluation for those working in any context. In 1993 he received the AEA Paul F. Lazarsfeld Award for contributions to Evaluation Theory and in 1998 he received the CDC Senior Biomedical Research Service Award.
Offered (Two Rotations of the Same Content - Do not register for both):
· Tuesday, June 12, 9:25 – 12:45 (20 minute break within) · Wednesday, June 13, 8:30 – 11:50 (20 minute break within)
Level: Beginner Many evaluators and program staff are concerned with designing evaluations that are intended to inform more immediate decision-making and promote organizational change. A goal, then, is to offer the most useful information to answer the evaluation questions given the program context and the resources available to conduct the investigation. This session will focus on developing participants’ understanding of, and methods for increasing, evaluation use. We will begin by providing a theoretical framework for understanding and promoting evaluation use. Employing interactive and small group exercises, we will examine strategies and techniques for increasing the use of both the evaluation process and findings. Upon completion of the session, you will: understand use as a primary purpose of evaluation, understand the difference between evaluation process and evaluation findings use, become familiar with a framework for promoting use, and identify strategies for promoting and increasing evaluation use. Audience: Those who are new to evaluation, including stakeholders in the evaluation design process, who are working in any context. Christina A. Christie is an Assistant Professor, Director of the Masters of Arts Program in Psychology and Evaluation, and Associate Director of the Institute of Organizational and Program Evaluation Research at Claremont Graduate University. Her research, which has been supported by several funders including the Haynes Foundation, focuses on investigating the relationship between evaluation theory and practice and issues related to the development of descriptive theories of evaluation. She has also received funding from a variety of sources, including the National Science Foundation, the U.S. Department of Education, and the Hewlett-Carnegie Foundation to evaluate social, education, and health behavior programs targeting high-risk and underrepresented populations. Christie co-founded the Southern California Evaluation Association, a local affiliate of the American Evaluation Association, and is the former Chair of the Theories of Evaluation Division of the American Evaluation Association. In 2004, Christie received the American Evaluation Association’s Marcia Guttentag Early Career Achievement Award. Offered (Two Rotations of the Same Content - Do not register for both):
An Introduction to Economic Evaluation
Level: Intermediate
Economic evaluation refers to applied analytic methods used to identify, measure, value, and compare the costs and consequences of prevention and treatment strategies. This course provides an overview of these methods, including cost analysis, cost-effectiveness analysis (CEA), and cost-benefit analysis (CBA) with an opportunity for hands-on application of each. You will leave understanding when and how to apply each method appropriately in a range of evaluation contexts.
Audience: Attendees with a basic background in statistical methods and understanding of evaluation.
Phaedra S. Corso, Ph.D. is currently an Associate Professor in the Department of Health Administration, Biostatistics, and Epidemiology at the University of Georgia College of Public Health. Previously, she served as the senior health economist in the National Center for Injury Prevention and Control at CDC where she worked for over a decade in the areas of economic evaluation and decision analysis, publishing numerous articles on the cost-effectiveness of prevention interventions and co-editing a book on prevention effectiveness methods in public health. She holds a Master’s degree in public finance from the University of Georgia and a Ph.D. in health policy and decision sciences from Harvard University.
Offered:
Economic Evaluation for Decision Making in Health
Level: Intermediate
Building on the understanding of Economic Evaluation methods that you gained earlier in the day during the introductory course, or that you have from your own practice or readings, this breakout will place special emphasis on the conceptual and empirical bases for alternative health-benefit measures − quality-adjusted life years (QALYS) for use in cost-effectiveness analyses and the willingness-to-pay methodology for use in cost-benefit analyses, and the ways in which such measures are used in policymaking.
Audience: Attendees working in health contexts and possessing a basic understanding of economic evaluation methodologies.
Phaedra S. Corso, Ph.D. is currently an Associate Professor in the Department of Health Administration, Biostatistics, and Epidemiology at the University of Georgia College of Public Health. Previously, she served as the senior health economist in the National Center for Injury Prevention and Control at CDC where she worked for over a decade in the areas of economic evaluation and decision analysis, publishing numerous articles on the cost-effectiveness of prevention interventions and co-editing a book on prevention effectiveness methods in public health. She holds a Master’s degree in public finance from the University of Georgia and a Ph.D. in health policy and decision sciences from Harvard University.
Offered:
Enhanced Group Facilitation: Techniques and Process
Level: All
Description: This workshop will familiarize participants with a variety of group facilitation techniques as well as the management of the facilitation process. You will learn how to choose a facilitation technique based on goals and objectives, anticipated outcome, type and number of participants, and logistics. Two to three facilitation techniques for generating ideas and focusing thoughts on both defined and undefined issues will be explored in greater detail. These techniques, and their variations, will include focus groups, item writing, and nominal group technique. Participants will also learn more about the different roles and responsibilities they may have in group facilitation (there are more than you think!), and how these roles intersect with the tasks inherent in planning and managing a group facilitation experience. Job aides and reference lists will be provided.
Audience: Attendees working in any context who work with, or expect to be working with, client groups of any size.
Jennifer Dewey, Ph.D., is a Technical Director with the research and evaluation professional services firm of Macro International, Inc.. Jennifer leads Site Management, overseeing ongoing training and technical assistance to local evaluation teams for the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program. Prior positions include Director of Internal Evaluation at Learning Point Associates, Senior Consultant at Andersen, and post-doctoral scholar at the Center for Prevention Research at the University of Kentucky. Jennifer holds a doctorate in Applied Experimental Psychology with a specialization in program evaluation. Her knowledge and skills encompass project management, proposal development, methodological and statistical design, qualitative and quantitative analysis, needs assessment, survey development, telephone and in-person interviews, and group facilitation. Jennifer has published in the Journal of Primary Prevention, American Journal of Evaluation, Advances in Developing Human Resources, and has made over 40 professional conference presentations.
Offered (Two Rotations of the Same Content - Do not register for both):
What Counts as Credible Evidence in Contemporary Evaluation Practice? Level: Beginner to Intermediate This workshop is designed to explore one of the most fundamental issues facing evaluators today, and the 4th step in CDC's Framework for Program Evaluation, what counts as credible evidence in contemporary evaluation practice? Many thorny debates about what counts as evidence have occurred in recent years, but few have sorted out the issues in a way that directly informs contemporary evaluation and evidence-based practice. Participants will come away from this workshop with an understanding of the philosophical, theoretical, methodological, political, and ethical dimensions of gathering credible evidence and will apply these dimensions to fundamental evaluation choices we encounter in applied settings. Audience: Attendees with a basic background in evaluation Stewart I. Donaldson, Ph.D. is Professor and Chair of Psychology, Director of the Institute of Organizational and Program Evaluation Research, and Dean of the School of Behavioral and Organizational Sciences, Claremont Graduate University. He has conducted numerous evaluations, developed one of the largest university-based evaluation training programs, published numerous evaluation articles and chapters, and his recent books include Program Theory-Driven Evaluation Science: Strategies and Applications (2007), Applied Psychology: New Frontiers and Rewarding Careers (2006; with D. Berger & K. Pezdek), Evaluating Social Programs and Problems: Visions for the New Millennium (2003; with M. Scriven), Social Psychology and Policy/Program Evaluation (forthcoming; with M. Mark & B. Campbell), and What Counts as Credible Evidence in Evaluation and Evidence-Based Practice? (forthcoming; with C. Christie & M. Mark). He is co-founder of the Southern California Evaluation Association and is on the Editorial Boards of the American Journal of Evaluation and New Directions for Evaluation. Offered (Two Rotations of the Same Content - Do not register for both):
Advanced Applications of Program Theory
This session is oversubscribed. There is no waiting list. Please choose an alternate. Thank you!
Level: Intermediate
Description: While simple logic models are an adequate way to gain clarity and initial understanding about a program, sound program theory can enhance understanding of the underlying logic of the program by providing a disciplined way to state and test assumptions about how program activities are expected to lead to program outcomes. Lecture, exercises, discussion, and peer-critique will help you to develop and use program theory as a basis for decisions about measurement and evaluation methods, to disentangle the success or failure of a program from the validity of its conceptual model, and to facilitate the participation and engagement of diverse stakeholder groups.
You will learn:
Audience: Attendees with a basic background in evaluation and familiarity with logic models and program theory
Stewart I. Donaldson, Ph.D. is Professor and Chair of Psychology, Director of the Institute of Organizational and Program Evaluation Research, and Dean of the School of Behavioral and Organizational Sciences, Claremont Graduate University. He has conducted numerous evaluations, developed one of the largest university-based evaluation training programs, published numerous evaluation articles and chapters, and his recent books include Program Theory-Driven Evaluation Science: Strategies and Applications (2007), Applied Psychology: New Frontiers and Rewarding Careers (2006; with D. Berger & K. Pezdek), Evaluating Social Programs and Problems: Visions for the New Millennium (2003; with M. Scriven), Social Psychology and Policy/Program Evaluation (forthcoming; with M. Mark & B. Campbell), and What Counts as Credible Evidence in Evaluation and Evidence-Based Practice? (forthcoming; with C. Christie & M. Mark). He is co-founder of the Southern California Evaluation Association and is on the Editorial Boards of the American Journal of Evaluation and New Directions for Evaluation.
Offered:
Evaluating Culturally-tailored Health Communications
Level: Intermediate
Description: This breakout session employs the definition of cultural tailoring first advanced by Pasick et al in 1996 as the recognition and inclusion of specific cultural characteristics to improve the relevance and utility of health communications. Tailoring is a procedure selected from the social marketing arsenal, and founded on the concept that any program or educational materials will be more effective when audience characteristics are taken into account in their development and dissemination. There is increasing attention being paid in the health promotion literature to the concept of culturally-relevant or culturally-appropriate programs or messages. Many of the strategies for evaluating such communications focus on fairly explicit linguistic or formatting metrics; is the language used accessible to the population, the spokespeople ethnically-appropriate, culturally-relevant constraining and facilitating factors included? This session will explore the integration and application of cultural schemas and models, concepts derived from cognitive anthropology, and the mental modeling approach, a process derived from risk communication theory, to develop and apply additional evaluative metrics for communications. The session describes how these concepts can be employed in the evaluation of culturally tailored health communication campaigns in community settings.
Audience: Attendees with a basic background in evaluation and working in the public health arena. David Driscoll, Ph.D., MPH, is an anthropologist with extensive experience evaluating tailored health promotion and risk communications in with diverse audiences including efforts to promote cancer screening, immunization, risk perception, and environmental health. Ben Beatty, Ph.D., MPH, is a health communications researcher with extensive background in developing and evaluating health communications in community settings to help reduce health disparities. Offered (Two Rotations of the Same Content - Do not register for both):
Advances in Evaluating Health Communications and Public Health Branding in Social Marketing
Level: Beginner to Intermediate
Description: This workshop provides a decision-oriented framework for determining what to measure in evaluating health communications and social marketing campaigns. Within this framework we then offer a more in-depth training in to important topics: What to assess in formative evaluation: “Beyond the traditional focus group.” We discuss innovative formative evaluation methods, including telephone and web-based focus groups, dyad and triad groups, and computer-aided message testing. We help participants to identify the complimentary strengths and limitations of various evaluation methods for different types of questions and audiences. The session concludes with training about methods to evaluate public health branding. We review recent advances in public health branding and offer theory-based, practical approaches to evaluate the contribution of brand equity to changes in health behavior. The session includes hands-on practice with methods to develop measures tailored to brand image and communication focus. We discuss effective ways to provide feedback to tailor and refine health communication efforts. Audience: Attendees with a basic background in evaluation and working in the public health arena. W. Douglas Evans, Ph.D., is Vice President of RTI’s Public Health and Environment Division. He specializes in evaluation of social marketing campaigns and the translation of commercial marketing strategies, such as branding, to public health. He is writing a book on public health branding, to be published in 2008 by the Oxford University Press. James Hersey, Ph.D., director of RTI’s Health Psychology Program, has worked extensively in evaluation of broadcast media and social marketing campaigns in immunization, obesity prevention, and tobacco control. Julia Kish-Doto, PhD, and Katherine Treiman, Ph.D., MPH, are health communications researchers at RTI in reproductive health and nutrition. Jennifer Uhrig, PhD, MHA is a Senior Health Communication Scientist at RTI whose experience includes testing and evaluation of print and Web-based interventions designed in the area of HIV prevention and informed health-decision-making. The team has worked extensively in evaluation of social marketing efforts in tobacco control, obesity, and HIV/STDs. More information on the team’s work may be found at: http://www.rti.org/experts.cfm?objectid=119713C3-4480-44D8-99300B120F72BC6D Offered (Two Rotations of the Same Content - Do not enroll for both):
Evaluating Inter-organizational Collaborations
Level: Intermediate
Description: “Collaboration” is a misunderstood, under-empiricized and un-operationalized construct. Program and organizational stakeholders looking to do and be collaborative struggle to identify, practice and evaluate it with efficacy. This workshop aims to increase participants’ capacity to quantitatively and qualitatively examine the development of inter-organizational partnerships. Together, we will review, discuss, and try out specific tools for data collection, analysis and reporting, and we will identify ways to use the evaluation process to inform and improve collaborative ventures. You will practice using assessment techniques that are currently being used in the evaluation of PreK-16 educational reform initiatives and other grant-sponsored endeavors including the Safe School/Healthy Student initiative.
Audience: Attendees with a basic understanding of organizational change theory/systems theory and familiarity with mixed methodological designs
Rebecca Gajda, Ph.D. has been a facilitator of various workshops and courses for adult learners for more than 10 years. She was a top-10 workshop presenter at Evaluation 2007 , lauded for her hands-on, accessible, and immediately useful content. As Director of Research and Evaluation for a large-scale, grant-funded school improvement initiative, she is currently working collaboratively with organizational stakeholders to examine the nature, characteristics and effects of collaborative school structures on student and teacher empowerment and performance. Dr. Gajda received her Ph.D. from Colorado State University and is currently an assistant professor at the University of Massachusetts Amherst.
Offered (Two Rotations of the Same Content - Do not register for both):
Improving Survey Quality: Assessing and Increasing Survey Reliability and Validity
This session is oversubscribed. There is no waiting list. Please choose an alternate. Thank you!
Level: Intermediate
Description: Develop higher-quality surveys! This workshop is designed to teach participants how to improve survey quality, thus increasing their utility of and confidence in the data they collect. Through the use of hands-on activities, mini-lectures, and demonstrations participants will understand what is meant by reliability and validity with respect to surveys and will learn ways to improve each during the survey design phase. Next, using a case example and SPSS we will explore ways to use pilot test responses to assess the reliability of survey constructs by conducting confirmatory factor analysis and calculating Cronbach’s alpha. We will work together to understand what our findings tell us as well as what they don’t tell us, and consider other ways to assess survey quality. Last we will explore the types of validity associated with surveys and ways to assess the various forms of validity again using our case example. You will receive a workbook and SPSS screenshots to help you remember how to perform many of the computations we will perform. Participants will be surprised by how easy it is to improve survey quality through a few easy to implement steps!
Audience: Attendees working in any context with a basic background in survey development.
Amy A. Germuth, PhD earned her PhD in Evaluation, Measurement and Psychology from UNC Chapel Hill and is a founding partner of Compass Consulting Group, LLC, a private evaluation consulting firm. As part of Compass she has evaluated numerous initiatives, including health prevention and outreach programs, Math Science Partnerships, K-12 science outreach programs, and workforce development initiatives, and has worked with a variety of organizations including the Department of Education, Department of Health and Human Services, Westat, Georgia Tech., Virginia Tech., University of North Carolina, the New York State Education Department, multiple NC Childhood Education Partnerships, and Hawaii’s Kamehameha Schools. As part of her evaluation work she has developed and guided large-scale survey initiatives including the NC and SC Teacher Working Conditions survey. Dr. Germuth is the incoming chair of AEA’s Independent Consulting TIG and teaches evaluation and instrument development as part of Duke University’s Certificate Program in Non-Profit Management.
Offered:
Advanced Concepts in Community Health Evaluation
This session is oversubscribed. There is no waiting list. Please choose an alternate. Thank you!
Level: Intermediate
In this session participants will learn different evaluation methodologies for community assessments and be able to apply multiple approaches to the evaluation of community-based health promotion programs. Among other topics, the course will address:
Audience: Attendees working with community health programs with experience conducting evaluations.
Robert M. Goodman, Ph.D., M.P.H., M.A., is a Professor and Chair, Department of Behavioral and Community Health Sciences, Graduate School of Public Health, University of Pittsburgh. Dr. Goodman has written extensively on issues concerning community health development, community capacity, community coalitions, evaluation methods, organizational development, and the institutionalization of health programs. He has been the principal investigator and evaluator on projects for CDC, The National Cancer Institute, The Centers for Substance Abuse Prevention, The Children’s Defense Fund, and several state health departments. In 2004, Dr. Goodman received the Distinguished Fellow Award from the Society for Public Health Education, the highest honor it bestows. Currently, Dr. Goodman is consulting on community-based public health practices and empowerment evaluation with the Diabetes Translation and Injury Prevention Branches at CDC. Also, he is leading an evaluation of community-based approaches to increasing interest in cancer clinical trials.
Offered:
Advanced Qualitative Evaluation Approaches for Community Health Programs
Level: Intermediate
Practical qualitative evaluation methods will be shared for community health programs. The workshop will be an overview of qualitative methods that include the use of logic models, social ecology principles, participatory-based strategies, and basic data analysis methods. The session is designed to foster creativity when planning, implementing and evaluating a community-based program. Simulation activities will be used to dramatize many of the concepts.
Audience: Attendees working with community health programs with a basic background in qualitative methods.
Robert M. Goodman, Ph.D., M.P.H., M.A., is a Professor and Chair, Department of Behavioral and Community Health Sciences, Graduate School of Public Health, University of Pittsburgh. Dr. Goodman has written extensively on issues concerning community health development, community capacity, community coalitions, evaluation methods, organizational development, and the institutionalization of health programs. He has been the principal investigator and evaluator on projects for CDC, The National Cancer Institute, The Centers for Substance Abuse Prevention, The Children’s Defense Fund, and several state health departments. In 2004, Dr. Goodman received the Distinguished Fellow Award from the Society for Public Health Education, the highest honor it bestows. Currently, Dr. Goodman is consulting on community-based public health practices and empowerment evaluation with the Diabetes Translation and Injury Prevention Branches at CDC. Also, he is leading an evaluation of community-based approaches to increasing interest in cancer clinical trials.
Offered:
Using the Guiding Principles to Improve Your Evaluation Practice
Level: Beginner to Intermediate
Description: The Guiding Principles for Evaluators focus on five areas of evaluation practice: systematic inquiry, competence, integrity and honesty, respect for people, and responsibilities for general and public welfare. The Principles guide the professional practice of evaluators, and inform evaluation clients and the general public about the principles they can expect to be upheld by professional practitioners. This session will share ways to use the Principles to improve the ways in which you plan for and conduct evaluations and work with stakeholders and clients. After a brief presentation that introduces the Principles, participants will work together in small groups to discuss the Principles as they relate to a topical case study. Through case explorations, lecture and small and large group discussions, you will gain a deeper understanding of the practical applications of the Principles. The workshop will also introduce resources—print, web-based and collegial networks—that evaluators can consult to handle professional dilemmas that arise in their practice. You will receive copies of the workshop presentation, the case study, the Principles in full and abbreviated brochure format, and a list of resources for more information and consultation.
Audience: Evaluators and commissioners of evaluation working in any context
Leslie Goodyear, PhD is a Research Scientist at Education Development Center, where she conducts research and evaluation for a large and small scale initiatives. As a program evaluator and researcher, Dr. Goodyear has worked with programs focused on HIV/AIDS Prevention; Out-of-School Time; Youth Engagement and Youth Media; Educational Research; and Science Education. Leslie is the recent past Chair of the AEA Ethics Committee and a new AEA Board member. She earned her M.S. and Ph.D. in Human Service Studies, with focus on Program Evaluation, from Cornell University.
Offered (Two Rotations of the Same Content - Do not register for both):
Employing Mixed-Methods in Evaluation
Level: Intermediate
Description: This workshop will engage participants in a mixed methods way of thinking. Conceptual questions to be addressed include, “just what is being mixed in mixed methods evaluation designs?” and “what are varied legitimate purposes for mixing methods in applied research and evaluation?” Practical issues to be discussed include alternative dimensions of, and approaches to, mixed methods design, data analysis, and representation. In particular, we will examine ways of working with and respecting multiple inquiry traditions, forms of data, and norms regarding what count as credible results or warranted assertions in evaluation. The workshop will include activities related to mixed methods evaluation planning and data analysis.
Audience: Attendees familiar with multiple evaluation methods, working in any context.
Jennifer C. Greene received her doctorate in educational psychology from Stanford University in 1976 and has held academic appointments at the University of Rhode Island, Cornell University, and presently the University of Illinois, Urbana-Champaign. Her evaluation scholarship has broadly focused on probing the intersections of social science method with policy discourse and program decision-making, with the intent of making evaluation useful and socially responsible. Dr. Greene has concentrated specifically on advancing qualitative, mixed method, and democratic approaches to evaluation. She has published widely in journals and books on program evaluation, has held leadership positions in AERA and AEA, and was recently co-editor-in-chief of New Directions for Evaluation.
Offered:
Sampling 101: Basics of Probability and Purposeful Sampling
Level: Beginner
Description: Choosing and implementing an appropriate sampling strategy can affect the validity, credibility and cost of an evaluation. Some studies require sophisticated probability sampling methods to produce accurate estimates of the characteristics of the populations served or of the size of the effects of the program or policy on the target population. Other studies may appropriately use purposeful samples to support theory development or to do detailed case analysis. In this workshop, you will be exposed to alternative sampling strategies that are frequently used in evaluation and social research. The instructor will address the 14 questions from his book Practical Sampling (Sage, 1990) that should be answered prior to sample design, as a part of sample design, and prior to analysis of the data. You will become acquainted with ways to plan and implement sampling strategies that meet the needs of an evaluation. Examples will be used to illustrate the designs and issues that arise in implementation and you will have the opportunity to raise specific sampling issues that encountered in your own work.
Audience: Attendees who are new to sampling and working in any context
Gary T. Henry is a professor in the Andrew Young School of Policy Studies at Georgia State University and, as of July 1, 2007, the William Neil Reynolds Distinguished Visiting Professor at the University of North Carolina at Chapel Hill. He previously served as the Director of Evaluation and Learning Services for the David and Lucile Packard Foundation. Henry has evaluated a variety of public policies and programs and is the author of Practical Sampling (Sage 1990), Graphing Data (Sage 1995) and the co-author of Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs (Jossey-Bass 2000). He received the Evaluation of the Year Award from the American Evaluation Association in 1998 for his work with the Georgia’s Council for School Performance and the Joseph S. Wholey Distinguished Scholarship Award in 2001 from the American Society for Public Administration and the Center for Accountability and Performance. Dr. Henry currently serves on scientific peer review panels for the Institute of Education Sciences, U.S. Department of Education and on a National Research Council/National Academy of Science panel assessing the effects of “green schools” on the health and productivity of teachers and students.
Offered (Two Rotations of the Same Content - Do not register for both):
Economic Solutions to OMB PART Evaluations Level: Intermediate
Description: OMB’s Program Assessment Rating Tool (PART) is a uniform series of questions asked to each program in the Federal system. Although, certain questions differ between broad categories of federal programs, the PART is based on the theory that the efficacy and value of disparate federal programs can be directly compared to one another using identical criteria within government program categories. Conceptually, this is similar to economic cost-effectiveness theory which argues that interventions for a wide range of health outcomes can be directly compared using common summary measures such as the cost per life year saved and the cost per quality adjusted life year saved. In this breakout, we describe the economic evaluation methods used to help the 317 Immunization Grant Program respond to their 2002 OMB PART evaluation. These methods include econometric fixed-effects models to demonstrate efficacy, the development of summary cost-effectiveness measures, and the analysis of studentized residuals to explore variations in individual program performance. The session will identify the conceptual issues related to separating the impact of a program from the impact of environment, and will provide examples of how qualitative choices about the definition and measurement of the program and its outcomes can influence quantitative evaluative results.Audience: Attendees with a basic background in evaluation and working in federal government contexts. Drs. Thomas J. Hoerger, Amanda A. Honeycutt, and David B. Rein are senior research staff in RTI International's Public Health Economics Program. Combined they bring approximately 25 years of experience conducting economic evaluations for CDC Clients. Dr. Rein's work focuses on outcomes-research, cost-effectiveness simulations, applied econometrics, and program evaluation. He has published on the National Immunization Program's response to their initial OMB PART evaluation. Dr. Honeycutt is the Director of RTI's Economics and Prevention Effectiveness task order contract with the Centers for Disease Control and Prevention (CDC) which has sponsored over 50 economic evaluation task orders for CDC. Dr. Hoerger is an RTI Senior Fellow and Director of the RTI-UNC Center for Excellence in Health Promotion Economics, specializes in health economics, hospital and physician behavior, health care reform, and cost-effectiveness analysis. Offered:
Essential Competencies for Program Evaluators
Level: All
Description: This workshop is designed to teach participants the Essential Competencies for Program Evaluators, a set of knowledge, skills, and attitudes in six categories. The session will begin with the analysis of program evaluation vignettes representing diverse areas of practice to show both the common competencies across settings and those unique to specific contents or contexts. Following a brief history of how the competencies were developed, the session will then examine the competencies in all six categories: professional practice, systematic inquiry, situational analysis, project management, reflective practice, and interpersonal skills. This discussion, which builds on the continuum of interpersonal evaluation practice, will ground participants in the competencies’ content and allow people to ask questions as they think about their own evaluation practice. After a short break, participants will develop concept maps to explore how the competencies make sense in their roles or content areas. Comparative discussion will further illuminate the competencies, and then participants will complete a self-assessment tool and discuss how to set priorities and action steps for professional development. Most of the session will consist of interactive exercises with just enough lecture to frame the discussion.
Audience: All evaluators, and those thinking about entering the field of evaluation, working in any context
Jean A. King, Ph.D., is a Professor in the Department of Educational Policy and Administration at the University of Minnesota where she serves as the Director of Graduate Studies and Coordinator of the Evaluation Studies Program. She holds an M.S. and Ph.D. from Cornell University and prior to her graduate study taught middle school English for a number of years. In 1995, her work using participatory evaluation methods resulted in the Myrdal Award for Evaluation Practice from the American Evaluation Association, and in 1999, she was awarded the Association’s Robert Ingle Award for Extraordinary Service. Professor King received the University of Minnesota, College of Education and Human Development’s Beck Award for Outstanding Instruction in 1999, the College’s 2002 Distinguished Teaching Award, and the 2005 Community Service Award. She is the author of numerous articles and chapters and, with Laurie Stevahn, continues writing a book on interactive evaluation practice.
Offered (Two Rotations of the Same Content - Do not register for both):
Strengthening Evaluation Through Cultural Relevance and Cultural Competence
Level: Intermediate
Description: This skill-building session addresses the centrality of culture in evaluation. It is organized in two segments. The opening segment addresses the relevance of culture to all stages of the evaluation process, to the fundamental validity of our work as evaluators, and to ethical standards and guidelines of our profession. Presenters will use an FAQ format to raise questions and address common misconceptions that marginalize discussions of culture within the evaluation community (e.g., Is “culture” really just a code-word for “race”? How does culture apply to me as a white evaluator working within predominantly white populations? What is the “value added” of culture in evaluation? Why should I care?) The second segment extends cultural relevance to present strategies for building cultural competence through experience, education and self-awareness. Theoretical frameworks that situate culture in evaluation (e.g., Frierson, Hood & Hughes, 2002; Hall & Hood, 2005; Kirkhart, 2005) are presented as advance organizers for practice and application purposes. Presenters use case scenarios and participants’ own examples to integrate workshop content with participants’ field experience, interests, and concerns. They rely on various theoretical frameworks to guide the two segments in tangible and practical ways. Additional resources are provided to extend and reinforce participant learning.
Audience: Attendees working in any context with a basic background in evaluation.
Karen E. Kirkhart holds a Ph.D. in Social Work and Psychology from The University of Michigan and is currently Professor, School of Social Work, College of Human Services and Health Professions, Syracuse University. Rodney K. Hopson has undergraduate and graduate degrees in English Literature, Educational Evaluation, and Linguistics from the University of Virginia, and he is Associate Professor and Chair, Department of Educational Foundations and Leadership and faculty member in the Center for Interpretive and Qualitative Research at Duquesne University. Karen and Rodney have served in positions of leadership within the American Evaluation Association, and both are actively involved in education and scholarship on culture, diversity, and social justice in evaluation. Rodney serves as Project Director for the American Evaluation Association/Duquesne University Graduate Education Diversity Internship Program. Karen is a member of the AEA Diversity Committee task force charged with developing a public interest statement on the subject of cultural competence and evaluation.
Offered (Two Rotations of the Same Content - Do not register for both):
Speednetworking: Building Your Evaluation Network
Level: All
Description: Making professional connections can be a challenge. Looking for the right opening, hoping not to impose, not being sure if someone has the information you seek, feeling shy or awkward, dreading a drawn out conversation that is off topic, all can inhibit our ability to make contacts and build professional networks. This session will bring speednetworking, popular already in business, to the summer institute. We’ll provide a structured process through which you can connect with, and talk directly to, at least 25 other attendees in 90 minutes and share both your own background as well as any burning information needs that you have. Bring a notebook and pen, business cards if you have them, and any burning questions or information needs that you hope you may be able to glean from other attendees. Speednetworking is an active, fun, process – please come prepared to move about and to talk one-on-one with multiple other attendees. This session runs at the end of the day on both Monday and Tuesday and we invite attendees to reconnect at the end of the breakout time to add depth to the quick exchanges that have taken place.
Audience: Attendees working in any context.
Susan Kistler is the Executive Director of the American Evaluation Association. She has taught evaluation and methodology courses at the University of Minnesota, worked as a contracted workshop facilitator for the Corporation for National Service, and offered training on association business practices for the American Society of Association Executives.
Offered:
Qualitative Evaluation in the Real World
Level: Intermediate
Description: Qualitative evaluation allows program planners and evaluators to apply qualitative methods to the questions and concerns of those in the world of practice. Participants will learn under what circumstances qualitative methods are appropriate in the real world, how to balance the limitations of field data collection, ethical considerations that must be taken into account and how to focus and present the evaluation findings to meet stated and unstated program priorities. Case studies, both domestic and international, will be presented to add context to the step by step process that will be discussed.
Audience: Attendees with a basic background in qualitative research and working in any context.
Cynthia Klein, PhD, a research and evaluation project director for Constella Group, currently manages the evaluation of CDC’s Choose Respect Initiative and the Heads Up: Concussion in High School Sports Toolkit projects. Her areas of expertise include qualitative and quantitative research methodologies, health communication evaluations, field data collection, and research planning. She holds a PhD in Social Psychology from the University of Florida. Pat Shifflett, RN, MS, provides consulting for program planning and development projects for Constella Group. Ms. Shifflett has extensive experience in designing, implementing and evaluating community-based interventions. In addition, she has expertise in organization and change management, using data for decision-making and public health planning. A public health nurse for over 20 years, Ms. Shifflett, also, has a graduate degree in Organization and Management from Capella University.
Offered (Two Rotations of the Same Content - Do not enroll for both):
Level: Intermediate
Description: In this session we will create a context for performance improvement work in the public health sector and explore the methodologies of systems dynamics modeling as a means of helping to frame the strategic questions and interventions necessary for impacting public health outcomes. Examples will be drawn from diabetes, school health, cancer, and HIV/Aids. Mr. Klein will engage participates in exercises that illustrate how to use these methodologies with a diverse audience of stakeholders to gain a better understanding of the whole system before intervening. In addition, we will explore approaches that allow the participants to help their constituency make more explicit their theories for intervening in the system. Lastly, we will explore ways that you can start using systems thinking approaches immediately, without having to have an advanced degree to engage your constituencies.
Audience: Attendees working in public health contexts with a basic background in data collection techniques.
Kevin “Doc” Klein, PhD. Doc Klein founded Uncharted Territories, Inc in 1997, a consulting firm dedicated to helping communities, organizations, and agencies create their desired futures using a whole systems approach. Since 2002, he has provided consultation to the CDC’s Diabetes Brach as part of a partnership with the Center for Public Health Practice at the Rollins School for Public Health at Emory University. Working closely with CPHP and the Sustainability Institute out of Vermont, Doc and team developed a systems dynamics model that explores the critical interrelationships and insights of Type II diabetes and the possible places to intervene in the system. In addition, they have worked closely with the staff over the years to develop their strategic vision and organizational structures. Lastly, they are currently helping them implement a series of pilot programs with the states using the systems dynamic model and insights to facilitate strategic conversation for improving health system performance.
Offered:
Level: Intermediate to Advanced
Description: Building on the insights from the first session we will help participants think critically about the step-by-step process for population health performance improvement using a system dynamics modeling approach. Each participant will have the opportunity to work on issues relevant to their interests. We will walk through a process of (1) Learning to Better See the Whole System, (2) Getting Clear About Desired System Outcomes, (3) Exploring Where to Intervene in the System, and lastly (4) Clarifying What Contributions Will Be Needed From Stakeholders to Succeed. Individual or small group work will be included and Mr. Klein will provide coaching and support during these hands-on sessions. Note: For those who join us having not participated in the previous session, a brief introduction and review will be included, but it will be helpful to have attended the session entitled "Integrating Systems Thinking into Strategic Planning Work: A Primer for Using Systems Thinking Tools and Methodologies to Improve System-wide Performance Within the Field of Public Health" in order to maximize your learning.
Audience: Attendees working in public health contexts with a basic background in data collection techniques and in systems thinking.
Kevin “Doc” Klein, PhD. Doc Klein founded Uncharted Territories, Inc in 1997, a consulting firm dedicated to helping communities, organizations, and agencies create their desired futures using a whole systems approach. Since 2002, he has provided consultation to the CDC’s Diabetes Brach as part of a partnership with the Center for Public Health Practice at the Rollins School for Public Health at Emory University. Working closely with CPHP and the Sustainability Institute out of Vermont, Doc and team developed a systems dynamics model that explores the critical interrelationships and insights of Type II diabetes and the possible places to intervene in the system. In addition, they have worked closely with the staff over the years to develop their strategic vision and organizational structures. Lastly, they are currently helping them implement a series of pilot programs with the states using the systems dynamic model and insights to facilitate strategic conversation for improving health system performance.
Offered (Two Rotations of the Same Content - Do not register for both):
Conducting and Using Success Stories for Capacity Building
Level: Intermediate
In order to build program capacity a programs “success” must be told at many levels. In addition, impacts of prevention programs may not be able to be demonstrated for several years therefore communicating success during the various life stages of a program is important for long term sustainability. The presenters will use their experience with 13 national oral health grantees to demonstrate how to use success stories to build both program and evaluation capacity. The session will be a practical and hands on session enabling attendees to begin writing their own success stories. This is an expanded version of last year’s session of the same title with more time for practicing practical applications for use in your own practice. Attendees will receive the newly developed workbook: Impact and Value: Telling Your Program’s Story for use during the class and to take home for reference.
Audience: Attendees working in any context with a working knowledge of both evaluation and qualitative inquiry
René Lavinghouze is with the Division of Oral Health at CDC where she leads a multi-site, cluster evaluation designed to assess infrastructure development. Rene has over 15 years experience with CDC and in the private sector. She is Coo-chair of AEA’s TIG for Cluster, Multi-site/level evaluations and serves on the communications team for the local evaluation affiliate, AaEA. Ann Price, PhD, is president of Community Evaluation Solutions, Inc and has over 20 years experience in both treatment and prevention. She has conducted evaluations in many areas including intimate partner violence, mental health, substance abuse, tobacco prevention and oral health. Prior to CES, Dr. Price was a Senior Data Analyst at ORC Macro on a multi-site national child mental health evaluation. Kisha Ann Smith, MpH, is the data manager for the CDC’s Division of Oral Health.
Offered (Two Rotations of the Same Content - Do not enroll for both):
Principled Discovery: Learning From Your Data, with Reason
Level: Intermediate - one should take the Training or the Breakout with Dr. Mark, but not both
Description: Too often, quantitatively-trained evaluators are taught to test only a priori hypotheses. But evaluation data are too valuable not to try to learn the unanticipated lessons they may have to tell. These can be lessons about how the program operates differently in different contexts or with different subgroups of client. Or lessons about why the program works (or doesn’t). Or lessons about different subgroups of clients, or service providers, that we didn’t realize existed in advance. Some evaluators have learned ways to "snoop" in or “mine” their data. However, without ways to make the snooping/ mining principled, there are serious risks of getting it wrong.
We will explore many ways of learning from one's data, including tests of moderation; graphical methods; post hoc use of clustering methods; exploratory investigation of higher-level variables in multilevel models; and exploratory searching for mediators and countervailing effects. In addition, we will discuss multiple ways of making new discoveries principled, other than the familiar notions of replication. The session will also provide a conceptual introduction to (or review of) some current quantitative methods such as multilevel models and structural equation modeling.
Audience: Attendees with a basic background in both evaluation and quantitative data analysis through regression and working in any context
Melvin M. Mark, Ph.D. is Professor of Psychology at Penn State University, Editor Emeritus of the American Journal of Evaluation and Past-President of the American Evaluation Association. He has conducted federally funded evaluations in the areas of prevention programs for at-risk youth, federal personnel policies, and industrial modernization, and has been involved in evaluations of state and local programs. An award-winning teacher, he has published numerous papers and chapters on the theory and design of evaluation. Among his books are Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs (Jossey-Bass, 2000) and a forthcoming Handbook of Evaluation (SAGE London).
Offered:
What Works When: Unraveling How Context Affects Program Effectiveness
Level: Intermediate - one should take the Training or the Breakout with Dr. Mark, but not both
Description: How can evaluators move beyond overall tests of program effectiveness and learn what works under what conditions? And, how can they do this without being misled by chance findings that arise only because one has sliced the data repeatedly? Most standard statistical training emphasizes the testing of planned, a priori hypotheses, and understates the importance of "discovery." But if you don't explore your data, you may forego a valuable opportunity to target interventions to those for whom they are effective, avoid giving them to others for whom they are ineffective or even harmful, and perhaps help create better or cheaper programs. On the other hand, unfettered snooping and data mining creates other serious problems: Paraphrasing Stigler, if you torture your data enough, they'll confess to something, even if "not guilty". In this session we'll present an approach that (1) allows for uncovering some of the contingencies that limit average intervention effects, but (2) tries, using other data, to keep you from being misled by chance. Termed "principled discovery" (Mark et al.1998, 2000; Mark, in press), we'll look at some examples of the approach in practice, and as time allows discuss how principled discovery might be applied in the work of those at the session.
Audience: Attendees with a basic background in both evaluation and quantitative data analysis through regression and working in any context
Melvin M. Mark, Ph.D. is Professor of Psychology at Penn State University, Editor Emeritus of the American Journal of Evaluation and Past-President of the American Evaluation Association. He has conducted federally funded evaluations in the areas of prevention programs for at-risk youth, federal personnel policies, and industrial modernization, and has been involved in evaluations of state and local programs. An award-winning teacher, he has published numerous papers and chapters on the theory and design of evaluation. Among his books are Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs (Jossey-Bass, 2000) and a forthcoming Handbook of Evaluation (SAGE London).
Offered (Two Rotations of the Same Content - Do not register for both):
Building Community Evaluation Capacity in a Multi-Site Context: Case Studies in Disaster Preparedness and HIV
Level: Intermediate
Description: Building evaluation capacity in health departments, community-based organizations and community planning groups is an ongoing issue in an era of increased accountability. This session will provide an overview of understanding an organization’s level of program evaluation expertise in order to build evaluation capacity. Case studies will be presented in disaster preparedness, HIV and other community interventions in order to provide a real world context for practitioners. In addition, sample tools and templates for assessing organizational capacity readiness to conduct program evaluation will be shared.
Audience: Attendees with a basic background in evaluation and working in evaluating community interventions.
Julie Marks, PhD, a research and evaluation project director for Constella Group, currently oversees the Kent County Prevention Initiative Evaluation and the Supportive Networks of HIV Care technical assistance project. Her areas of expertise include quantitative and qualitative evaluation design and methodology, survey development & psychometrics, and designing and evaluating mediated health communications. She holds a PhD in Nutrition from the University of North Carolina, Chapel Hill. Bina Jayapaul-Philip, PhD, a senior program evaluation specialist for Constella Group, provides technical assistance to CDC grantees utilizing the CDC Framework for Program Evaluation concepts and guidelines. She provides scientific input on various phases of program evaluation projects and conducts program evaluation activities such as data collection, field activities, and development of logic models. She holds a PhD in in Sociology from Fordham University. Offered (Two Rotations of the Same Content - Do not enroll for both): · Breakout Rotation I: Monday, June 11, 2:30 – 4:00 PM · Breakout Rotation II: Tuesday, June 12, 2:30 – 4:00 PM
Logic Models as a Platform for Program Evaluation Planning, Implementation, and Use of Findings This session is oversubscribed. There is no waiting list. Please choose an alternate. Thank you!
Level: All
Practitioners use logic models to describe important components of a program; make visible a theory of change; and link activities to intended outcomes. For the purposes of evaluation practice, a well-constructed logic model provides a program-specific foundation for identifying evaluation questions; prioritizing data needs; and translating findings into recommendations for ongoing program improvement. Aimed directly at improving the utility of logic models and quality of evaluation practice in your setting, the workshop addresses 2 questions: (1) What are the hallmarks of a well-constructed, scientifically-sound and useful logic model? (2) How do we maximize the use of logic models for program evaluation planning, implementation and use of findings? Workshop Objectives: · Demystify and define the logic model as a starting point for everyday evaluation practice · Identify the hallmarks of a well-constructed, scientifically-sound logic model · Clarify the relevance of process and outcome evaluation to preparing and using program logic models · Demonstrate the use of logic models to identify and prioritize evaluation questions and data needs · Examine the use of logic models to identify opportunities/options for demonstrating accountability for scarce resources · Demonstrate use of a logic model to guide preparation of findings/recommendations aimed at ongoing program improvement · Pinpoint additional resources for continued study/application
Audience: Attendees working in any context who are new to logic modeling.
#1 Ranked Presenter from 2006! Goldie MacDonald, Ph.D. is a Health Scientist in the Coordinating Office for Global Health (COGH) at the Centers for Disease Control and Prevention (CDC). Currently, she is on assignment to the International Influenza Unit (IIU) at the Department of Health and Human Services (HHS) in Washington, DC. Dr. MacDonald leads and coordinates monitoring and evaluation of HHS and CDC investments in international avian and pandemic influenza preparedness. She provides technical expertise and practice wisdom in the areas of program evaluation planning, implementation, and the use of findings to inform ongoing program improvement. Much of her work focuses on identifying appropriate strategies to document program implementation and progress toward intended outcomes in an effort to demonstrate accountability for resources in public health contexts. She offers practical guidance on participatory approaches to program evaluation, resource-efficient strategies for data collection, and the value of logic models as a necessary platform for program evaluation. She is lead author of “Introduction to Program Evaluation for Comprehensive Tobacco Control Programs.” For their work on this resource, the authors received the Alva and Gunnar Myrdal Award for Government from AEA in November 2002.
Offered (Two Rotations of the Same Content - Do not register for both):
Building a Vision for Program Evaluation: A Systematic Approach to Increase Quality, Participation, and Consistency in Multi-Site, Multi-Level Program Evaluation
This session is oversubscribed. There is no waiting list. Please choose an alternate. Thank you!
Level: Intermediate
Most large-scale evaluation efforts require the coordinated action of multiple participants or stakeholders over time. Often, those tasked with program evaluation quickly identify intended outcomes, indicators and data collection methods; however, they may pay only limited attention to establishing an overarching vision for practice among participating sites. With regard to multi-site, multi-level program evaluation, how do we balance the need for well-coordinated, strategic action and site autonomy? This session begins with discussion of the process used to develop a vision, and related checkpoints, for evaluation practice across a multi-site, multi-level public health program—the Steps to a HealthierUS Cooperative Agreement Program. The speaker presents a set of “foundational elements” to guide program evaluation across the Steps Program, and the payoffs of the vision in terms of increased participation and consistency in evaluation activity across funded communities. Finally, the speaker makes visible the interplay of practice standards (i.e., accuracy, feasibility, propriety, and utility), and the hard choices required to actually complete the work of program evaluation.
Audience: Attendees working in any context that includes multiple actors (sites, levels) and with a basic knowledge of program evaluation.
#1 Ranked Presenter from 2006! Goldie MacDonald, Ph.D. is a Health Scientist in the Coordinating Office for Global Health (COGH) at the Centers for Disease Control and Prevention (CDC). Currently, she is on assignment to the International Influenza Unit (IIU) at the Department of Health and Human Services (HHS) in Washington, DC. Dr. MacDonald leads and coordinates monitoring and evaluation of HHS and CDC investments in international avian and pandemic influenza preparedness. She provides technical expertise and practice wisdom in the areas of program evaluation planning, implementation, and the use of findings to inform ongoing program improvement. Much of her work focuses on identifying appropriate strategies to document program implementation and progress toward intended outcomes in an effort to demonstrate accountability for resources in public health contexts. She offers practical guidance on participatory approaches to program evaluation, resource-efficient strategies for data collection, and the value of logic models as a necessary platform for program evaluation. She is lead author of “Introduction to Program Evaluation for Comprehensive Tobacco Control Programs.” For their work on this resource, the authors received the Alva and Gunnar Myrdal Award for Government from AEA in November 2002.
Offered:
Measurement for Evaluators: Key Issues in Reliability, Validity, and Instrument DesignLevel: Intermediate
The session will include informative discussion on the some key issues facing evaluators in the area of measurement. We will focus on the importance of testing instruments for reliability and validity prior to data collection, define and explain the various types of reliability and validity testing, and discuss implications of not testing instruments prior to data collection. The audience will also learn about current methods being used to assess reliability and validity. We’ll also touch on other issues in measurement that are critical for today's practicing evaluators, such as questionnaire design and using technology in data collection. The presenter will bring dual perspectives from experiences in education and public health to this discussion, and will facilitate interactive activities to engage the audience in the presentation.
Audience: Evaluators working in any context with a basic knowledge of instrument development.
Frances McCarty, Ph.D. is a Biostatistician in the Behavioral Sciences and Health Education Department in the Rollins School of Public Health. She currently serves as the statistician on several grants that focus on interventions related to HIV prevention, nutrition and exercise, and HIV medication adherence. She teaches research methods and statistics courses. Prior to her position at Emory, she was an assistant project director for a five-year grant at Georgia State University that focused on Head Start program quality and child outcomes. She holds a B.S. in Health/ Physical Education from Bridgewater College and a M.Ed. in exercise physiology from the University of Virginia. She earned her Ph.D. in measurement and statistics from Georgia State University (Educational Policy Studies Department). Her research interests include the application of item response theory to instrument development and topics related to hierarchical linear modeling.
Offered (Two Rotations of the Same Content - Do not register for both):
Introduction to Statistics for Evaluation
Level: Beginner
Description: Brush up on your statistics! We will focus primarily on descriptive statistics: measures of central tendency (e.g., the mean) and measures of dispersion (e.g., standard deviation), defining what these terms mean, when and how to use them, and how to present them to evaluation stakeholders. From there, we will move to a basic discussion of inferential statistics, the difference between a statistic and a parameter, and the use of population parameters. The workshop will involve demonstrations by the presenter using SPSS software as well as hands-on exercises for you to calculate some of the descriptive statistics we cover. This workshop is not designed to teach you how to use SPSS or any other statistical software package, but rather to introduce you to the process of statistical analysis and interpreting statistical output. We will focus on interpreting statistical indices (e.g., means and standard deviations) as well as graphical output, such as histograms and stem-and-leaf plots. The understanding you gain regarding the concepts presented in this course, coupled with the chance to interpret statistical output, should ready you to employ the most basic descriptive statistics and interpret them in your evaluation work.
Audience: Attendees who have never taken a statistics course before, or who feel they need a refresher workshop on BASIC statistics. Those who attend this workshop need no prior background in statistics
Katherine McKnight, PhD., has taught statistics workshops at AEA's annual conferences for many years. An outstanding facilitator, she is known for making difficult concepts accessible and interesting.
Offered:
Methods for Analyzing Change Over Time
Level: Intermediate
Description: We will focus on a variety of methods for analyzing change in outcomes over time, including the traditional fixed effects methods of pre/posttest ANCOVA and Repeated Measures ANOVA; the slopes-as-outcomes individual regression analysis approach; multilevel modeling and random coefficients models; and latent growth models. The purpose of the workshop is to explore the conceptual underpinnings of these different approaches to assessing change, and to compare the kinds of statistical information one is able to glean from these types of analyses when addressing questions of change. We will discuss what it means to measure change, how each method attacks that task, and how to determine which measure to use in a given situation since each method has its strengths and weaknesses with respect to its conceptual approach, parameter estimation, precision of estimates and handling missing data. Due to the nature of the topic, the majority of the workshop will involve presentations. Conceptual information, statistical output and graphs will be shared in a give-and-take format, where participants bring their own questions and concerns about analyzing change over time. Demonstration of how to set up longitudinal data for the different analytical methods will be included as well as interpreting statistical output.
Audience: Attendees with a good understanding of General Linear Models (including the ANOVA family and MRC) and some basic experience with longitudinal analysis.
Katherine McKnight, PhD., has taught statistics workshops at AEA's annual conferences for many years. An outstanding facilitator, she is known for making difficult concepts accessible and interesting.
Offered:
Using Program Evaluation to Improve Nonprofit Service Outcomes
Description: This workshop will provide an overview of the practice of outcomes assessment in nonprofit service providers. Attendees at the session will:
Kathryn Newcomer, Ph.D. is the Director of the doctoral program in Public Policy and Administration, and Associate Director of the School of Public Policy and Public Administration, at the George Washington University where she teaches program evaluation, research design, and applied statistics. She is a sought-after trainer on the topics of performance measurement and program evaluation and has authored or co-authored numerous journal articles as well as five books, among them: The Handbook of Practical Program Evaluation (1994, 2004), and Meeting the Challenges of Performance-Oriented Government (2002). She was identified as one of the top 25 evaluation experts in the country in 2001 by the American Journal of Evaluation, is a Fellow of the National Academy of Public Administration, and currently serves on the Comptroller General’s Educators’ Advisory Panel. Dr. Newcomer earned her Ph.D. in political science from the University of Iowa.
Offered:
Case Study Methods for Evaluators
Level: Beginner
Description: Case Study Methods allow evaluators to approach program assessment from a powerful and flexible design palette. While often heavily steeped in the use of qualitative methods, case studies also may include the use of quantitative data. The approach is particularly rich for tinting and shading the effects of programs as well as investigating important program questions in depth.
This interactive, three-hour session will provide participants with an overview and examples of case study research methods as they apply to evaluation settings. Through the development and expansion of sample case studies, by the end of the session participants will:
Audience: Attendees working in any context
Dr. Rita O’Sullivan is Associate Professor of Evaluation and Assessment at the University of North Carolina at Chapel Hill where she teaches graduate courses in Educational Program Evaluation, Case Study Methods, Research Design, Measurement, and Statistics. She is also Executive Director of Evaluation, Assessment, and Policy Connections (EvAP), a unit she founded within the UNC School of Education that conducts local, state, and national evaluations. Dr. O’Sullivan has specialized in developing collaborative evaluation techniques that enhance evaluation capacity and utilization among educators and public service providers. She is senior author of Programs for At-Risk Student: A Guide to Evaluation (Corwin Press, 1993) and wrote Practicing Evaluation: A Collaborative Approach (Sage) in 2004.
Offered (Two Rotations of the Same Content - Do not register for both):
Level: Intermediate
Description: Ethnography is a rich, scientifically rigorous method for situating evaluation in the context of key stakeholders and program consumers. In this workshop, participants will learn how to plan, organize, and implement team-oriented time-constrained systematic qualitative methods whose results can stand alone or complement quantitative data collection and analysis in process and outcome evaluation work. Among the topics addressed in this course will be the use of appropriate case study design, the development and use of site selection criteria, choosing and working with key informants and collaborators, qualitative data collection and data management strategies, and analysis methodologies that support evidence-based conclusions and/or recommendations. A key concern is presentation of analytic methods that will link qualitative findings to evaluation questions and conclusions with demonstrable validity and reliability. The session will include case studies, discussion, and a participatory exercise designed to illustrate the difference between development of survey items and ethnographic interviewing topic guides.
Audience: Attendees working in any context with a basic background in qualitative methodology.
Mary Odell Butler, PhD is a Research Leader in the Battelle Centers for Public Health Research and Evaluation. She has worked extensively with research design, management, and supervision of projects on dozens of evaluations for CDC over the past eighteen years. She specializes in the application of qualitative methods in mixed method designs in program evaluation, evaluation research, and case study methods. In addition to her work for CDC, she has conducted projects for the Centers for Disease Control and Prevention (CDC), the Health Resources and Services Administration (HRSA), the Environmental Protection Agency (EPA), the National Institutes of Health (NIH), and private foundations. Dr. Butler is trained as an anthropologist and frequently presents workshops on evaluation methods at the American Anthropological Association and the Society for Applied Anthropology.
Offered (Two Rotations of the Same Content - Do not register for both):
Knowledge Management 101: Taming the “Infolanche”
Level: Beginner to Intermediate
Description: Information overload–from emails, data, reports, surveys, notes, the web and more–causes frustration, costs time and money, and can sidetrack the evaluation. Knowledge management (KM) systems provide tools and techniques for taming the glut of information at our disposal. This course will provide an introduction to knowledge management (KM) and its strategic application to evaluation, illustrating how logic models can be used as the framework for designing practical knowledge management approaches to support the transfer of learning and best practices distilled from evaluation. We will begin with a brief overview of concepts, tools and techniques, followed by examples and critique of KM as applied currently in a variety of evaluation contexts – from simple document storage to comprehensive knowledge generating systems. You will have the opportunity to engage in extensive small group “hands-on” (but NO tech) discovery using a variety of case examples from small to large scale health, philanthropy and other efforts to experiment with the concepts and techniques. The workshop will conclude with a whole group reflection that contrasts the value and benefit of KM for evaluation practice in the context of real world barriers and facilitators.
Audience: Attendees with little or no prior exposure/background in knowledge management and some familiarity with logic models.
Cynthia Phillips, Ph.D. holds a doctorate in evaluation, measurement and research. Her work history includes extensive consultation with the W.K. Kellogg Foundation and other foundations to design and implement evaluation, evaluation training and knowledge management projects in health and education. She is the author of the W.K. Kellogg Foundation Logic Model Development Guide. Lisa Wyatt Knowlton, Ed.D. holds an education doctorate with specialties in management and policy. Her work history includes extensive assignments in programming and management of private, community and corporate philanthropy as well as organization development. She has managed multi-million dollar initiatives as an intermediary for the W.K. Kellogg Foundation and has served as a strategic planner, trainer, facilitator and coach to both the nonprofit and for-profit sectors the past two decades. Together, Cynthia and Lisa are the principals at Phillips Wyatt Knowlton, Inc., a consulting firm emphasizing learning strategies and systems thinking for organizations, teams and individuals.
Offered:
Gender Issues in Global Evaluation
Level: Intermediate
A brief review of the various gender and feminist approaches and their dissimilar histories, contexts, and critiques will set the stage for participant discussion and practical application. Participants will engage in facilitator led discussions regarding the different, appropriate, and practical applications of gender and feminist evaluation, and how these approaches have the potential to enhance various evaluation contexts throughout the ‘developed’ and ‘developing’ world. Participants will demystify feminist and gender evaluation by examining case studies from Africa and Asia in various fields including HIV/AIDS, human rights, public private partnerships, and environment. Through small group work participants will apply elements of both approaches resulting in an advanced understanding of how feminist and gender approaches can enhance evaluation and the evaluator in any context.
Audience: Attendees with a basic understanding of evaluation and working in all contexts although examples are drawn from international development.
Donna Podems, M.P.H., PhD. is a senior evaluation facilitator for Macro International. She has conducted trainings, developed M&E systems, and implemented evaluations for government agencies and grassroots and nongovernmental organizations. She has practical evaluation experience in the United States, Botswana, Cameroon, Ghana, Kenya, Madagascar, Namibia, Republic of South Africa, Somaliland, Swaziland, Uganda, Zimbabwe, Bosnia, Guatemala, Panama, Peru, Belize, Nicaragua, India, Indonesia, Pakistan, Sri Lanka, Thailand and Vietnam. An evaluation generalist, she has experience in researching, evaluating and/or developing M&E systems for a wide range of projects. Topics have included gender, women’s empowerment, HIV/AIDS, youth interventions, natural resource management, education, capacity building, human rights, public private partnerships, and community needs. Her doctorate in interdisciplinary studies, focused on Program Evaluation and Organizational Development, is from the Union Institute and University and her Masters degree in Public Administration and BA in Political Science are from The American University.
Offered (Two Rotations of the Same Content - Do not register for both):
Using Appreciative Inquiry in Evaluation Practice
Level: All
Description: Appreciative Inquiry (AI) has been used in the organizational change and development field for nearly twenty years as a means for creating a future based on positive past experiences. It is a collaborative process that inquires into, identifies, and further develops the best of “what is” in organizations in order to create more successful experiences, outcomes, and results. Appreciative Inquiry conducted within an evaluation context has the capacity to: a) teach organization members about evaluation (build their evaluation capacity), b) affirm that there are many ways to solve a problem or address an issue, c) help organizations move from lethargy to action, and d) provide organization members with hope and a vision of what can be, based on their own authentic successful and positive experiences. An advantage of applying AI to evaluation is its infinite flexibility to various evaluation contexts, purposes, cultures, and stakeholder groups. In this interactive workshop, participants will learn about the underlying principles and processes of Appreciative Inquiry, benefits of using AI in evaluation, and how to apply AI to: 1) focus an evaluation and to develop an evaluation plan, 2) design and conduct interviews and surveys, and 3) design an evaluation system.
Audience: Attendees with a basic background in evaluation and working in any context
Hallie Preskill, Ph.D., is a Professor in the School of Behavioral and Organizational Sciences at Claremont Graduate University. Her books include: Reframing Evaluation Through Appreciative Inquiry (Preskill & Catsambas, 2007 ), Building Evaluation Capacity: 72 Activities for Teaching and Training (Preskill & Russ-Eft, 2005), Evaluation Strategies for Communicating and Reporting (Torres, Preskill & Piontek, 2005), Evaluation in Organizations (Russ-Eft & Preskill, 2001), and Evaluative Inquiry for Learning in Organizations (Preskill & Torres, 1999). She received the American Evaluation Association’s Alva and Gunnar Myrdal Award for Outstanding Professional Practice in 2002, and the University of Illinois Distinguished Alumni Award in 2004. For over 20 years she has provided consulting services and workshops in the areas of program evaluation, organizational learning, appreciative inquiry, and training design and design. She has conducted evaluations in schools, healthcare, non-profit, human service, and corporate organizations. She is currently President of the American Evaluation Association.
Offered (Two Rotations of the Same Content - Do not register for both):
Focus Group Research: Understanding, Designing and Implementing Level: All As a qualitative research method, focus groups are an important tool to help researchers understand the motivators and determinants of a given behavior. This course provides a practical introduction to focus group research. At the completion of this course, participants will be able to 1) identify and discuss critical decisions in designing a focus group study, 2) understand how research or study questions influence decisions regarding segmentation, recruitment, and screening; and, 3) identify and discuss different types of analytical strategies and focus group reports. Audience: Attendees working in any context who are new to focus group facilitation Michelle Revels and Bonnie Bates are technical directors at ORC Macro specializing in focus group research and program evaluation. Ms. Revels attended Hampshire College in Amherst, MA and the Hubert H. Humphrey Institute of Public Affairs at the University of Minnesota. Ms. Bates, also a trained and experienced focus group moderator and meeting facilitator, received her bachelor’s and master’s degree in criminal justice from the University of Maryland. Offered (Two Rotations of the Same Content - Do not register for both):
RealWorld Evaluation I: Getting Started
Level: Intermediate
This session is oversubscribed. There is no waiting list. Please choose an alternate. Thank you!
Description: What do you do when asked to perform an evaluation on a program that is well underway? When your questions about baseline data and control groups are met with blank stares? When time and resources are few, yet clients expect “rigorous impact evaluation”? When there are political expectations and pressures to deal with? This workshop presents a seven-step approach that seeks to ensure the best quality evaluation under real-life constraints. Through presentations and discussion, with real-world examples drawn from international development evaluation, participants in this workshop will be introduced to the RealWorld Evaluation approach. The workshop focuses on developing country evaluation, but the techniques are applicable to evaluators working in any context with budget, time, data and political constraints. The fundamental framework for the session is an exploration of the seven steps of the RealWorld Evaluation approach, including ways to negotiate RealWorld solutions with clients. Finally, we’ll identify seven evaluation designs appropriate for RealWorld evaluations and discuss their appropriate context-dependent applications.
Audience:
Jim Rugh brings 42 years of experience in international development, including 26 years specializing in program evaluation. For 11 years he has been the Coordinator of program Design, Monitoring and Evaluation for CARE, a large international NGO with programs in 70 countries reaching over 48 million beneficiaries. Under Jim’s leadership CARE has adopted an evaluation policy, strategies and standards to promote learning for improved program quality and accountability for effectiveness. An active member of AEA, and a founding member of AaEA (the Atlanta-area Evaluation Association), Jim has co-facilitated or facilitated evaluation workshops at AEA’s annual conference as well as in Cape Town South Africa, Kiev Ukraine, Washington DC and Atlanta. He is the co-author with Michael Bamberger and Linda Mabry of the RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints (SAGE Publications).
Offered:
RealWorld Evaluation II: Advanced Applications
Level: Intermediate to Advanced
Description: We’ll build on the
lessons learned in RealWorld Evaluation I or your own
expertise as a seasoned evaluator to f
Audience: Attendees who took RealWorld Evaluation I yesterday or at last year’s Institute, or who have read the RealWorld Evaluation book, or who have significant experience as practicing evaluators in varied contexts.
Jim Rugh brings 42 years of experience in international development, including 26 years specializing in program evaluation. For 11 years he has been the Coordinator of program Design, Monitoring and Evaluation for CARE, a large international NGO with programs in 70 countries reaching over 48 million beneficiaries. Under Jim’s leadership CARE has adopted an evaluation policy, strategies and standards to promote learning for improved program quality and accountability for effectiveness. An active member of AEA, and a founding member of AaEA (the Atlanta-area Evaluation Association), Jim has co-facilitated or facilitated evaluation workshops at AEA’s annual conference as well as in Cape Town South Africa, Kiev Ukraine, Washington DC and Atlanta. He is the co-author with Michael Bamberger and Linda Mabry of the RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints (SAGE Publications).
Offered:
Accountability for Health Promotion Programs
This session is oversubscribed. There is no waiting list. Please choose an alternate. Thank you!
Level: Intermediate
Many health promotion programs aim for individual and systems-level change to impact public health. For accountability purposes, how do we assess whether our immediate efforts result in progress toward long-term program goals? This session will define performance measurement for health promotion programs; share practical strategies for identifying appropriate outcomes, indicators and data sources; and present examples from several CDC programs. With ever-decreasing resources for public health, decision-makers want specific types of information to assess the “value” of continued investment in disease prevention and health promotion programs. Discussion of developing and implementing performance measurement activities will highlight practical and tested action steps, real-world challenges, and lessons learned relevant to performance measurement as a tool for accountability, improvement and sustainability of health promotion programs.
Audience:
Attendees with a basic background in evaluation and
performance measurement who are working in Public Health.
Offered:
Offered:
Facing Evaluation Challenges in the Real World: A Case-Based Approach
Level: Beginner
This session will use case study analysis to highlight strategic, ethical, and methodological challenges that evaluation practitioners encounter in the real world and explore effective strategies for meeting those challenges. As a result of class discussion, case analysis and small group activities, participants will be able to:
Audience: Novice evaluators working in any context.
Iris Smith, PhD, MPH, holds a doctorate in Community Psychology from Georgia State University and a Master’s Degree in Public Health from Emory University. She is currently an Associate Professor and Director of the Career Master of Public Health Program (CMPH) at Emory University’s Rollins School of Public Health, where she also teaches a graduate level online course in Evaluation Research. Her current evaluation projects include the cross-site evaluation of the American Cancer Society’s National Urban School Health Leadership Institute. She also serves as the Co- Director of the Evaluation Core for the Emory Prevention Research Center and Evaluator for the Emory Center for Public Health Preparedness. Previously, Iris was the Director of National Evaluation Services for the American Cancer Society, and has also served as a Deputy Commissioner for the Georgia Department of Juvenile Justice.
Offered (Two Rotations of the Same Content - Do not register for both):
Lenses, Filters and Frames: Calibrating and Cultivating Self As Responsive Instrument Level: Beginner to Intermediate Description: Who are you as you walk in the world? To what extent is who you think you be in alignment with who you actually be and are striving to become as a person and as an evaluator? Does your envisioned image of self radiate and would others agree? Which others? How do you know what you think you know about this—evidential cues, clues and signposts? Engaging and using the self as responsive instrument summons one’s capacities to move beyond unilateral self-awareness and personal declarations towards multilateral self-awareness and boundary-spanning communications as you work with colleagues, clients, and stakeholders. This workshop will give you the opportunity to mindfully develop your Self-as-Instrument Portfolio: i.e., discovering, calibrating and cultivating your lenses, filters and frames. More specifically, this would involve discerning and exploring:
Audience: Attendees working in any context with a basic background in evaluation practice Hazel Symonette, Ph.D. is the Founder and Director of the Excellence Through Diversity Institute, a campus workforce learning community for faculty, staff, and administrators at the University of Wisconsin. She is an educator, researcher and intervention development and assessment specialist, as well as a Senior Policy and Program Development Specialist at the University. Dr. Symonette has offered training on the self as responsive instrument throughout the United States and in venues as far-flung as Niger, South Africa, New Zealand, and Australia. She has served as the Co-Chair of AEA’s Building Diversity Initiative and as an elected member of AEA’s Board of Directors. For ten years, she was a member of the External Evaluation Consultant Review Panel of the NSF-funded All Nations Alliance for Minority Participation in Science, Math, Engineering and Technology; for 3 years with Northwest Regional Education Lab. She serves as faculty for the Health Research and Education Trust's Cultural Competency Fellowship Program.
Offered:
Building Capacity of Community Organizations in Outcome EvaluationLevel: Intermediate Description: Since “outcomes” hit the community scene in the mid-1990’s, community organizations have been scrambling to meet expectations of funding agencies, redirect their work to be outcomes-focused, and use the outcomes movement to meet their own program improvement and learning needs. In response, a team at the University of Wisconsin-Extension has developed a comprehensive resource of activities and facilitated learning experiences to help increase the evaluation capacity of community organizations. In this interactive workshop, participants will be introduced to these materials and gain skills in a variety of practical tools and techniques for evaluating outcomes and facilitating learning about and from evaluation. The workshop uses community organization ‘cases’ to explore realities and challenges. We will model and practice strategies for effective teaching/learning within the areas of logic modeling, data collection, analysis and communications as we build our skills to strengthen the evaluation capacity of community organizations. You will take home a variety of handouts and ideas for direct application in your community setting. Audience: staff of community organizations (coalitions, nonprofits, agencies, schools, local governments) who are doing and/or managing evaluation; evaluation practitioners who are engaged in building the evaluation capacity of community organizations Ellen Taylor-Powell, Ph.D. has statewide responsibility for training and technical assistance in program development and evaluation through the University of Wisconsin – Extension. She provides training, works directly with faculty and staff on program evaluation and outcome measurement, develops resource materials, conducts high priority evaluation studies, consults with local groups, state agencies and national partners. Dr. Taylor-Powell is a sought-after trainer in the area of logic model development and use, evaluating collaborations, and evaluation in community contexts. She is past chair of the Extension Evaluator's Topical Interest Group of AEA, served as chair of the Publications Committee of AEA from 2002-2004, and was elected to the AEA Board in 2004.
Offered (Two Rotations of the Same Content - Do not register for both):
Issues in Instrument Development
Level: All
Before program managers can use evaluation information to guide programmatic decision-making, they must be assured that the information obtained is accurate. A major factor in the accuracy of information is the instrument through which it was collected. This is true for qualitative as well as quantitative information. There are a number of issues that evaluation managers and practitioners must address with respect to instrument design. One is the biases inherent in creating an interview guide for a qualitative study. Another is the balance between including sufficient information on an instrument to ensure data quality versus overtaxing respondents or meeting the demands of federal agencies or institutional review boards. A third issue is incorporating the developmental level of the respondents into the creation of the instrument. A fourth issue is how best to assess reliability and validity. An additional issue is pilot testing of the instrument (especially with a limited population) and when and how to include pilot data in subsequent analyses. This course will explore the decision-making process for addressing issues such as these.
Audience: Attendees working in any context who are new to instrument development or who wish to have a refresher on the key issues surrounding instrument development.
Nancy Thompson, Ph.D. has been an Associate Professor of Behavioral Sciences and Health Education, jointly appointed in Epidemiology, in the Rollins School of Public Health of Emory University since 1983. Prior to her appointment at Emory, she spent 15 years with CDC working in areas as diverse as laboratory testing and tuberculosis control. In 1998, under an Interagency Personnel Agreement with the CDC, she authored Demonstrating Your Program’s Worth: A Primer on Evaluation for Programs to Prevent Unintentional Injury. Dr. Thompson has been involved in developing and conducting surveys in a wide range of areas such as consumers’ health insurance information needs, the end-of-life care practices of health care professionals, and knowledge about head and spinal cord injury. Dr. Thompson served as a faculty mentor for the American Cancer Society’s Collaborative Evaluation Fellows Program throughout its existence, and has taught courses in evaluation and research design as well as social behavior, behavioral theory, behavioral epidemiology, basic epidemiology, and statistics.
Offered (Two Rotations of the Same Content - Do not enroll for both):
Evaluation Strategies for Communicating and Reporting
Level: All
Description: Communicating evaluation processes and results is one of the most critical aspects of evaluation practice. This is especially true when learning and evaluation use is an expected goal of the evaluation. Yet, evaluators continually experience frustration with hours spent on writing reports that are seldom read, or shared with others. While final reports will continue to be an expectation of many evaluation contracts, there are other ways in which evaluators can communicate and report on the progress and findings from an evaluation. Using a variety of strategies for communicating and reporting can increase learning from the evaluation’s findings, stakeholders’ understanding of evaluation processes, the evaluation’s credibility, and action on the evaluation’s recommendations. In this session, you will: explore reasons for communicating and reporting throughout an evaluation’s life cycle, consider how stakeholders’ characteristics and information needs influence the choice of communicating approaches, and learn about more than 15 strategies for communicating and reporting evaluation processes and findings.
Audience: Attendees working in any context with a basic understanding of evaluation processes as well as the relevant stakeholders to an evaluation.
Rosalie T. Torres, Ph.D. is president of Torres Consulting Group, a research, evaluation and management consulting firm specializing in the feedback-based development of programs and organizations since 1992. She earned her Ph.D. in research and evaluation from the University of Illinois and formerly was the Director of Research, Evaluation, and Organizational Learning at the Developmental Studies Center, an educational non-profit. Over the past 28 years, she has conducted more than 60 evaluations in education and nonprofit organizations, and has worked extensively with U.S. schools and districts in both internal and external evaluator roles. Drawing on this evaluation practice, she has authored/co-authored numerous books and articles including, Evaluation Strategies for Communicating and Reporting (Torres, Preskill, & Piontek, 2005), and Evaluative Inquiry for Learning in Organizations (Preskill & Torres, 1999). She has served on the AEA Board, taught graduate research and evaluation courses, and is a sought after workshop facilitator on various topics related to evaluation practice.
Offered (Two Rotations of the Same Content - Do not register for both):
Getting to Outcomes in Public HealthLevel: All
Description: Getting To Outcomes: Methods and Tools for Planning, Evaluation, and Accountability (GTO) was developed as an approach to help practitioners plan, implement, and evaluate their programs to achieve results. GTO is based on answering 10 accountability questions. By answering the questions well, program developers increase their probability of achieving outcomes and demonstrate their accountability to stakeholders. Addressing the 10 questions involves a comprehensive approach to results-based accountability that includes evaluation and much more: needs and resource assessment; identifying goals, target populations, desired outcomes (objectives); science and best practices; logic models; fit of programs with existing programs; planning; implementation and process evaluation; outcome evaluation; continuous quality improvement; and sustainability. Research funded by a grant from CDC has shown that use of the GTO model can improve individual capacity and program performance to facilitate the planning, implementation, and evaluation of prevention programs. GTO has been customized for several areas of public health including: substance abuse prevention, underage drinking prevention, positive youth development, teen pregnancy prevention, and emergency preparedness. This workshop will focus on learning the basics of the GTO approach and present a case study of how GTO is being used in a CDC-sponsored, multi-site capacity building project intended to promote the use of science-based approaches in teen pregnancy prevention.
Audience: Attendees working in public health who are new to the GTO approach.
Abraham Wandersman, Ph.D., is a Professor of Psychology at the University of South Carolina-Columbia. He received his doctorate from Cornell University. Dr. Wandersman performs research and program evaluation on citizen participation in community organizations and coalitions and on interagency collaboration. He is a co-author of Prevention Plus III and a co-editor of Empowerment Evaluation: Knowledge and Tools for Self Assessment and Accountability and Empowerment Evaluation: Principles in Practice. Catherine (Cathy) Lesesne, PhD, MPH, is a Behavioral Scientist in the Division of Reproductive Health, Applied Science Branch (ASB) and has been with the Centers for Disease Control and Prevention since 1998. Dr. Lesesne is the lead author of the newly developed GTO manual, "Promoting Science-based Approaches to Teen Pregnancy Prevention Using Getting to Outcomes". She has her PhD in Community Psychology from Georgia State University and her MPH in Public Health Administration from the University of South Carolina.
Offered (Two Rotations of the Same Content - Do not register for both):
Introduction to Qualitative Data Analysis
Level: Beginner
Description: Qualitative methods, such as interviews, qualitative responses to surveys, and focus groups, can provide a wealth of information about specific points of view, cultural contexts, and the meaning behind health beliefs and behaviors. The key to accessing this information is learning how to analyze qualitative data. This session will provide step by step lessons in basic qualitative analysis. A group activity will allow participants to practice coding qualitative data, triangulating findings between multiple analysts, and developing conclusions based upon the findings.
Audience: Attendees working in any context with little or no experience undertaking qualitative data analysis.
Beverly A. Warden, PhD, MPH, is the Director of the Surveillance and Evaluation Division at Constella Group. She is currently involved in overseeing the process, outcome and impact evaluation functions at Constella. She has planned and managed variety of evaluation projects involving collection and analysis of both qualitative and quantitative data. She holds a PhD in Biomedical Sciences from Northwestern and an MPH from the University of North Carolina at Chapel Hill. Sharon Wong, MPH, is an evaluation specialist and qualitative analyst for Constella Group. She is currently involved in planning, facilitating, and analyzing interviews and focus groups for a variety of projects. In addition, Ms. Wong conducts evaluation activities such as the development of program evaluation plans, goals and objectives, qualitative data collection tools, and survey instruments. She holds an MPH in Health and Social Behavior from the University of California, Berkeley. Offered (Two Rotations of the Same Content - Do not enroll for both):
SCHEDULE OVERVIEW ¨ WORKSHOP ¨ INDEX OF CONCURRENT SESSIONS ¨ KEYNOTES ¨ SESSIONS |
|
|