|
Session Title: Needs Assessment and Evaluation: Strategies and Practical Applications
|
|
Think Tank Session 837 to be held in International Ballroom A on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Needs Assessment TIG
, the Business and Industry TIG, and the Presidential Strand
|
| Presenter(s):
|
| Darlene Russ-Eft,
Oregon State University,
darlene.russeft@oregonstate.edu
|
| Discussant(s):
|
| Catherine Sleezer,
Baker-Hughes,
catherine.sleezer@centrilift.com
|
| Marcie Bober,
San Diego State University,
bober@mail.sdsu.edu
|
| Kelly Hannum,
Center for Creative Leadership,
hannumk@leaders.ccl.org
|
| Jennifer Martineau,
Center for Creative Leadership,
martineauj@leaders.ccl.org
|
| Abstract:
This Think Tank session will focus on issues related to needs assessment as a form of evaluation. Participants will be introduced to three types of needs - strategic or operational needs, performance needs, and learning needs. In addition, participants will receive a brief overview of four approaches to needs assessment: knowledge and skills assessment, job and task analysis, competency assessment, and strategic assessment. In the Think Tank, participants will be provided with one or more case scenarios. Then within small groups, participants will consider: (1) which kinds of needs should be considered; (2) which type or types of needs assessment are most appropriate, (3) what stakeholders would be involved in the needs assessment and how, (4) what needs assessment and evaluation tools and approaches should be used, and (5) how the proposed needs assessment design and implementation would contribute to learning among all relevant stakeholder groups.
|
|
Session Title: Applications Quasi-Experimental Evaluation Techniques
|
|
Multipaper Session 838 to be held in International Ballroom B on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Sally Bond,
The Program Evaluation Group,
usbond@mindspring.com
|
|
Findings and Methodological Lessons From an Evaluation of a Project to Integrate the Arts Into Elementary School Reading and Mathematics Instruction
|
| Presenter(s):
|
| Paul Brandon,
University of Hawaii,
brandon@hawaii.edu
|
| Brian Lawton,
University of Hawaii,
blawton@hawaii.edu
|
| Abstract:
This paper discusses the results of a quasi-experimental evaluation of an elementary-school project (with three project schools and three control schools) to improve students' achievement and attitudes toward school by integrating arts strategies into reading and mathematics instruction, and it comments on problems with conducting studies of this sort. The results of a fixed-effects analysis of differences in project outcomes among treatment and control groups show some positive results, but the results on project implementation show that the teachers did not use arts strategies frequently, that the average level of quality of implementation was middling, and that implementation showed no correlation with outcomes. We briefly describe the results in the paper, discuss their contradictions, describe the threats to the validity of the findings, and present an overview of some of the difficulties, which we believe are common, in conducting small experimental studies of this sort.
|
|
Selecting Matched Samples Using Propensity Scoring Analysis
|
| Presenter(s):
|
| Manya Walton,
Windwalker Corporation,
manya.walton@windwalker.com
|
| Abstract:
Windwalker Corporation provides technical assistance to 14 Magnet Schools Assistance Program (MSAP) grantees conducting rigorous evaluations. Most rigorous evaluation grantees are implementing quasi-experimental designs with non-equivalent control groups to evaluate the impact of their interventions on student achievement. Propensity scoring analysis was used by some grantees to select treatment and comparison students for their study sample. This presentation discusses the uses of propensity scoring in education evaluation; the propensity scoring methods applied by MSAP grantees to match treatment and comparison groups; and the result of the student matching as it relates to sampling techniques for quasi-experimental evaluations.
|
|
Applying an Experimental Design to Evaluate the Effects on Student Achievement and Attitudes of a Materials Science Program Versus a Traditional Science Curriculum
|
| Presenter(s):
|
| Kris Juffer,
Action Research & Associates Inc,
drkjuffer@comcast.net
|
| Abstract:
Using a multiple method, experimental design, Action Research & Associates, Inc. evaluated the effects of an innovative science program developed with NSF funding, Materials World Modules (MWM). Action Research designed a blind, experimental study with two-tiered-randomized sampling with students randomly selected from a pool developed across Maryland and then randomly assigned in matched pairs to Treatment and Control Groups. Data was triangulated to ensure reliable data. Action Research evaluated a the Materials World Modules Program vs. traditional science study in a 24-hour, 7-days-a-week, 4-week long immersion residential Summer Institute for 96 middle school and high school students, controlling intervening variables. MWM is a multi-disciplinary materials science program for secondary classrooms developed on the principles of hands-on, inquiry-and- design-based learning incorporating scientific problem-solving and iterative design projects and labs, including student presentations and defense of project process and outcomes. The effects of learning science via MWM compared with traditional science textbooks was explored by looking at relative changes in students' science knowledge and attitudes towards studying science. Student and educator collected information about changes in the students' interests, attitudes and experiences in science and other mediating variables, before and after experiencing Institute instruction. Student outcome variables were assessed with a science knowledge test including items assessing their abilities to problem-solve. To triangulate the data, students and teachers were administered surveys about their own experiences, their perceptions of each other, and independent researchers made classroom observations.
|
| | |
|
Session Title: Promoting Organizational Learning to Increase International Development Effectiveness: Examples From Three Catholic Relief Services Field Offices, Europe; Middle East; Southern Africa; and El Salvador
|
|
Panel Session 839 to be held in International Ballroom C on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Chair(s): |
| Carlisle Levine,
Catholic Relief Services,
clevine@crs.org
|
| Discussant(s):
|
| Guy Sharrock,
Catholic Relief Services,
gsharroc@crs.org
|
| Abstract:
In international development, a field that aims to achieve long-term and often intangible positive change in people's lives, determining whether an international non-governmental organization (NGO) is doing the right things and whether it is doing things right is both critical and hard to do. To help international NGOs understand what differences their interventions are making, identify and replicate good practices more quickly, and reduce unintended negative consequences, many are establishing systematic approaches to learning from their interventions. In this panel, representatives of three Catholic Relief Services field offices - Europe and Middle East, Southern Africa, and El Salvador - will share their experiences institutionalizing learning, highlighting factors critical to their success, challenges they have faced, and benefits they have gained from their efforts. Since many organizations are similarly institutionalizing learning to increase effectiveness, this panel will create a useful forum for discussion of good practices for accomplishing this.
|
|
Promoting a Participatory and Local Approach to Organizational Learning in Catholic Relief Services' Europe and Middle East Region
|
| Velida Dzino-Silajdzic,
Catholic Relief Services,
vdzino@eme.crs.org
|
| Meri Ghorkhmazyan,
Catholic Relief Services,
mghorkhmazyan@eme.crs.org
|
|
This presentation focuses on the advantages of taking a participatory and local approach to organizational learning, as opposed to a more global approach. This choice is based on lessons learned from measuring and analyzing progress toward achieving regional strategy objectives. The global approach successfully reveals overall regional achievements against regional objectives. However, it presents challenges for undertaking locally meaningful data analysis and interpretation, suggesting that it is difficult to use general, global findings and recommendations to inform local, program-level decision making. A monitoring and evaluation system grounded in strategies developed for smaller, sub-regional administrative units and integrated within project-based monitoring and tracking systems brings not only more accurate and timely data collection, analysis and reporting, but also increases the relevance of the learning and enhances ownership of the responses required for further improvement and aimed at maximizing the effects of the larger strategy.
|
|
|
Managing Monitoring, Evaluation and Learning within Catholic Relief Services' Southern African Regional Office
|
| Christopher Michael Reichert,
Catholic Relief Services,
creichert@crssaro.org
|
| Driss Moumane,
Catholic Relief Services,
dmoumane@crs.org
|
| Guy Sharrock,
Catholic Relief Services,
gsharroc@crs.org
|
|
Catholic Relief Services' (CRS') Southern Africa Regional Office (SARO) has embraced CRS' mandate to promote organizational learning to improve agency effectiveness by encouraging learning at the regional and country levels. Five years ago, SARO's program value expanded over tenfold, the vast majority in food security and HIV & AIDS programming. The rapid expansion was accompanied by the realization that the region needed to increase its knowledge in order to simultaneously maintain high program quality. Thus, SARO created a learning agenda around food security and HIV & AIDS, raised the profile of monitoring and evaluation, and increased its investment in its monitoring and evaluation capacity. Building on this monitoring and evaluation foundation and its learning agenda, SARO's organizational learning focuses on monitoring and evaluation to promote change. This presentation will share how SARO is piloting various learning approaches, describe the challenges of learning for change, and suggest some replicable good practices.
| |
|
Institutionalizing Learning in Catholic Relief Services' El Salvador Program
|
| Carolina Castrillo,
Catholic Relief Services,
ccastrillo@crs.org.sv
|
|
Since 2004, Catholic Relief Services (CRS)/El Salvador has strived to institutionalize learning. Obtaining management commitment, developing a conceptual framework, establishing a learning system and a team to lead the effort, identifying cross-cutting themes relevant across program sectors, and developing an implementation plan have been fundamental to institutionalizing learning. Strategies include sensitizing staff to the utility of organizational learning, conducting evaluations and feeding findings into the learning system, selecting priority learning themes in a participatory manner, and systematizing innovative and high-impact projects and processes. A variety of priority learning topics have emerged from the process: Are we really targeting the poor? Are we contributing to justice and social change? Do our services fully address people's need for integral human development? Are we honoring CRS principles in our work with local partners? Systematizing experiences is a learning and innovative process. The knowledge is enriching CRS and partners' learning agenda and program quality.
| |
|
Session Title: Still Learning, After All These Years
|
|
Panel Session 841 to be held in International Ballroom E on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Independent Consulting TIG
|
| Chair(s): |
| Deborah Bonnet,
DBonnet Associates,
dbonnet@dbonnet.com
|
| Discussant(s):
|
| Michael Quinn Patton,
Independent Consultant,
mqpatton@prodigy.net
|
| Michael Hendricks,
Independent Consultant,
mikehendri@aol.com
|
| Marilyn Ray,
Finger Lakes Law and Social Policy Center Inc,
mlr17@cornell.edu
|
| Abstract:
A panel of seasoned evaluation consultants will attempt to demonstrate that evaluation practice offers rich opportunities for life-long learning. To begin, each panelist will describe something new they learned in 2007 - a fact, principle, skill, or insight that will influence their evolving evaluation or business practices. The discussion will focus on learning as a shared value of the evaluation profession and ways to sustain our learning curve trajectories over several decades in the field.
|
|
Old Tricks, New Territory
|
| Deborah Bonnet,
DBonnet Associates,
dbonnet@dbonnet.com
|
|
Deborah Bonnet began evaluation consulting in 1974 and formed her own firm in 1982. She is now re-igniting her solo evaluation practice after a three-year stint as vice president for evaluation of a national start-up foundation. Her 2007 projects include standard fare for members of AEA: a community needs assessment and two multi-site evaluations of programs calling for school-community collaboration (one state, one national; one K-12, one postsecondary; one about college access, the other about civic engagement). She also had a rare opportunity to apply common evaluation methods to a challenge normally relegated to financial analysts: seeking explanations for a seemingly paradoxical pattern across several measures of a foundation's investment portfolio performance.
|
|
|
There's a First for Everything
|
| John Seeley,
Federal Emergency Relief Administration,
jaseeley@aol.com
|
|
John Seeley co-founded Formative Evaluation Research Associates (FERA) more than 30 years ago. FERA's mission is to enhance the performance of nonprofit organizations that are attempting to improve the quality of life for any person. Its current emphasis is to work with intermediaries that are supporting nonprofits.
| |
|
Life Lessons 2007: The Secrets of Senior Evaluators Revealed!
|
| Gail Barrington,
Barrington Research Group Inc,
gbarrington@barringtonresearchgrp.com
|
|
Based on recent case study research she conducted on lifecycle events among senior evaluators, as well as on her own reflections regarding experiences during the past rather eventful year, Gail will share some insights about what senior evaluators really think and what they want to do with the rest of their lives. While some may think about retirement, others find that their career has turned into something much bigger and much more exciting!
| |
|
Session Title: Evaluation and Social Metrics in the Nonprofit Sector
|
|
Panel Session 842 to be held in Liberty Ballroom Section A on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Chair(s): |
| Huilan Yang,
W K Kellogg Foundation,
hy1@wkkf.org
|
| Discussant(s):
|
| Victor Kuo,
Bill & Melinda Gates Foundation,
victor.kuo@gatesfoundation.org
|
| Abstract:
Are traditional approaches to program evaluation losing ground in foundations? What other methods are foundations employing to address questions of accountability and organizational learning? What, if anything, can evaluation practitioners do to deliver findings to foundations seeking timely information in a rapidly changing environment? This panel will attempt to address these questions by discussing the role and use of evaluation in the nonprofit sector, especially in private foundations, as well as the changes evaluation has been undergoing. The panelists will also explore the potential future of evaluation in foundations, with performance and social metrics gaining prominence in nonprofits as a context. Concrete examples will be given to enrich the conversation. The panel will conclude with questions, insights and comments from the audience in this subject.
|
|
Performance Metrics and Evaluation in Large Foundations
|
| Victor Kuo,
Bill & Melinda Gates Foundation,
victor.kuo@gatesfoundation.org
|
|
Over the last decade, foundations have experimented with various forms of evaluative activities. Basic outcome monitoring, referred to by some as 'performance metrics' or 'social metrics', have gained prominence in some large foundations. Dashboards and scorecards containing summaries of program outcomes, funding levels, exemplary projects, and critical contextual factors are featured. Interest in social metrics is a departure from traditional approaches, such as large, longitudinal studies. Foundation staff members have found themselves dissatisfied with these studies because of their format and lack of timeliness. Alternatives such as participatory evaluations and case studies have also been rejected. This presentation will offer examples of foundation developed 'social metrics' templates and will consider how they might be used in tandem with other types of evaluation activities. A framework for designing the evaluation function in large foundations using multiple evaluation approaches will be offered along with suggestions for evaluators on how to work with foundations.
|
|
|
Social Metrics for Accountability
|
| Jianping Shen,
Western Michigan University,
shen@wmich.edu
|
|
Foundations are conscious about social metrics for accountability. However, Social metrics for accountability are not easy to be defined in the foundation's world. Our experience illustrates that foundation staff are conscious about accountability and the metrics evolve with the development of the funded program. For example, the evaluation of Kellogg Foundation's Unleashing Resources Initiative was first on the fidelity of program implementation. Then, in consultation with the foundation staff, we added a component of the amount of resources unleashed and how the programming investment is related to the amount of unleashed resources. More recently, we added another component to evaluate the impact on communities and individuals' lives. Our evaluation of Kellogg Foundation's School-based Health Care Policy Program first focused on policy making activities, such as youth engagement and community mobilization. Now we have begun to tally policy achievements. We will start to evaluate the impact of policy achievements.
| |
|
Social Metrics for Organizational Learning
|
| Shao-Chee Sim,
TCC Group,
ssim@tccgrp.com
|
|
Social metrics can ease concerns and get buy-ins from various stakeholders as a 'learning' tool to build capacity among foundation staff and their grantees. Some concrete examples will be offered about how to frame social metrics in a learning context. Specifically, I will draw from two recent foundation experiences in developing scorecard and success measure framework. I will discuss how the development of social metrics has been helpful in clarifying grant-making strategies as well as short-term and long-term outcomes. More importantly, I will highlight how program staff, along with evaluation staff, can benefit from working together in building their collective knowledge and capacity of evaluation. I will also discuss how grantee organizations can benefit from this learning approach. By framing social metrics in a learning context, it brings evaluation officers closer to their program colleagues while ensuring that grantees and foundations are being held accountable for their work.
| |
|
Session Title: Opportunities and Challenges: Evaluating for Impact in Formal and Informal Learning Settings
|
|
Panel Session 843 to be held in Liberty Ballroom Section B on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the AEA Conference Committee
|
| Chair(s): |
| Sylvia James,
National Science Foundation,
sjames@nsf.gov
|
| Abstract:
Public and private agencies funding education or extension in one or more of the sciences, technology, engineering and mathematics (STEM) are looking for cost effective ways to measure impact. This presentation focuses on cases from the National Science Foundation (NSF) and Cooperative State Research, Education, and Extension Service (CSREES), United States Department of Agriculture.
New evaluation and research efforts to assist 4-H (CSREES) and projects funded by the NSF's Informal Science Education (ISE) Program, Graduate Education and Human Resource Development Divisions, and the anticipated Division of Research on Learning in Formal and Informal Settings, to recognize and report the impact of S&E learning have begun. Discussion will include (1) articulating valid and measurable outcomes that occur following a formal or informal learning experience, (2) documenting project outcomes within the funding period, (4) the role of partnerships and stakeholder participation and (3) developing affordable and rigorous evaluation strategies.
|
|
Introductory Remarks From the Panel Chair
|
| Sylvia James,
National Science Foundation,
sjames@nsf.gov
|
|
The first presenter will chair the panel. Arlene de Strulle will introduce the panelists and speak briefly about the establishment of the Division of Research on Learning in Formal and Informal Settings (DRL) at the National Science Foundation.
|
|
|
Evaluating for Impact: The 4-H Science, Engineering, and Technology Initiative
|
| Suzanne Le Menestrel,
United States Department of Agriculture,
slemenestrel@csrees.usda.gov
|
| Jill Stephanie Walahoski,
University of Nebraska,
jwalahos@unlnotes.unl.edu
|
|
This presentation will describe the 4-H Science, Engineering, and Technology (SET) Initiative and the work of an interdisciplinary evaluation design team comprised of individuals both within and external to the 4-H system. The authority for the 4-H Youth Development program is vested in a cooperative structure between youth, interested adult volunteers, and the professional expertise of the land-grant universities and the United States Department of Agriculture. The foundation of 4-H Youth Development is in the practical application of the land-grant university knowledge by youth in their communities. SET is one of three national mission areas for the 4-H Youth Development program. Specifically, the presentation will describe the process of designing an evaluation for a complex and multi-layered national initiative with multiple components including a national 4-H SET launch; a professional development and training component; and a curriculum redesign among other components.
| |
|
Opportunities and Challenges Associated With the Informal Science Education Program's Transformation From Documenting Outputs to Outcomes: Experiences With the Informal Science Education Online Monitoring System
|
| Gary Silverstein,
Westat,
silverg1@westat.com
|
|
The National Science Foundation's Informal Science Education program supports projects designed to increase public interest in, understanding of, and engagement with science, technology, engineering, and mathematics. This session will examine the process by which the ISE program has shifted its emphasis from documenting outputs to measuring outcomes. Of particular interest will be the opportunities for obtaining outcome-oriented results that program officers can use to identify ISE projects with promising practices. We will also focus on the challenges that ISE projects encountered in specifying and measuring progress toward audience outcomes-including difficulty
(1) articulating valid and measurable outcomes that occur after exposure to an ISE event,
(2) documenting project outcomes within the grant period, and
(3) developing an effective and rigorous evaluation strategy.
The presentation will also describe the range of technical assistance provided during the collection required to help projects devise and measure valid and measurable ISE outcomes.
| |
|
Impacts and Legacies From Learning in Higher Education Settings
|
| Mary Frances Sladek,
NASA Headquarters,
Mary.F.Sladek@nasa.gov
|
|
This presentation will describe several evaluations underway examining the National Science Foundation's (NSF) investments in learning in institutions of higher education. For over fifty years NSF has funded directly and indirectly (e.g. via institutions), tens of thousands of individuals who pursue undergraduate and post-undergraduate education or research training in STEM and STEM education. Evaluators are looking for evidence of a initial impacts and legacies from NSF-funding that changed degree-granting department beyond number of students supported and degrees awarded and to what extent have NSF-funded projects or programs broadened participation by diverse individuals, particularly individuals traditionally underemployed in science or engineering, including but not limited to women, minorities, and persons-with-disabilities.
| |
|
Session Title: Working to Address the Evaluation Needs of More of the People More of the Time: Conducting Evaluation in Multi-site and Multi-context Settings
|
|
Multipaper Session 844 to be held in Mencken Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG
|
| Chair(s): |
| Frances Lawrenz,
University of Minnesota,
lawrenz@umn.edu
|
| Discussant(s): |
| Frances Lawrenz,
University of Minnesota,
lawrenz@umn.edu
|
| Abstract:
The Evaluation Center is currently conducting the eighth annual survey of the National Science Foundation's Advanced Technological Education (ATE) program. The ATE program goals are to increase the nation's capacity to provide advanced technological education and to increase the number and quality of skilled technicians in the workforce. This survey collects information about the general characteristics of the program's grantees and their work activities, accomplishments, and impacts. This multipaper session presents highlights of data collected in 2006 from more than 160 individual projects and sites throughout the nation. The first presentation will discuss information about project-level evaluation practices occurring in the ATE program. The second presentation will focus on findings related to workforce needs assessment activities. A final presentation will discuss how the Center has addressed the challenges of conducting an online survey and evaluation across multisite and multicontext settings and examine possible future directions for the survey.
|
|
Results From the National Science Foundation (NSF) Advanced Technological Education (ATE) Survey: Project-level Evaluation Practices
|
| Arlen Gullickson,
Western Michigan University,
arlen.gullickson@wmich.edu
|
|
This paper presents highlights of project-level evaluation practices in NSF's ATE program. The first section provides an overview of ATE expectations for evaluation and how projects report having met those requirements. Section 2 describes PI perceptions of the utility of their evaluations and the extent to which these perceptions are related to the characteristics described in Section 1. Section 3 focuses on the activities of external evaluators-PI satisfaction with these evaluators, the relationship between PI ratings and standards for sound program evaluations, whether the PIs view their evaluations as meeting ATE intellectual merit requirements, and PIs' characterizations of the attributes of their external evaluators. Section 4 summarizes findings reported in previous sections to identify strengths and weaknesses of project-level evaluations and to suggest changes for improvement in current evaluation practices.
|
|
Evaluation of Workforce Needs Assessments Conducted by Advanced Technological Education Projects and Centers
|
| Liesel Ritchie,
Western Michigan University,
liesel.ritchie@wmich.edu
|
|
This presentation reviews workforce needs assessment activities conducted by ATE grantees. The process is considered an evaluative activity with potential to concentrate the expertise of evaluators, increasing their capacity to make more meaningful summative evaluation statements about ATE program impacts. Using 2006 survey data, this paper discusses ways in which current evaluation activities might incorporate, support, and utilize workforce needs assessment to assist in determining project impact. Findings reveal that (1) a majority of grantees gather workforce needs assessment information, although there is a substantial dependency on 'weak' forms of needs assessment; (2) centers are much more likely than projects to conduct workforce needs assessment once funding is awarded; (3) a variety of factors appear to be associated with whether or not grantees conduct workforce needs assessment; and (4) the purposes for which workforce needs assessment information is considered most useful by grantees include developmental and formative evaluation activities.
|
|
Progression and Challenges of Conducting an Annual Evaluation in Multi-site Settings
|
| Barbara Wygant,
Western Michigan University,
barbara.wygant@wmich.edu
|
|
The final paper presents an overview of the annual ATE survey process and breaks down the steps of the overall work plan and its related tasks. Major discussion will focus on the survey design and review processes to illustrate the time and effort needed in the early stages of survey implementation. The importance and significance of review panel input will be discussed, along with highlights of the most recent year's activities, which involved major revisions to the annual survey as a way to improve upon previous years' data gathering efforts. The challenges in conducting this multisite evaluation and how they were addressed will be highlighted as will future opportunities in the survey implementation and analysis efforts. The project managers will also discuss getting the sample audience on board and prepared for the survey launch, conducting the survey, and data analysis techniques.
|
|
Session Title: Connecting Research and Implementation to Enhance Youth Development Programming
|
|
Multipaper Session 845 to be held in Edgar Allen Poe Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Extension Education Evaluation TIG
|
| Chair(s): |
| Melissa Cater,
Louisiana State University,
mcater@agcenter.lsu.edu
|
| Abstract:
As youth development organizations design and implement programs, it becomes imperative that those programs are evaluated. Louisiana 4-H is attempting to align their youth development program with the Essential Elements of 4-H: Mastery, Belonging, Independence, and Generosity. As such, the program has undertaken to assess and evaluate these elements, and this session will share the results of those evaluations and promising practices and lessons learned from the efforts.
|
|
High School Youth: What do They Want?
|
| Robert Richard,
Louisiana State University,
rrichard@agcenter.lsu.edu
|
| Krisanna Machtmes,
Louisiana State University,
machtme@lsu.edu
|
|
A review of Louisiana 4-H club enrollment data indicated that approximately 30% of high school enrollment in a given year is composed of youth enrolling in 4-H for the first time . In addition, approximately 80% of those youth do not rejoin 4-H the following year.
Using a separate online survey for each group (those who joined for the first time and those who opted not to rejoin after one year) youth were asked to respond to open ended questions dealing with their participation in 4-H and their perspective of the program
Results from this study will be shared with the audience as well as steps being taken by the program to increase the number of youth who remain with the program after joining at the high school level.
|
|
Evaluating A Youth Development Program for Belonging and Independence
|
| Melissa Cater,
Louisiana State University,
mcater@agcenter.lsu.edu
|
| Krisanna Machtmes,
Louisiana State University,
machtme@lsu.edu
|
|
Recent research suggests that programs which incorporate opportunities for youth to build a positive relationship with a caring adult and that are safe and inclusive lead to an increased sense of belonging in youth. Additionally, programs that allow youth to have a voice and to have the opportunity to make decisions provide youth with an increased sense of independence. This study will share the results of an evaluation of a program developed to promote belonging and independence. Examination of the data and a tool used to measure youth voice will be reported.
|
|
Promising Practices for Volunteers for Including Youth Voice in the Decision-making Process
|
| Todd Tarifa,
Louisiana State University,
tatarifa@agcenter.lsu.edu
|
| Krisanna Machtmes,
Louisiana State University,
machtme@lsu.edu
|
|
This evaluation was aimed at determining the factors that effect the level of inclusiveness of youth voice in the decision-making process in the 4-H youth development program. State and field level 4-H professionals identified potential factors that effect youth voice in the decision-making process. The information gathered was utilized in identifying the level of inclusiveness of youth voice in the decision-making process to better understand how to suit youth's needs, identify promising practices, and barriers. When examining 4-H youth development professional's perceptions they considered the following factors: the level in which both youth and adults share responsibility, lack of transportation, the ability of youth and adults to work in partnership, the opportunity for youth to develop a caring relationship with adults, an adult's expectations of youth roles within the 4-H program, youth's expectations of adult roles within the 4-H program, the level in which the organization accepts youth involvement in the decision- making process, and youth having too many scheduling conflicts. This presentation will aid youth development professionals in identifying issues that limit youth voice in the decision-making process and developing promising practices that will aid them in assuring that the 4-H program actively encourages youth involvement at all levels of the decision-making process.
|
|
The Questioning Mind: Utilizing Reflection to Promote Critical Thinking
|
| Janet Fox,
Louisiana State University,
jfox@agcenter.lsu.edu
|
| Melissa Cater,
Louisiana State University,
mcater@agcenter.lsu.edu
|
|
In today's world, youth often look for instant solutions to complex problems. Reflection is an important means by which students integrate prior knowledge and experiences with new experiences to develop critical thinking and problem solving skills. Reflection in service-learning provides students and teachers a way to look back at their experiences, evaluate them, and apply what is learned to future experiences. Through reflection, students build skills for analyzing and solving problems and developing creative solutions. Reflective activities that are designed well and implemented thoughtfully allow students to acquire a deeper understanding of the world around them and of how they can make positive contributions to society. Based on practical experience coupled with a research-based foundation, this session will focus on the lessons learned and the elements needed to prepare an organization, youth and adult evaluators for successful reflection either in a service-learning project or in an evaluation project.
|
|
Now What? We Have the Pieces. Can We Complete the Puzzle?
|
| Karol Osborne,
Louisiana State University,
kosborne@agcenter.lsu.edu
|
| Melissa Cater,
Louisiana State University,
mcater@agcenter.lsu.edu
|
| Robert Richard,
Louisiana State University,
rrichard@agcenter.lsu.edu
|
|
This paper will present a synthesis of the lessons learned from other presentations in this session and indicate how those lessons will be used as part of the program development process to enhance youth development programming at the local level. The implications for statewide programming efforts will also be shared.
|
|
Session Title: Storytelling in Program Evaluation: Putting the Timeless Miracles Into the Particulars of Experience
|
|
Panel Session 846 to be held in Carroll Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Chair(s): |
| Nuria Ciofalo,
The California Endowment,
nciofalo@calendow.org
|
| Abstract:
The awareness of the need to develop culturally appropriate evaluation approaches has promoted new ways of assessing program impacts. Some of these innovative evaluation approaches have focused on the inclusion of diverse voices and perspectives through the use of popular knowledge. Others are increasing the evaluation capacity of grassroots organizations to utilize their own resources and ways of learning and reflecting to conduct their own evaluations. Storytelling is one of these emerging, innovative, and culturally appropriate evaluation approaches. This presentation will describe four forms of storytelling: (1) the narrative approach as found in reports written by grantees to describe their grant accomplishments, (2) an innovative way of telling "digital stories" using technology, (3) community-based organizations' stories applied to evaluation, and (4) the use of popular theater as a community learning and evaluation tool.
|
|
Why Include Stories in Program Evaluation?
|
| Joseph Tobin,
Arizona State University,
jtobin@asu.edu
|
|
Storytelling is found in every culture, but that does not mean that each culture tells the same kind of stories in the same way. If storytelling is to be allowed and encouraged in the program evaluations serving multicultural groups, we should allow, encourage, and expect the tellers of these stories to do so in culturally patterned ways. To be a good storyteller requires learning the cultural conventions and mastering culturally specific forms of stories. Even amateur storytellers when they tell stories cannot help but to draw on storytelling conventions of their culture. Thus, storytelling values and respects diverse ways of knowing and learning, is empowering and participatory, and based on popular knowledge. Stories can be used effectively in program evaluations alongside statistics and surveys. They can help figure out what's working, what's not and why within the particularities of unique cultural contexts, including stakeholders' voices and perspectives
|
|
|
Digital Storytelling: Empowering Community Residents to Tell Their Stories and Getting Funders to Listen and Act Upon These Stories
|
| Zoe Clayson,
Abundantia Consulting,
zclayson@abundatia.org
|
|
Photovoice is a participatory research and popular education methodology that employs photographs to explore complex concepts. It involves community members in three consecutive processes: taking photographs that illustrate participants' responses to research questions, sorting and verbally processing the photographs in an interactive group setting, and relaying the resulting themes and concepts to initiative stakeholders and the community. The central assumption behind photovoice is that images can tell stories, teach, and inspire dialogue in ways that words often cannot. This becomes particularly salient in contexts in which multiple languages, literacy levels, and cultural assumptions are present. Photographs can also influence policy by documenting and communicating community members' needs and perspectives. This paper will present a photovoice project conducted by Abundantia Consulting in 2006 for the California Endowment's Poder Popular initiative. The photograph analysis allowed residents to share their views and encouraged 'bottom-up' dialogue.
| |
|
Community-based Organizations Tell their Stories
|
| Michael Lyde,
Role of Men,
mlyde562@charter.net
|
| Larry Ginn,
Role of Men,
larry_guinn@longbeach.gov
|
|
This session will describe the timeless miracles into the particulars of experience. Grantees of The California Endowment who have used storytelling will describe their experiences in using this tool in program evaluation, community learning, and community advocacy.
| |
|
Popular Theater: Moving Ahead with Storytelling and Social Change
|
| Mily Trevino,
Organizacion en California de Lideres Campesinas,
milyliderescampesinas@msn.com
|
|
Grantees of The California Endowment will describe their timeless miracles into the particulars of experience through the use of popular theater. This strategy, applied to evaluation, can not only stimulate community learning but also spark community action for social change.
| |
|
Session Title: National Performance Evaluation System of Research and Development Programs in Korea: System and Applications
|
|
Multipaper Session 847 to be held in Pratt Room, Section A on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Sang-Youb Lee,
Korea Institute of Science and Technology Evaluation and Planning,
sylee@kistep.re.kr
|
| Discussant(s): |
| Jiyoung Park,
Korea Institute of Science and Technology Evaluation and Planning,
jypark@kistep.re.kr
|
| Abstract:
There has been increased requirement on the accountability and efficiency of national R&D programs in according to the increased R&D investment within the limited national resources. To cope with these issues, Korea is also trying to upgrade its performance evaluation system under the guidance of NSTC(National Science and Technology Committee).
There are several points to be considered in Korean R&D evaluation system. Firstly individual program evaluation is working very well, but these evaluation methods are not systematically organized in the national evaluation system(NES). Secondly most of the current evaluation is focused on the monitoring of the R&D programs, therefore there is little in-depth performance evaluation to improve the programs.
To deliver these problems, Korean government set up systematic performance evaluation methodology in national scale. In this session, several performance evaluation methods and applications were introduced
|
|
Method and Application of the Survey and Analysis of National Research and Development (R&D) Programs for the Performance Evaluation in Korea
|
| Keun-Ha Chung,
Korea Institute of Science and Technology Evaluation and Planning,
khchung@kistep.re.kr
|
| Hyejung Joo,
Korea Institute of Science and Technology Evaluation and Planning,
hjoo@kistep.re.kr
|
| Herin Ahn,
Korea Institute of Science and Technology Evaluation and Planning,
herini@kistep.re.kr
|
|
The Survey and Analysis of national R&D programs has been performed by Korea Institute of S&T Evaluation and Planning (KISTEP) since 1999, in order to enhance the efficiency and effectiveness of national R&D investment as national R&D budget increases.
Every government-funded research project is the subject of the survey and analysis, and Korea Research and Development Integrated Management System (KORDI) is used for collecting the required data such as program information, project information and performance information. Using the collected data, annual government R&D investment is analyzed in the area of socio-economic objectives, research activities, research performing sectors and technology classifications, etc.
In this study, the result of the survey of the national R&D investment is analyzed and especially the feedback system of this survey and analysis to the annual performance evaluation and budget allocation for R&D programs is organized to maximize the efficiency of R&D investment.
|
|
The Performance Evaluation of Research and Development Programs in Korea
|
| Seong-jin Kim,
Korea Institute of Science and Technology Evaluation and Planning,
shaqey@kistep.re.kr
|
| Soon Cheon Byeon,
Korea Institute of Science and Technology Evaluation and Planning,
sbyeon@kistep.re.kr
|
|
Korea has a unique centralized overall evaluation and coordination system for the national research and development programs. Korea's national research and development evaluation system is administered by the National Science and Technology Council (NSTC). The evaluation of the important specific R&D programs are performed every three years, especially focusing on the performance of those programs, and the other programs are self-evaluated by each ministries followed by meta-evaluation performed by NSTC.
The purpose of this study is to set up guidelines for the performance evaluation of the specific programs and self-evaluation of the other programs in the national R&D evaluation system. This study also includes evaluation methods for the national research and development programs.
In conclusion, we hope that our guidelines and methods can contribute to the advancement of national R&D evaluation system.
|
|
Design of Metaevaluation Model for National Research and Development Programs in Korea
|
| Young Soo Ryu,
Korea Institute of Science and Technology Evaluation and Planning,
soory@kistep.re.kr
|
| Soon Cheon Byeon,
Korea Institute of Science and Technology Evaluation and Planning,
sbyeon@kistep.re.kr
|
| Byung Dae Choi,
Hanyang University,
choibd@hanyang.ac.kr
|
|
This study presents Meta-Evaluation Model as an evaluation tool for National R&D programs by examining the essence of R&D evaluation. In Korea, an in-depth analysis of the national R&D has been carried out partially since 1999. And since 2006, targets of in-depth analysis have been reduced and the Meta-Evaluation has been performed for the efficiency of evaluation of National R&D programs. Now, as a 2nd year, it is the right time to study on the Design of Meta-Evaluation Model in depth.
In this study, the characteristics of the meta-evaluation in R&D programs were derived considering the essence of R&D evaluation theoretically. Also a meta-evaluation model for National R&D programs was derived and the evaluation indicators were developed. The meta-evaluation model was also verified in the National R&D program evaluation system. The verification of evaluation indicators were performed by Delphi method utilizing meta-evaluation committee members for the National R&D program.
|
|
Development of the Evaluation Methodology for the Basic Research in Korea
|
| Hyeyoung Yang,
Korea Institute of Science and Technology Evaluation and Planning,
hyyang@kistep.re.kr
|
| Sangki Jeong,
Korea Institute of Science and Technology Evaluation and Planning,
sjeong@kistep.re.kr
|
|
The characteristics and outcomes of the basic research are quite different from those of the applied research or product development. However, the similar evaluation methodology has been generally applied to the basic research as compared to the others without consideration of its characteristics. This makes an obstacle to a quality improvement of the results of the basic research. The present study was undertaken to investigate the characteristics of the basic research and its outcomes through a survey of researchers in Korea and a network analysis methodology with SCI DB for the basic research evaluation. In this study, the implications will be presented about the basic research evaluation as well as the quality improvement of the outcomes of the basic research.
|
|
Performance Evaluation of Agriculture Research and Development Programs in Korea
|
| Hoijong Jung,
Korea Institute of Science and Technology Evaluation and Planning,
jhj119@kistep.re.kr
|
|
Agriculture R&D programs are mainly supported by the government, the Ministry of Agriculture and Forest and the Rural Development Administration because of a long-term investment and a high risk of the private research. Therefore agricultural R&D programs of the agencies are important for improving agricultural technology in Korea. The purpose of the present report is to suggest a new revenue for improving efficiency of programs by reflecting unique evaluation indices on a portion of unique features of agriculture R&D programs and evaluating its performance. According to the features, the results reveal that the overall performance of R&D effort is more positive than what has been known. It could provide the direction of a specific program with a variety of information about the performance.
This approach could be taken into account in improving the national evaluation system (NES) of government R&D programs.
|
|
Session Title: Outcomes Measurement in Human Services Evaluation: An Introduction
|
|
Skill-Building Workshop 848 to be held in Pratt Room, Section B on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Human Services Evaluation TIG
|
| Presenter(s):
|
| James Sass,
LA's BEST After School Enrichment Program,
jim.sass@lausd.net
|
| Abstract:
Identifying and measuring appropriate program outcomes are foundational processes in the work of evaluation. This workshop, designed for persons new to the field of evaluation, will promote the development of basic knowledge and skills for outcomes measurement in the human services and related fields. Beginning with a brief introduction to the concept and field of evaluation, the workshop will address major topics in outcomes measurement. These topics include the definition of outcomes, the limitations of traditional program tracking methods, logic models, identifying appropriate indicators, choosing measurement tools, and using results. After receiving an overview to outcomes measurement, workshop participants will work in groups to practice determining program outcomes and potential indicators. Participants will receive handouts on key concepts and terminology in outcomes measurement, as well as a bibliography of practical evaluation resources. Successful participants will gain knowledge and confidence for conducting outcomes measurement in their own evaluation work.
|
|
Session Title: Crossing the Threshold: Addressing Methodological, Institutional and Cultural Challenges in Environmental Evaluation in an Era of Performance Management
|
|
Panel Session 850 to be held in Hopkins Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Environmental Program Evaluation TIG
|
| Chair(s): |
| Katherine Dawes,
United States Environmental Protection Agency,
dawes.katherine@epa.gov
|
| Abstract:
Policy makers and program managers in the fields of natural resource conservation and environmental management are experiencing a growing need to assess impacts of existing environmental programs and to build capacity for assessing current and future programs. To learn more systematically from our collective experiences, the U.S. EPA and National Fish and Wildlife Foundation developed a network for environmental and conservation evaluators and a second networking forum for environmental, natural resource, and conservation evaluators and evaluation consumers last June. The previous year's forum brought together over 90 participants across four countries and two continents from the public, non-profit, academic, and private sectors. The presentations in this panel session focus on institutional and methodological insights that arose from this forum and areas for future contributions to the environmental and conservation evaluation field.
|
|
The Methodological Challenge of Estimating Net Impacts of Conservation Efforts: A Meta-Evaluation Analysis of Ten Impact Studies Conducted by the National Fish and Wildlife Foundation, 2003-2007
|
| Matthew Birnbaum,
National Fish and Wildlife Foundation,
matthew.birnbaum@nfwf.org
|
|
The field of environmental program evaluation is nascent with no dominant paradigm yet emerging. Many conservation efforts begin without any attempts at developing reliable baseline measures for monitoring progress or incorporating any other type of replicate. Consequently, problems of scaling, time horizons, and attribution present particularly huge methodological challenges for estimating net impacts of program portfolios and the projects contained in them. Given a general absence of baselines and controls, the evaluators have exercised a substantive level of creativity in discerning patterns in learning from past investments. This paper presents key findings of the methods, designs and processes adopted by evaluators for ten impact studies conducted by the National Fish and Wildlife Foundation during 2003-2007 to arrive at estimates of net impacts given the methodological realities confronting the investigators. It concludes with recommendations for moving the field of evaluation practice forward in the new milieu of environmental outcome-based grant making.
|
|
|
Environmental Program Evaluation Practice and Theory: Gaps and Overlaps
|
| Matt Keene,
United States Environmental Protection Agency,
keene.matt@epa.gov
|
|
To address the dearth of evidence-based decision making, the environmental community is demanding more quality evaluations of environmental programs. Researchers are supporting the demand by providing strategic recommendations for a diverse audience of practitioners including managers, researchers, and units responsible for evaluation. Here, the United States Environmental Protection Agency's evaluation unit, the Evaluation Support Division (ESD), reviews ten recommendations and presents them from a practitioner's perspective. In a real world' scenario, we discuss our role and responsibility for acting on recommendations and attempt to stoke the dialogue between researchers and practitioners. Our review considers overlap and gaps between strategic guidance and its implementation within ESD. From this review and our experience, we conclude that establishing a common vision for the future of evaluation is paramount to efficiently improving the practices of the environmental community.
| |
|
The Environmental Evaluator's Networking Forum: Insights and Future Applications
|
| Katherine Dawes,
United States Environmental Protection Agency,
dawes.katherine@epa.gov
|
|
Katherine Dawes is the Director of the Evaluation Support Division at the U.S. Environmental Protection Agency which is co-hosting the environmental evaluator's forum. Evaluations overseen or conducted by this division are used to improve the quality of public health and environmental protection by supporting innovation, encouraging continuous improvement, and informing management decisions. Ms. Dawes also serves as the chairperson of AEA's Environmental Topical Interest Group. In both these capacities, Ms. Dawes works at the forefront of the emerging environmental program and policy evaluation field and brings a critical perspective on the use environmental evaluation practices and methods in a major federal agency.
| |
|
Session Title: Evaluating Educational Programs in the Department of State's Bureau of Educational and Cultural Affairs: Multiple Goal Sets and Multi-dimensional Changes
|
|
Panel Session 851 to be held in Peale Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Chair(s): |
| Melinda Crowley,
United States Department of State,
crowleyml@state.gov
|
| Abstract:
The Bureau of Education and Cultural Affairs (ECA), in the Department of State, has developed innovative, rigorous research designs and strategies to evaluate its international educational programs, which have multiple goals and seek multidimensional changes.
The panel presentations are structured sequentially. They provide, first, an overview of the ECA's educational programs and their multiple goal sets (in the areas of education, civic education, and public diplomacy). The subsequent presentations discuss evaluation methodology development, evolution, and application globally, in the Middle East, North Africa, South Asia, and Eurasia.
The panel ends with presentations of two evaluations: The School Connectivity Program and the English Access Microscholarship Program. Each of these case studies offers insight into the challenges of methodology development, and the implementation of these methodologies in these diverse environments. In each case, evaluation findings have been integral to learning, including on-going program development, adjustment, and effectiveness, globally.
|
|
Department of State International Educational Programs and Multiple Goal Sets
|
| Robin Silver,
United States Department of State,
silverrs@state.gov
|
|
The first panel presentation provides an overview of the educational programs managed by the Bureau of Educational and Cultural Affairs (ECA) within the Department of State. It details the multiple goals sets common to these educational programs:
Educational goals:
1. Educational reforms (curricular, pedagogical, institutional)
2. Transfer of knowledge, skills, or aptitude
3. Integrating access to the Internet with training in new approaches to teaching
Civic education:
1. Civics
2. Community service
Public Diplomacy
1. Mutual understanding and cross-cultural learning
2. Creating linkages between citizen and institutions in the United States and abroad.
Within the context of these goal areas, each program seeks to effect complex, -multi-dimensional- changes in:
- Individual knowledge, attitudes, behaviors, and skills
- Societal/community knowledge and behavior
- Organizations/institutions
Dr. Robin Silver, a Senior Evaluation Officer in the Department of State, has managed evaluations of multiple international educational programs, including the School Connectivity Program evaluation.
|
|
|
Approaches to Learning: Department of State Educational Program Evaluation Methodologies
|
| Norma Fleischman,
United States Department of State,
fleischmanns@state.gov
|
|
This presentation discusses the sophisticated evaluation designs and strategies designed by State Department evaluators in the Bureau of Educational Affairs (ECA).
Evaluations of international educational programs, with multiple goal sets, rely on mixed methodologies and data sources, for example:
1. Surveys of program participants, staff, administrators, educators, etc.
2. Country case studies
3. Focus groups with all program participants
4. Focus groups with teachers, staff, administrators, educators
5. Key informant interviews with program grantees, program staff, administrators, etc.
6. Comparison Groups- surveys of non-participants
7. Focus groups or surveys of possible beneficiaries
8. Informal discussions
9. Program review
10. Review and analysis of program literature
11. Review of professional literature
Dr. Norma Fleischman is a Senior Evaluation Officer in the Department of State and has managed a portfolio of varied public diplomacy program evaluations.
| |
|
School Connectivity Program Evaluation: Methodology and Findings
|
| Jill Tirnauer,
United States Department of State,
tirnauerjx@state.gov
|
|
This presentation presents an overview of the School Connectivity Program, which provides Internet access and computer skills training to schools in 22 countries and supports teacher training in pedagogies that rely on new technologies.
It reviews evaluation design, findings in multiple goal areas, and use in program learning.
The evaluation relied on these primary data sources:
-Open-ended interviews with U.S. based and in-country grantee staff
-Online surveys for students, teachers, and Master Trainers in 12 countries.
-Country case studies in Armenia, Bosnia-Herzegovina, Tajikistan and Lebanon.
-Open-ended interviews with school principals and school program staff.
-Focus groups and informal discussions with program participants (students, teachers, Master Trainers and community members).
-Community members in Tajikistan.
-Comparison schools in Armenia
Ms. Jill Tirnauer is a Senior Evaluation Officer in the Department of State. She is responsible for managing numerous evaluations of its international educational programs.
| |
|
English Access Microscholarship Program: Methodology and Findings
|
| Leslie Esterrich,
United States Department of State,
esterrichlk@state.gov
|
|
This presentation presents an overview of the English Access Microscholarship Program which provides English language instruction to non-elite, Muslim high school students in approximately 44 countries.
It then reviews evaluation design, findings in multiple goal areas, and use in program learning.
The primary data sources were:
-Open-ended Interviews with State Department, Office of English Language Programs Staff
-Site Visits to Schools and Centers in Lebanon, Morocco, Oman, Bangladesh, India, and Pakistan
-Structured Interviews with:
-Public Diplomacy staff
-Regional English Language Officers (RELOs)
-Administrators
-Teachers
-Students
-Alumni
-Peers of Students
-Parents of Students
-Focus Groups Students
-Classroom Observations of Classes
Ms. Leslie Esterrich is a Senior Evaluator in the Department of State. She co-managed the English Access Micro-scholarship Program evaluation, among others.
| |
|
Session Title: Examining Evaluation Training, Practice and Contexts in Various Settings
|
|
Panel Session 852 to be held in Adams Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Teaching of Evaluation TIG
|
| Chair(s): |
| Christina Christie,
Claremont Graduate University,
tina.christie@cgu.edu
|
| Discussant(s):
|
| Christina Christie,
Claremont Graduate University,
tina.christie@cgu.edu
|
| Abstract:
Practicing evaluators hearken from many disciplines of study other than evaluation. These background disciplines encompass content areas of study that reflect the types of programs or policies with which the evaluator primarily works (e.g. public health, education), general social and behavioral sciences (e.g. sociology, psychology) as well as quantitative measurement (e.g. engineering, statistics). While each of these disciplines can provide strong training in research design, methodology, and theoretical approaches relevant to programs or policies under study, they do not provide those who will be evaluators with grounding in the theories, choices of approaches, or types of considerations unique to evaluative study. Panelists from multiple sectors (energy efficiency, public health, and professional evaluation contracting) explore the potential impact of discipline-specific training on evaluation practice and provide insight into how current evaluation training in academia and the professional sector can be structured to improve evaluation practice across disciplines.
|
|
What are we Preparing for? Evaluation Pedagogy Within Master's Level Epidemiology Programs at Schools of Public Health
|
| Leslie Fierro,
Centers for Disease Control and Prevention,
let6@cdc.gov
|
|
Recent findings from projects focusing on improving existing public health capacity have identified a need for training epidemiologists in the discipline of evaluation. In a recent assessment of state and territory health departments conducted by the Council of State and Territorial Epidemiologists, 93% of respondents noted that health department epidemiology staff needed additional training in the area of evaluating public health interventions. Various competency development projects underway may help to address this need by enhancing public health school curricula and trainings in applied epidemiology settings. This presentation provides a description of evaluation pedagogy across 38 schools of public health currently accredited by the Council on Education for Public Health. Findings from a review of over 1,000 courses included in curricula for students pursing a master's level degree in epidemiology will be presented. The relevancy of this coursework to evaluation practice and attaining evaluation competencies as currently proposed will be discussed.
|
|
|
Filling the Training Gap: A Short Course Curriculum for Practicing Evaluators
|
| Nicola Dawkins,
Macro International Inc,
nicola.u.dawkins@orcmacro.com
|
|
Most practicing evaluators have roots in other disciplines. Once entering the professional evaluation world, however, how can individuals 'round out' prior training? Affiliating with AEA or subscribing to journals on evaluation can help, but these resources provide only snapshots of discourse outside of the full context new evaluation practitioners may need to gain the most from the lessons presented. On the other end of the spectrum of comprehensiveness are graduate programs in evaluation-often unfeasible for the practicing evaluator. This presentation will share an evaluation curriculum developed and implemented at one professional consulting firm to help fill the gap. Grounded in literature on essential evaluator competencies, evaluation standards and guiding principles, this series of short courses can hone evaluators' understanding of evaluation theory, questions and designs, methods and considerations for selecting among them, techniques for analysis, determination of evaluative judgments, appropriate reporting of evaluation results, and management of evaluation projects.
| |
|
Understanding the Development of Energy Program Evaluation
|
| Cynthia Austin,
Heschong Mahone Group Inc,
austin@h-m-g.com
|
|
Energy program evaluation has been an integral and consistent piece of the energy efficiency sector for the past twenty-five years. Its premise was to solve the consistent barrier to the adoption and support of energy efficiency by removing the uncertainty about the promised performance of information, practices, and technologies when delivered through actual programs. The existence of a formal documentation of the strengths and weaknesses of programs in peer-reviewed evaluations has been the legitimating, objective evidence that was lacking in the energy program sector prior to the use of program evaluation.
To study the history of energy program evaluation, a group of regarded energy program evaluators will be interviewed to understand the background, influences, and trends seen in the field. The purpose is to document how the energy program evaluation field began and how it has changed since that time by examining the various influences and backgrounds of practitioners.
| |
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Attending to the Learning Needs of Program Stakeholders: Using Planned Variation Strategy in a College Access Program Evaluation |
|
Roundtable Presentation 853 to be held in Jefferson Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
| Presenter(s):
|
| Kathryn Hill,
Minnesota Office of Higher Education,
kathy.hill@state.mn.us
|
| Mary Lou Dresbach,
Minnesota Office of Higher Education,
marylou.dresbach@state.mn.us
|
| Abstract:
This session will discuss the design and use of planned variation in the evaluation of a state GEAR UP grant. A planned variation approach can increase the use of evaluation findings for program improvement and help to build the knowledge base of program components that have a significant effect on student success. Design considerations will be discussed, including the importance of linking the variations to corresponding theoretical ideas about the best way to improve student readiness for college. Also, considerations to ensure the effect of the variation is not confounded with other school characteristics will be addressed.
|
| Roundtable Rotation II:
The Influence of Context on the Collection of Needs Assessment Data |
|
Roundtable Presentation 853 to be held in Jefferson Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
| Presenter(s):
|
| Terry Ann Higa,
University of Hawaii, Manoa,
terryh@hawaii.edu
|
| Aric Nakamura,
University of Hawaii, Manoa,
aricn@hawaii.edu
|
| Abstract:
The panelist will discuss two contexts of collecting needs assessment data about beneficiaries of a program at the University Hawaii at Manoa, College of Education. In the first year of data collection, the program beneficiaries (students) were taking courses at the College. Data were collected through student focus groups, student interviews, college faculty questionnaires, and program staff questionnaires. The evaluators were able to identify need areas for the students and ways that the program may address those areas. In the subsequent year, the students were assigned to teach in schools. Various factors limited the evaluators' efforts to collect data, including school security rules, teacher union regulations, and program limitations. Eventually, the evaluators were able to collect needs assessment data from mentors that met with the students in the schools, at program workshops, and through an on-line questionnaire. The panelist will discuss the varying degrees of success of those methods.
|
|
Session Title: Evaluation Lessons in International Contexts: Health Services and Infectious Diseases
|
|
Multipaper Session 854 to be held in Washington Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Health Evaluation TIG
|
| Chair(s): |
| Monica Ruiz-Casares,
McGill University,
monica.ruizcasares@mail.mcgill.ca
|
|
The Tradeoff Between Centralized and Decentralized Health Services: Evidence From a Major Anti-Poverty Program in Mexico
|
| Presenter(s):
|
| Arturo Vargas-Bustamante,
University of California, Berkeley,
avargas@berkeley.edu
|
| Abstract:
This paper presents initial evidence on the effectiveness of two models of health services organization. It compares performance between centralized and decentralized services by taking advantage of health care provider duplication in rural Mexico. In contrast with the most common predictions in the literature and the recommendations of a wide array of international policy makers, the centralized provider of health services performs better. Households served by the centralized organization face less regressive out-of-pocket health care expenditures (45% lower), and observe higher utilization of preventive services (15% more). Fortunately, state providers improve significantly in those areas targeted by Oportunidades, the main anti-poverty policy in Mexico. The outcomes of centralized and decentralized organizations turn indistinguishable in those areas where Oportunidades operates. These findings are robust to the possible effect of time and type of decentralization, state and health infrastructure effects, Oportunidades' early treatment, the use of alternative measures and other confounders.
|
|
Using the Extended Parallel Processing Model to Evaluate the Impact of Avian and Pandemic Flu Communication Programs: Lessons From Egypt's National Communication Campaign 2006
|
| Presenter(s):
|
| Douglas Storey,
Johns Hopkins University,
dstorey@jhuccp.org
|
| Abstract:
This paper describes the impact evaluation of a 2006 avian flu communication campaign in Egypt. The campaign, which launched within hours of the first confirmed case of bird flu, was designed to manage the initial public response to an emerging avian outbreak and to minimize bird to human transmission of the H5N1 virus. Campaign messages and evaluation instrumentation were based on the Extended Parallel Processing Model (EPPM), which describes the interacting influence of emotional (threat) and rational (efficacy) perceptions on behavioral decisions. Measures of both individual and collective threat and efficacy were employed. Impact data were obtained from a nationally representative survey (n=4052) of 15-49 year old adults in 21 governorates. Results indicate positive effects of message recall on threat and efficacy perceptions and on the number of protective behaviors initiated following the launch of the campaign.
|
|
Evaluation of HIV/AIDS Awareness Programs: How Well African American Religious Community Do?
|
| Presenter(s):
|
| Julien Kouame,
Western Michigan University,
julienkb@hotmail.com
|
| Fatma Ayyad,
Western Michigan University,
fattmah@hotmail.com
|
| Abstract:
This evaluation examines the efficiency and effectiveness of a HIV/AIDS awareness programs in African American Religious community (AARC), in a USA's small town. Qualitative data collection methods including, interviews, observations and focus group were used to gather data from the community's churches. Church leaders and church goers provided information regarding their churches' activities for HIV/AID awareness.
Data collected were analyzed qualitatively with qualitative data analysis software to answer the following evaluation questions: (1) Is the AARC's involved in HIV/AIDW awareness programs. (2) What are the obstacles preventing them (if not involved) to be involved? (3) Are the existence programs (if any) relevant for HIV/AID awareness and education? This paper presents potential gaps between community need and the church's activities, and our expectations for strengthening programs intended to HIV/AID awareness and Education in the AARC. As a result of this study, potential barriers to church involvement were discovered.
|
|
Health Workers' Performance Evaluation: A Better Approach for Developing Countries
|
| Presenter(s):
|
| Julien Kouame,
Western Michigan University,
julienkb@hotmail.com
|
| Abstract:
To observe health workers' (HW) performance over time after training, we assessed a sample of consultations of ill children 2–59 months of age with non-severe pneumonia. Data include information gathered from patient registers, interviews and observations. Our analysis included three indicators of case management quality: correct assessment, classification, and treatment. We assumed assessment was correct when a child with an acute respiratory infection had all ARI-related assessments done; classification was correct when a child with no severe classification who had IMCI-defined non-severe pneumonia was classified as having non-severe pneumonia by the HW; and treatment was correct when the HW prescribed a 7-day course of cotrimoxazole with a dosage appropriate for the child's weight. The study showed a short-term positive impact of the IMCI follow-up visit (i.e., when an IMCI trainer visits the HW shortly after IMCI training) on HW performance.
|
| | | |
|
Session Title: Does Quality Improvement Lead to Organizational Learning?
|
|
Think Tank Session 855 to be held in D'Alesandro Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Presenter(s):
|
| Stanley Capela,
HeartShare Human Services,
stan.capela@heartshare.org
|
| Rashi Rohatgi,
Harlem United Community AIDS Center,
rrohatgi@harlemunited.org
|
| Discussant(s):
|
| Margaretta Fox,
Harlem United Community AIDS Center,
mfox@harlemunited.org
|
| Heather N Chapman,
Chesapeake Center for Youth Development Inc,
heatherc@ccyd.org
|
| Charles Gasper,
Missouri Foundation for Health,
cgasper@mffh.org
|
| Candi Chitty,
Quality First Healthcare Consulting Inc,
qualityfirstcjc@cs.com
|
| Melanie Hwalek,
Social Program Evaluators and Consultants Inc,
mhwalek@specassociates.org
|
| Abstract:
Over the years, the paradigm of evaluation has shifted from quality assurance (i.e., assessing performance) to quality improvement (i.e., assessing and systematically attempting to change performance). In some instances, quality improvement initiatives are perceived as helpful tools for guiding an organization down the path of improvement and, even, guiding strategic redirection. At other times, however, quality improvement initiatives have been seen as an albatross, failing to accomplish stated improvement goals and doing little to promote organizational learning.
The purpose of this think tank is to provide a forum to address these two opposing views of quality improvement, review case studies of various quality improvement projects from around the country, identify key characteristics of successful quality improvement initiatives, and provide participants with a template for quality improvement initiatives that promote organizational learning.
|
|
Session Title: Cross-validation of Empirical Findings From Evaluations
|
|
Panel Session 856 to be held in Calhoun Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Barbara Brumbach,
Northern Arizona University,
barb.brumbach@gmail.com
|
| Abstract:
Data analyses are frequently insufficiently guided by theory so that ultimate decisions and interpretations are subject to the phenomenon of "capitalizing on chance." That occurs because, inescapably, data are subject to error. Aside from theory, the primary check on the possibility of capitalizing on chance is cross-validation. Remarkably little, beyond statistical procedures, in the way of specification for the methodology of cross-validation has been written. Cross-validation is not so simple since even the criteria are for deciding whether a finding has been cross-validated have not been specified. The common method of cross-validation based on splitting a large sample into halves is useful only for narrow purposes. Better methods need to be developed. One possibility is the method we call "identifying quasi-populations." The problems involved in cross-validation will be discussed and illustrated and potential improvements in methods will be demonstrated in relation to cross-validation of parameter estimates and of models.
|
|
Rationale for Cross-validation
|
| Lee Sechrest,
University of Arizona,
sechrest@u.arizona.edu
|
|
Cross-validation is an attempt to estimate (or demonstrate) the dependability of research findings by showing that they are replicable. There is no general agreement on just what aspects of a research effort are required to be dependable in order for cross-validation to be considered a success. Cross-validation depends on the assumption that errors are independent across sets of observations involved in the cross-validation. Theory and empirical data may be said to involve such independence and, hence, cross-validation if they are truly independently derived. In the absence of strong theory, conclusions drawn solely from a single set of observations are suspect. A frequent method of cross-validating is to split a large sample and show that findings from one half are replicated in the other half. That method of cross-validation has very limited usefulness. Replicability of findings in independent samples is by far the preferable method.
|
|
|
The Quasi-population Approach as a Tool for Cross-validation
|
| Mei-kuang Chen,
University of Arizona,
kuang@email.arizona.edu
|
|
Replicabilitiy is the hallmark of science and the focus of cross-validation efforts, but replicability is not often undertaken in the social and behavioral sciences. Obstacles to replication are great. If one considers how replication is actually carried out when it does occur, one will find that many large data sets include incidental variables that permit partitioning of the data set into two or more subsamples that may be said to represent "quasi-populations," i.e., subsamples that emulate differences between samples that might be found if different investigators were undertaking the work. Socio-demographic variables, e.g., sex, age, place of residence, marital status, religious affiliation, may be used to partition the original sample. As long as the partitioning is not along lines that would be confounded with the variables of theoretical interest, two or more analyses could be considered to constitute a cross-validation design. Examples of such data sets are informative.
| |
|
Cross-validation of Parameter Estimates
|
| Michele Walsh,
University of Arizona,
mwalsh@u.arizona.edu
|
|
Probably the most common cross-validation effort involves assessing the dependability of parameter estimates. Parameters most commonly of interest with respect to cross-validation are means and measures of association (correlation). Cross-validation by split-sample methods can result in high, but misleading estimates of dependability because the random (error) component in one half of a random sample should be virtually the same as in the other half of the sample. Thus, what can be determined by split-sample cross-validation are simply the magnitudes of standard errors. That can easily be illustrated with a variety of data sets. An alternative, more demanding but more telling, approach to cross-validation is to try to define quasi-populations within the data set, i.e., subsamples that, in theory should not differ in ways related to the parameter(s) of interest and then determine the closeness of estimates obtained from the quasi-populations. Again, this process is easily demonstrated with actual data sets.
| |
|
Cross-validation of Models
|
| Barbara Brumbach,
Northern Arizona University,
barb.brumbach@gmail.com
|
|
Despite the widespread popularity of the idea of "confirmatory" analyses, most published accounts of multivariate models will be found to have been "modified" in some ways based on empirical data. That makes such models at least somewhat suspect of the possibility of "capitalizing on chance." Hence, even though it does not happen often, multivariate models should be cross-validated. It is not easy to specify just what would constitute cross-validation, e.g., the general structure of the overall model, the proportion of variance in the dependent variable that is accounted for, the robustness of specified paths, or specific path coefficients. When it is possible to do so, quasi-population approaches to cross-validation are of great interest. The use of such approaches is illustrated with large data sets.
| |
|
Session Title: Multicultural Evaluation: Issues in Health and Education Settings
|
|
Multipaper Session 857 to be held in McKeldon Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Multiethnic Issues in Evaluation TIG
|
| Chair(s): |
| Kien Lee,
Association for the Study and Development of Community,
kien@capablecommunity.com
|
| Discussant(s): |
| Imelda Castańeda-Emenaker,
University of Cincinnati,
castania@ucmail.uc.edu
|
|
How Culture Influences Program Impact: Evaluating a HIV/AIDS Prevention Program in Tanzania
|
| Presenter(s):
|
| Dominica McBride,
Arizona State University,
dominica_34@hotmail.com
|
| Abstract:
Culture is an integral and dynamic force in a program, having inevitable influence on the program process and outcomes. This force can help and/or hinder program impact in various ways. This paper describes an evaluation implemented on a HIV/AIDS prevention program in rural Tanzania. The evaluation of Students for International Change (SIC) was conducted over the period of a year. This evaluation employed a mixed methods approach, using baseline and follow-up surveys, as well as focus groups, sampling both students and adults in the target community. SIC is a non-governmental organization created to reduce the affects of HIV/AIDS on the people of northern Tanzania. This paper delineates the findings and how the culture of this particular community influenced the impact of the program. The results from the evaluation provide insight not only into the local culture and the interaction of local culture and program features, but also for future evaluations.
|
|
Evaluating Systems, Delivery, and Organizational Structure of a Health Care Delivery Program in India
|
| Presenter(s):
|
| Vamsi Vasireddy,
University of Illinois, Chicago,
vvasir2@uic.edu
|
| Shankar Rao Vasireddy,
National Leprosy Eradication Programme, India,
vasireddy77@gmail.com
|
| Abstract:
Government of India started the National Leprosy Eradication Programme (NLEP), with the aid of W.H.O. and other international funding agencies, to eradicate Leprosy in India by year 2000. NLEP was only partially successful. The goal of this evaluation project is to identify challenges to data collection and evaluate the program using qualitative research methods. Ethnographic interviews and focus groups were conducted with medical officers and auxiliary staff involved in the program to identify factors contributing to failure. Qualitative analysis software (Atlas.ti®) was used to identify major themes and associate them with the delivery system and organizational structure. Thematic mapping was done to evaluate various components of the delivery system. Network analysis was performed to evaluate the organizational structure. This process enabled us to identify areas that were not identified using quantitative methods. The results of evaluation were used to develop logic models, strategic plans, and build systems capacity.
|
| |
|
Session Title: Using Appreciative Inquiry to Facilitate Change in Organizational Practice
|
|
Skill-Building Workshop 858 to be held in Preston Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Organizational Learning and Evaluation Capacity Building TIG
|
| Presenter(s):
|
| Elaine Van Melle,
Queen's University,
vanmelle@post.queensu.ca
|
| Sheila Pinchin,
Queen's University,
pinchins@post.queensu.ca
|
| Abstract:
Traditionally change mechanisms involve evaluation of problems or gaps in a system in order to rectify them. Rather than emphasizing problems, Appreciative Inquiry, grounded in social constructivism, evaluates what is working in an organization to capitalize and build on those strengths. The ensuing dialogue allows space for individuals to evaluate success in light of their own affirmed experience and provides a basis for others to benefit.
Participants in this workshop will by introduced to the principles of evaluation using Appreciative Inquiry. An interactive narrative of the Appreciative Inquiry journey of the presenters will illustrate how Appreciative Inquiry techniques and principles were implemented to evaluate a successful teaching program in medical education, as a strategy to effect organizational change and encourage innovation. Using individual reflection and group activities, participants in this session will have the opportunity to apply the principles of Appreciative Inquiry to their own context and initiatives.
|
|
Session Title: Using Baseline Assessment to Build Evaluation Capacity and Foster Organizational Learning as Part of a Transformative Change Process
|
|
Demonstration Session 859 to be held in Schaefer Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Organizational Learning and Evaluation Capacity Building TIG
|
| Presenter(s): |
| Srik Gopalakrishnan,
The Ball Foundation,
srik@ballfoundation.org
|
| Michael Palmisano,
The Ball Foundation,
mpalmisano@ballfoundation.org
|
| Rex Babiera,
The Ball Foundation,
rbabiera@ballfoundation.org
|
| Abstract:
The Ball Foundation works in partnership with school districts to increase literacy achievement for all students by improving the professional practice of educators. While the end-goal of this effort is literacy for all students, the foundation primarily sees itself as building the organizational capacity of the school system to facilitate transformative change. The first year in a five-year long partnership is an `inquiry and engagement' phase leading up to a partnership work plan. A key facet of the inquiry and engagement phase is baseline assessment of the school district's organizational capacity. While the primary purpose of this assessment is to inform partnership planning and lay out a baseline 'reference' point, the foundation believes that it's more than just a data-collection exercise - it's the beginning of the overall transformative change process. Hence, the foundation views baseline assessment as an intentional means of building evaluation capacity and fostering organizational learning.
|
|
Session Title: Federal, State, and Local Disaster and Emergency Planning and Preparedness: Strategies and Case Studies for Evaluation
|
|
Multipaper Session 860 to be held in Calvert Ballroom Salon B on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Disaster and Emergency Management Evaluation TIG
|
| Chair(s): |
| Denise Bulling,
University of Nebraska,
dbulling@nebraska.edu
|
| Discussant(s): |
| Mary Davis,
University of North Carolina, Chapel Hill,
mvdavis@email.unc.edu
|
|
Public Health Readiness: Lessons Learned From Using a Community-Based, Participatory Approach to Evaluate Local Public Health Preparedness Planning in Massachusetts
|
| Presenter(s):
|
| Bryan Hall,
Institute for Community Health,
brhall@challiance.org
|
| Justeen Hyde,
Institute for Community Health,
jhyde@challiance.org
|
| Abstract:
Community-based participatory evaluation strives to engage community partners in the development and implementation of studies, from the formulation of questions and methodologies to the interpretation and dissemination of results. In Massachusetts, this approach is being used to evaluate public health-based emergency preparedness planning activities directed by the Cambridge, MA Public Health Department, Advanced Practice Center in a 27-community emergency preparedness region surrounding Boston.
This paper will focus on the strategies developed to utilize a participatory approach in a newly emerging field of practice with shifting expectations and responsibilities. In addition to an overview of the process and outcome measures developed for the evaluation, strategies for engaging and eliciting the input from community partners with limited time and resources will also be discussed. The lessons learned from this unique collaborative venture will provide helpful information for other evaluators interested in evaluation of emergency preparedness planning and other local public health activities.
|
|
Homeland Security: The Federal, State, and Local Conundrum
|
| Presenter(s):
|
| Christopher Cihlar,
DFI Government Services,
ccihlar@dfi-intl.com
|
| Sean McLaughlin,
DFI Government Services,
smclaughlin@dfi-intl.com
|
| Abstract:
Federal, state and local homeland security authorities have a shared responsibility in preparing for terrorist attacks and natural disasters. However, these stakeholders often hold different and sometimes conflicting views about what it means to be "prepared.” While the federal government has in place a national strategy and seeks data from local jurisdictions to demonstrate progress in achieving national goals, this data is, at a local level, generally insufficient to demonstrate local preparedness improvements. This paper begins with an analysis of the dichotomy between federal and state and local preparedness efforts. It then presents a series of recommended evaluation strategies that jurisdictions can use to ensure they both fulfill crucial federal data requirements and are able to appropriately meet their own critical need to measure local preparedness improvements.
|
|
Planning for Pandemic Influenza: Evaluation of a National Public Engagement Process
|
| Presenter(s):
|
| Denise Bulling,
University of Nebraska,
dbulling@nebraska.edu
|
| Mark DeKraai,
University of Nebraska,
mdekraai@nebraska.edu
|
| Abstract:
A series of facilitated public forums were held across the United States to get the public's input regarding vaccination prioritization in the event of a pandemic. A mixed methods evaluation framework was employed to assess the impact of forums with participants. Additionally, the evaluation included a component to assess the impact the entire series had on the ultimate decisions of policy makers regarding vaccination priorities for the United States. This presentation highlights the results of the evaluation and the benefits of evaluation to disaster planners.
|
|
Responder Alert Prophylaxis Initiative Drill (Operation RAPID): Developing a Comprehensive Evaluation Strategy for a Multi-site Evaluation
|
| Presenter(s):
|
| Elizabeth Ablah,
University of Kansas School of Medicine,
eablah@kumc.edu
|
| Kurt Konda,
University of Kansas School of Medicine,
kkonda@kumc.edu
|
| Eileen Scanlon,
Nassau County Department of Health,
eileen.scanlon@hhsnassaucountyny.us
|
| Leslie Horn,
Columbia University,
lah2110@columbia.edu
|
| Kristine Gebbie,
Columbia University,
kmg24@columbia.edu
|
| Abstract:
Operation RAPID, an exercise simulating the airborne release of anthrax requiring the mass prophylaxis of first responders throughout Nassau County (Long Island), necessitated a multi-faceted exercise evaluation utilizing a pool of 100 volunteer evaluators to observe 25 separate sites, differentiated by geography and function, including variable POD types, central distribution, and emergency operations. The exercise’s five competency-based objectives were utilized in the development of standardized tools, including checklists of observable actions and participant and staff surveys based on the job action sheets for the exercise. Though the tools necessarily differed based on the specific tasks performed by each functional area of the exercise, a uniform evaluation was made possible by focusing on the tasks of each functional area as related to the overall exercise objectives and by providing a uniform training in the week leading up to the exercise for all evaluators, regardless of the exercise portion for which they were responsible. Operation RAPID was the first test of new mass dispensing plans, so evaluation was critical to analyzing the effectiveness of these plans that may be relied upon to protect the population of the county in a public health emergency.
|
| | | |
|
Session Title: Perspectives on Federal Evaluation
|
|
Multipaper Session 862 to be held in Calvert Ballroom Salon E on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Government Evaluation TIG
|
| Chair(s): |
| Kent Hutchinson,
Florida State Legislature,
hutchinson.kent@mail.oppaga.state.fl.us
|
|
Measuring the Effectiveness of Safety Training in the Mining Industry: A Mixed-methods Evaluation Report
|
| Presenter(s):
|
| Lori Matthews,
Spokane Research Laboratory,
lmatthews@cdc.gov
|
| Abstract:
Ten safety training videos were produced as part of a 1998 project to develop and evaluate effective training for the mining industry by the Spokane Research Laboratory (SRL), part of the National Institute for Occupational Safety and Health (NIOSH). This paper presents the mixed-methods evaluation results of the two most recent NIOSH/SRL training videos: Aggregate Training for the Safety Impaired (2003) and The Sky is Falling! (2004). Quantitative measures indicated that the videos were effective in educating and entertaining miners, as well as providing good reminders to work safely. The qualitative data illustrated the transferable nature of the evaluation results; safety trainers from the mining industry reported using surface videos for training in underground operations. In addition, other occupational industries including fire-fighting, construction, and logging have reported using the mining videos to communicate and promote the common goal of working safely.
|
|
Striking a Balance: Guiding Local Evaluations From a Federal Program Perspective
|
| Presenter(s):
|
| Melanie Besculides,
Mathematica Policy Research Inc,
mbesculides@mathematica-mpr.com
|
| Charlotte Hanson,
Mathematica Policy Research Inc,
chanson@mathematica-mpr.com
|
| Heather Zaveri,
Mathematica Policy Research Inc,
hzaveri@mathematica-mpr.com
|
| Ryan Loo,
Centers for Disease Control and Prevention,
rloo@cdc.gov
|
| Lisa Levy,
Centers for Disease Control and Prevention,
lrl2@cdc.gov
|
| Abstract:
Funding agencies recognize the importance of program evaluation and require their grantees to conduct evaluations. However, evaluation guidance typically lacks enough detail, which can be detrimental, as evaluation expertise can vary greatly among grantees. Conversely, too much guidance can tie the hands of evaluators and inhibit the development of appropriate evaluations for a given setting, context, and purpose. We will present a 'how to' paper with our approach to achieving the appropriate balance between these issues, using the Well-Integrated Screening and Evaluation Program for Women Across the Nation (WISEWOMAN) federal program's evaluation guidance and evaluation plan as examples. The developmental process we will present can be used by a multitude of other grantees and funding agencies. As a grantee, participants will apply the lessons learned by facilitating change in their funding agency's evaluation policy. As a funding agency, participants will apply the lessons learned by improving evaluation guidance and policy.
|
|
Conducting a Utilization-focused Evaluation in Government Contract Work: Perspectives From the Contractor and the Government Client
|
| Presenter(s):
|
| Sarah Shoemaker,
Abt Associates Inc,
sarah_shoemaker@abtassoc.com
|
| Claire Kendrick,
Agency for Healthcare Research and Quality,
claire.kendrick@ahrq.hhs.gov
|
| Meg Hargreaves,
Abt Associates Inc,
meg_hargreaves@abtassoc.com
|
| Rachel Sayko Adams,
Abt Associates Inc,
rachel_adams@abtassoc.com
|
| Tricia Trinite,
Agency for Healthcare Research and Quality,
tricia.trinite@ahrq.hhs.gov
|
| Janice L Genevro,
MasiMax Resources Inc,
janice.genevro@ahrq.hhs.gov
|
| Harmon Jordan,
Abt Associates Inc,
harmon_jordan@abtassoc.com
|
| Abstract:
Abt Associates Inc. was contracted to evaluate a government agency's research Portfolio. Abt submitted a proposal that embodied a utilization-focused evaluation (UFE) framework, which differs from traditional evaluation in that it requires clarifying the intended uses of the evaluation and being particularly responsive to the client's current evaluation needs. Abt worked collaboratively with the client during the initial months of the project to clarify and prioritize the study objectives. The evaluation evolved from a summative evaluation to a formative, developmental evaluation to understand better what Portfolio-related work has been conducted and to identify gaps in the research area. This presentation will summarize the evaluation methods and techniques; the UFE process; and the successes and challenges for both the contract research firm and the government client.
|
| | |
|
Session Title: Math and Science Evaluation
|
|
Multipaper Session 863 to be held in Fairmont Suite on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Anane Olatunji,
George Washington University,
dr_o@gwu.edu
|
|
Do You Really Know What You're Using?
A Comparison of Two Widely Used Science and Math Observation Instruments
|
| Presenter(s):
|
| Martha Henry,
M A Henry Consulting LLC,
mahenry@mahenryconsulting.com
|
| Keith Murray,
M A Henry Consulting LLC,
keithsmurray@mahenryconsulting.com
|
| Abstract:
Observation of classroom lessons is an important if problematic component of educational evaluation. Where programs aim at teacher professional development, student performance enhancement, curricular change, or implementation of reform teaching methods, direct observation is crucial to validating and explaining results. However, the challenges of funding and maintaining uniformly trained observers and optimally timing observations have led some evaluators to avoid this evaluative technique. Of greatest importance is the selection of an observational instrument that captures information across domains of interest with sufficient validity and reliability. This presentation offers an analysis and comparison of the two most often used observational instruments in math and science K-12 classrooms. These instruments – Inside the Classroom Observation and Analytic Protocol and the Reform Teaching Observation Protocol – evidence not only similarities but marked differences in design, focus, documentation and observer skill requirements. The authors offer recommendations for potential users to enhance usefulness and avoid pitfalls.
|
|
The Impact of Benchmark Assessments Aligned to State Standards on Student Achievement
|
| Presenter(s):
|
| Susan Henderson,
WestEd,
shender@wested.org
|
| Sarah Guckenburg,
Learning Innovations at WestEd,
sgucken@wested.org
|
| Anthony Petrosino,
WestEd,
apetros@wested.org
|
| Steve Hamilton,
WestEd,
|
| Abstract:
This study examines whether districts using quarterly benchmark exams in middle school mathematics show greater gains in student achievement than those not employing this practice. The study will examine differences in student achievement as measured by the Massachusetts State Comprehensive Assessment (MCAS) in schools using quarterly benchmark assessments aligned with Massachusetts Curriculum Frameworks Standards for mathematics in grades 6 and 8. This study offers practical insight to evaluators in conducting scientifically based evaluations of educational programs through a quasi-experimental design that does not require randomization to treatment group prior to the implementation of an educational innovation. The results of this study have the potential to provide a solid research base to inform district and school level practices in the use of formative and benchmark assessment to increase student achievement on state education standards.
|
|
Layered Learning: Evaluating a Constructivist Mathematics Approach in the Accountability Era
|
| Presenter(s):
|
| Jennifer Coyne Cassata,
Prince William County Public Schools,
cassatjc@pwcs.edu
|
| Abstract:
Program evaluations conducted in schools involve multiple layers of learning. The proposed paper will describe an evaluation of a constructivist approach to mathematics adopted by a relatively large school district. Internal program evaluators were asked to conduct an evaluation of the program's implementation and impact, beginning with the first phase of implementation (grades K-2). From the outset, the evaluation process generated learning about some teachers' and administrators' reservations about this mathematics approach, as well as about concerns over appropriate ways to measure student learning. The paper will discuss the learning that occurred during the evaluation process, including: 1) learning by evaluators about the program and about the district's evaluation capacity, 2) learning by stakeholders about the program, the evaluation process, and the nature of teaching and learning, and 3) student learning. The paper will also consider the fear of learning from evaluation that can occur in the atmosphere of accountability.
|
| | |
|
Session Title: Evaluating After School, Supplemental Services, and Out of School Programs
|
|
Multipaper Session 864 to be held in Federal Hill Suite on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Alison Williams,
Clark County School District,
alisonw@interact.ccsd.net
|
|
Assessing the Effectiveness of Online Supplemental Education Services
|
| Presenter(s):
|
| Joel Shapiro,
Rockman Et Al,
joel@rockman.com
|
| Abstract:
This paper examines the effectiveness of online supplemental education services (SES) and factors that can encourage and inhibit its success. NCLB requires that underperforming districts provide SES, typically via after-school tutoring by external vendors. Face-to-face SES suffers from low participation rates, difficulty in providing instruction to students in remote geographic areas, and other problems. SES that is delivered through remote technologies has the potential to remedy some of these problems, due largely to students' ability to participate from home. This study uses a randomized design of approximately 300 middle school students in six schools in three states to isolate the effect of a large SES provider's math program on students' achievement, motivation, and attitudes towards school and learning. Finally, the study reports on the results of school site visits to identify those district, school and classroom teacher-level factors that can encourage and/or inhibit the success of online SES provision.
|
|
Considering Dosage in After-School Programs: Linking Activity Types to Outcomes
|
| Presenter(s):
|
| Laurie Van Egeren,
Michigan State University,
vanegere@msu.edu
|
| Celeste Sturdevant Reed,
Michigan State University,
csreed@msu.edu
|
| Heng-Chieh Wu,
Michigan State University,
wuhengch@msu.edu
|
| Nai-Kuan Yang,
Michigan State University,
yangnaik@msu.edu
|
| Abstract:
Despite advances in conceptualizations of participation in after-school programs, measures of dosage often remain global and difficult to link to specific outcomes. Using data from the state evaluation of the Michigan 21st Century Community Learning Centers, we demonstrate methods for: (a) examining links between students' dosage of specific types of activities and hypothesized outcomes; (b) characterizing dosage in multiple ways that have differential relationships with outcomes; and (c) controlling for differences in students' dosage that result from constraints in what programs offer and policies about activity choice. Multi-level analyses of 187 sites serving over 25,000 students will be used to demonstrate ways to distinguish between students' individual dosage (collected through a web-based tracking system) and site-level factors that affect dose-response rates in student-, parent-, and teacher-reported outcomes and student achievement.
|
|
Building Evaluation Capacity in 21st Century Community Learning Center After-school Programs
|
| Presenter(s):
|
| Tara Donahue,
Learning Point Associates,
tara.donahue@learningpt.org
|
| Abstract:
Quality evaluations are critical for after school programs to gauge the effectiveness of programs and to help programs engage in the process of continuous program improvement. This paper examines how a 21st Century Community Learning Center (21st CCLC) after school program in northern Michigan worked with its local evaluator through both utilization and empowerment evaluation frameworks to build evaluation capacity within the program. This paper offers lessons learned about how to effectively integrate multiple methodologies including quantitative analysis through surveys and outcome measures, qualitative analysis through interviews and document review, and self-assessment analysis through the use of a validated instrument in after school evaluations. This evaluation informed the program staff and helped them make decisions for continuous program improvement leading to positive outcomes. Additionally, the evaluation work done in this program helped the evaluator gain insight into how to apply these concepts, strategies, and tools to the larger field of after school evaluation.
|
|
Keeping Youth Engaged in Out-of-school Time Programs: Results From a Five-year Evaluation in Denver Public Schools.
|
| Presenter(s):
|
| Robin Leake,
JVA Consulting LLC,
robin@jvaconsulting.com
|
| Veronica Gardner,
JVA Consulting LLC,
v@veronicagardner.com
|
| Sheridan Green,
JVA Consulting LLC,
sheridan@jvaconsulting.com
|
| Abstract:
This is the fifth year of an evaluation of the Denver Public Schools out-of-school (OST) program to examine whether participation in OST improves students' academic achievement, school-day attendance and behavior. The purpose of this evaluation is to investigate the impact of attendance, intensity, duration and breadth of attendance in OST programming on student outcomes compared to a control group of students from the same school who did not participate in OST activities. Student achievement was measured using the state-mandated Colorado Student Assessment Program reading, writing and math test scores. Results across eleven elementary and middle school sites demonstrated that involvement in OST activities for three or more years improves students' school-day attendance and achievement scores in reading and math compared to the control group students. A formative evaluation component was added in Year-5 to identify programmatic characteristics associated with greater retention of students, particularly from elementary to middle school.
|
| | | |
|
Session Title: Centralized E-Tool for Organizational Performance Management: National Institutes of Health (NIH) Government Performance Result Acts (GPRA) & Program Assessment Rating Tool (PART) Assessments
|
|
Expert Lecture Session 865 to be held in Royale Board Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Deborah Duran,
National Institutes of Health,
durand@od.nih.gov
|
| Presenter(s): |
| Deborah Duran,
National Institutes of Health,
durand@od.nih.gov
|
| Abstract:
In the current environment of accountability, organizations are expected to practice Budget-Performance Integration (BPI) that enables transparent productivity associated with funding. Organizations are assessed and performance ratings are presented to the public to demonstrate transparency. To address the BPI reporting requirements, including GPRA and PART, the National Institutes of Health (NIH), the nation's primary biomedical research agency, developed a centralized online e-tool, called Program Performance Monitoring System (PPMS). This management tool provides automated technology for planning, collecting, monitoring, assessing and reporting performance goals, annual targets, and associated budgets, while making data available to decision-makers in a timely manner. The expert lecture will outline components of PPMS, provide insight into agency implementation and adoption, and discuss how the single point access application enhances communication and management to foster research planning. A visual graphics and Strategic Planning initiative, the next generation of development, moves PPMS towards a strategic level tool.
|
|
Session Title: Substance Abuse Programming: Assessing Fidelity of Implementation and Community-readiness
|
|
Multipaper Session 866 to be held in Royale Conference Foyer on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG
|
| Chair(s): |
| Rena A. Agee,
Macro International Inc,
rena.a.agee@orcmacro.com
|
|
Analysis of Strategic Plans to Assess Fidelity to a Model-based Planning Process
|
| Presenter(s):
|
| Melissa Gutierrez Barrett,
Westat,
melissagutierrez@westat.com
|
| Ann Landy,
Westat,
annlandy@westat.com
|
| Robert Orwin,
Westat,
robertorwin@westat.com
|
| Abstract:
In this presentation, we describe the analysis of 26 strategic plans to assess fidelity to SAMHSA's Strategic Prevention Framework (SPF). The SPF is a five-step model that requires grantees to engage in data-driven planning. Specifically, grantees must demonstrate that needs assessment results are linked to selection of substance use-related priorities and community funding allocation plans. Three-point scales were used to assess the strength of these linkages. The assessment procedure involved four-steps: 1) plans were reviewed by multiple raters, 2) descriptive information was extracted for 10 topic areas, 3) thematic codes were defined and scale anchors were constructed, and 4) fidelity scores were assigned using a consensus process. The scores will be used to predict outcomes. The development of this procedure serves not only the SPF SIG cross-site evaluation but is potentially applicable for evaluating prevention initiatives of the future.
|
|
Assessing School “Community Readiness” to Take Action: Comparison of Structured Versus Semi-structured Interview Methods
|
| Presenter(s):
|
| Kristen Ogilvie,
Pacific Institute for Research and Evaluation,
kogilvie@pire.org
|
| Knowlton Johnson,
Pacific Institute for Research and Evaluation,
kwjohnson@pire.org
|
| Abstract:
Community readiness is increasingly recognized as important in achieving substance abuse prevention program success. The Community Readiness Model (CRM), an established and respected model developed at Colorado State, involves a lengthy ethnographic in-person or telephone interview for assessment. Copious notes or transcripts are laboriously coded to score readiness. While this method provides invaluable information on both readiness and community dynamics, it is time-consuming, especially when involving multiple communities. As an alternative, this paper describes the conversion of the respected CRM assessment instrument to a structured telephone survey for the purpose of assessing the readiness of 15 rural Alaskan school districts to participate in a school-based prevention research study to address youth substance abuse sponsored by the National Institute on Drug Abuse. Drawing on the researchers' experience, the paper explores the benefits and liabilities of the different interview methods in assessing readiness and their implications for program evaluation.
|
|
Evaluating the Fidelity of Implementation of Research-based Substance Abuse/School Crime Prevention Programs in Schools
|
| Presenter(s):
|
| Scott Crosse,
Westat,
scottcrosse@westat.com
|
| Michele Harmon,
Westat,
micheleharmon@westat.com
|
| Barbara Williams,
Westat,
barbarawilliams@westat.com
|
| Carol Hagen,
Westat,
carolhagen@westat.com
|
| Liam Ristow,
Westat,
liamristow@westat.com
|
| Abstract:
To increase the effectiveness of school-based prevention programs, funding sources have embraced the idea that such programs should be research-based and be implemented in ways that maximize the likelihood of their producing desired outcomes. However, very little is known about either the prevalence of such programs or the quality of their implementation. This paper describes how the Study of Research-based Programs developed national estimates of the fidelity of implementation of research-based prevention programs in schools. Doing so required creating fidelity criteria that were specific to each of several research-based programs, by reviewing implementation materials for those programs. For a given program, the Study compared the fidelity criteria against survey data gathered on corresponding measures of program implementation. In addition, the paper reports findings on fidelity, as well as on factors associated with it, such as training on program implementation and monitoring of program providers.
|
|
26 States, 300 Communities, and One Fidelity Instrument: Fabulous Feat or Futile Folly in the Strategic Prevention Framework Cross-site
|
| Presenter(s):
|
| Robert Orwin,
Westat,
robertorwin@westat.com
|
| Roy Gabriel,
RMC Research Corporation,
rgabriel@rmccorp.com
|
| Ann Landy,
Westat,
annlandy@westat.com
|
| Janis Wolford,
Westat,
janiswolford@westat.com
|
| Abstract:
All Strategic Prevention Framework (SPF) states are required to implement the 5-step framework: 1) Needs/Readiness Assessment, 2) Capacity Building, 3) Strategic Planning, 4) Intervention Implementation, and 5) Monitoring/Evaluation. The communities each state funds are also required to implement the steps. Recent work has highlighted the strong relationship between implementation fidelity and outcomes in community prevention programming. Will that relationship hold when the program is a “model” and the model will be implemented differently in up to 300 communities across 26 states? This paper describes the collaborative development of a generalizable metric of community-level SPF fidelity whose intent is to determine exactly that. It covers how we 1) developed rating scales to assess implementation quality of each of the 5 steps, 2) developed scoring rubrics for a composite rating for each step, and 3) provided guidance to grantees for using these tools. Preliminary data will also be presented
|
| | | |
|
Session Title: Building and Assessing Capacity for Evaluation: Creating Communities of Learners Among Service Providers
|
|
Panel Session 867 to be held in Hanover Suite B on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Chair(s): |
| Tina Taylor-Ritzler,
University of Illinois, Chicago,
tritzler@uic.edu
|
| Discussant(s):
|
| David Fetterman,
Stanford University,
profdavidf@yahoo.com
|
| Abstract:
Community-based organizations are currently experiencing pressure to learn about evaluation and conduct their own evaluations. Some are able to meet these demands through partnerships with academic institutions designed to build capacity for evaluation utilizing empowerment and participatory approaches. Although there is literature available on evaluation capacity building, much is needed in terms of understanding its conceptualization, the process of building it and how to measure it. In this session, several researchers will present their work with a variety of community-based organizations in creating capacity for evaluation. First, we will present an ecological, contextual and interactive framework of evaluation capacity building that integrates models from the evaluation literature. Second, we will describe methods and strategies in measuring capacity building and we will discuss how to create learning communities with agency staff. Third, we will provide exemplars and discuss challenges encounter when doing this work and implications for the field of evaluation. Fourth, we will discuss evaluation capacity building strategies (ECB) used and how they have been evaluated based on a review of research on ECB. Finally, we will hear commentaries from a prominent researcher in the field: David Fetterman.
|
|
Building Capacity for Evaluation Among Service Providers: Conceptual Framework and Exemplar
|
| Yolanda Suarez-Balcazar,
University of Illinois, Chicago,
ysuarez@uic.edu
|
|
Based on the work we have been doing at the Center for Capacity Building for Minorities with Disabilities Research, we propose a contextual framework of capacity for evaluation.
A contextual and interactive model suggests a dynamic interplay between person factors and organizational factors. The person or group factors are exemplified by agency staff and/or program implementers while organizational factors speak for organizational policies, organizational culture and support systems that create an environment that facilitates capacity for evaluation. The framework assumes interplay between personal factors and organizational factors. As such, a CBO staff member may be willing and ready to learn how to evaluate a program he/she implements but lacks organizational support to do so in the form of lack of allocated time and resources. Capacity for evaluation can be created and facilitated at the individual level. Here, we are referring to the staff member or members who implement agency programs, are in direct contact with participants, and are experiencing tremendous pressure to document what they do and to produce tangible outcomes. We will discuss individual factors such as personal readiness, level of competence and experience, and individual leadership. The environment, policies, procedures and culture of the organization may be more or less facilitative of building capacity for evaluation in individual staff and the organization as a whole. The presenters will also discuss several factors at the organization level that can facilitate the process of building capacity: including organizational readiness, organizational resources and support allocated to evaluation, organizational leadership, organizational culture, organizational capacity to mainstream evaluation practices, organizational capacity to utilize findings and develop practices that sustain evaluation capacity, and organizational capacity to seek funding for their programs. We will also discuss implications for the art and the science of evaluation.
|
|
|
Measuring Evaluation Capacity: Methodologies and Instruments
|
| Tina Taylor-Ritzler,
University of Illinois, Chicago,
tritzler@uic.edu
|
|
Although there is a large literature on evaluation capacity building, it lacks specificity on issues of measurement and assessment of evaluation capacity. Most have looked only at evaluation products agencies generate (reports to funders) and satisfaction with training. We will present our multiple method system for assessing and measuring evaluation capacity. In this session, we will present the work being conducted nationally by the Center for Capacity Building on Minorities with Disabilities Research. We will describe in detail the instruments and procedures we use and challenges we encounter when measuring evaluation capacity building with organizations serving ethnic minorities with disabilities. We will share data drawn from multiple case study examples. Finally, our discussants will share their perspective on the contribution of our work to scholarship on evaluation capacity building.
| |
|
Using Collaborative Evaluation as a Strategy for Evaluation Capacity Building: First 5 Los Angeles' Quality Care Initiative
|
| Rita O'Sullivan,
University of North Carolina, Chapel Hill,
ritao@email.unc.edu
|
|
Collaborative Evaluation (O'sullivan, 2004) uses an approach to evaluation, which results in enhanced evaluation capacity building among key stakeholders. Evaluation Assessment and Policy Connections (EvAP) at the University of North Carolina at Chapel Hill worked collaboratively with First 5 Los Angeles staff and its 53 Childcare grantees to design a 30-month evaluation that would provide process, outcome, and policy information about the initiative. The evaluation activities also addressed enhancing the evaluation capacity of First 5 grantees, staff, partners, and Commissioners. Collaborative evaluation engages key program stakeholders actively in the evaluation process. Unlike distanced evaluation, where evaluators have little or no contact with program staff, collaborative evaluation deliberately seeks involvement from all program stakeholders during all stages of the evaluation. A collaborative stance can strengthen evaluation results and increase utilization of evaluation findings. Additionally, programs participating in collaborative evaluations develop an enhanced capacity to consume and conduct evaluations, while evaluators gain a better understanding of the program. The collaborative evaluation approach assumes that evaluation expertise within programs is developmental; and thus, the degree of collaboration must vary by the nature and readiness of the program. Evaluations completed with this collaborative approach have yielded improved evaluation capacity as measured by data quality, report writing, and evaluation use with program in the areas of education, social services, and health; the presenter also has found that collaborative evaluation may increase the resources available to the evaluation. This presentation will report how the evaluation contributed to the capacity building of the 53 grantees the majority of which were community based organizations.
| |
|
A Review of Research on Evaluation Capacity Building Strategies
|
| Jennifer Duffy,
University of South Carolina,
jenduffy@sc.edu
|
|
The growing literature on evaluation capacity building is one resource for learning more about what evaluation capacity building looks like in the field and what evidence there is for the success of these strategies. We will present findings from a review of empirical research on evaluation capacity building. The strategies for building evaluation capacity that are identified in this research will be described, and the methods used to evaluate these strategies will be discussed. We will highlight the evidence for successful strategies and limitations of the existing research. Questions for future research will be identified, with a focus on identifying successful strategies for building evaluation capacity.
| |
|
Session Title: Application of Computerized Adaptive Testing in Clinical Assessment
|
|
Demonstration Session 869 to be held in International Room on Saturday, November 10, 3:30 PM to 5:00 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Kendon Conrad,
University of Illinois, Chicago,
kjconrad@uic.edu
|
| Presenter(s): |
| Barth Riley,
University of Illinois, Chicago,
bbriley@chestnut.org
|
| Michael Dennis,
Chestnut Health Systems,
mdennis@chestnut.org
|
| Abstract:
The demand for increasingly complex clinical assessments that cover more areas and constructs mean that such assessments can take 1-2 hours, resulting in respondent burden and requiring considerable staff resources. Computerized adaptive testing (CAT) is a promising approach for improving assessment efficiency, but faces several challenges when applied to clinical assessment. Following an overview of CAT, we will discuss three important issues regarding CAT in clinical assessment. First, we will demonstrate various approaches to estimating measures on multiple clinical domains during a single CAT administration of a clinical assessment battery. Second, we will discuss how precision can be varied across the measurement continuum in order to maximize assessment efficiency and accuracy in clinical decision making. Third, we will discuss the use of fit statistics in the context of CAT to identify persons who present with atypical patterns of symptoms (e.g., persons who endorse suicidal ideation but are not depressed).
|
