|
Session Title: Evaluation Policy in the Changing United States Government Foreign Assistance Community
|
|
Panel Session 504 to be held in Capitol Ballroom Section 1 on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Chair(s): |
| Cynthia Clapp-Wincek,
Independent Consultant,
ccwincek@aol.com
|
| Discussant(s):
|
| Richard Blue,
Independent Consultant,
richardblue@earthlink.net
|
| Abstract:
In the 1970s and into the 1980s, USAID's evaluation system and practices were both innovative and well focused on impact and learning from experience. USAID was considered a leader in evaluation practices. In recent years, new actors have joined USAID in foreign assistance management. During this period, USAID has struggled with support to their evaluation system. This panel includes representatives from USAID, the Department of State's strategic planning office, the office of the Director of Foreign Assistance in the State Department and Millennium Challenge Corporation. Each will talk about evaluation policies and standards, how these are being developed, and how evaluation is playing out in practice in their agencies. After short presentations by each representative, the audience will be asked to contribute their experience and compare and contrast the different agency's approaches.
|
|
USAID Evaluation Policy and Practice
|
| Gerry Britan,
United States Agency for International Development,
gerry@usaid.gov
|
|
The USAID experience in developing a stable and useful evaluation policy and implementation structure has had considerable ups and downs since the mid-1970s when the Office of Evaluation was created to strengthen the Agency's evaluation system and to carry out evaluations. By 1981, under President Ronald Reagan, USAID had formed the Center for Development Information and Evaluation, consolidating into one center the archival and evaluation leadership function within the Bureau for Policy and Program Coordination. The creation of the Director for Foreign Assistance shifted the evaluation staff and the focus of the work to the development of the FACTS monitoring and reporting system leaving no personnel or resources for evaluation for over a year. Evaluation seems to be back under discussion at both DFA and USAID. By the time of the Annual Meeting in November, USAID will have made progress in reinvigorating evaluation.
|
|
|
Bureau of the Director of Foreign Assistance Evaluation Policy and Practice
|
| Donna Stauffer,
United States Department of State,
stauffer@state.gov
|
|
The bureau of the Director for Foreign Assistance was created in 2006 with a strong initial emphasis on input output monitoring within the context of the New Strategic Framework for foreign assistance. In 2007, DFA conducted an assessment of its own practices and made changes. DFA is currently working to define an evaluation policy.
| |
|
United States Department of State Evaluation Policy and Practice
|
| Melinda Crowley,
United States Department of State,
crowleym@state.gov
|
|
The U.S. Department of State implements foreign assistance programs as well but has not had a department-wide evaluation policy to support its strategic planning or results monitoring. The Bureau for Educational and Cultural affairs has had a strong evaluation unit and some other pockets of evaluation have developed as well. In the fall of 2008, the strategic planning office in the Resources Management Bureau (RM/SPP) began meeting with other bureaus to discuss evaluation issues and is working towards an evaluation policy.
| |
|
Millennium Challenge Corporation Evaluation Policy and Practice
|
| Harry Carr,
Millennium Challenge Corporation,
carrhc@mcc.gov
|
|
In 2004, Congress established the Millennium Challenge Corporation. MCC was 'purposely designed as a small corporation to promote, support and ensure accountability for the innovative foreign aid strategies it administers.' Due to that emphasis on accountability, MCC has used a set of performance indicators to determine countries' eligibility to receive programs. In 2006, MCC issued policy and guidance in monitoring and evaluation and the presentation will discuss what's working and what isn't.
| |
|
Session Title: Policy and Practice Implications of a Multi-faceted Approach to Building Evaluation Capacity in Community Colleges
|
|
Multipaper Session 506 to be held in Capitol Ballroom Section 3 on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Assessment in Higher Education TIG
|
| Chair(s): |
| William Rickards,
Alverno College,
william.rickards@alverno.edu
|
| Discussant(s): |
| Mary Williams,
Lumina Foundation,
mwilliams@luminafoundation.org
|
| Peggy Valdez-Fergason,
Independent Consultant,
pvaldezfergason@comcast.net
|
| Abstract:
This session brings together two independently developed approaches that are now coming together to create a complementary assessment strategy for community colleges. The strategy leverages both the organized and self-organizing system dynamics within community colleges. One approach focuses on the use of college-wide multiple data sources to identify college-wide issues about student learning and success that need to be addressed. The second approach uses a strategy of building Communities of Learning, Inquiry, and Practice (CLIPs). CLIPs are a structure for self-selected groups of faculty and staff to collaboratively investigate questions about their work. They undertake a three step evaluative inquiry process. The two processes work together to build a college-wide culture of inquiry.
These papers address issues of policy and practice based on over three years of experience and research about the assessment and inquiry processes discussed in the papers.
|
|
Using Institutional Databases to Improve Student Achievement: The Role of Data Facilitators
|
| Richard Voorhees,
Voorhees Group LLC,
rick@voorheesgroup.org
|
|
Over 80 community colleges nation-wide are involved in developing strategies to ensure student success through the 'Achieving the Dream' initiative of the Lumina Foundation. A vital part of the initiative is providing a 'data facilitator' for each college. The data facilitator plays a key role in helping a team (composed of many different stakeholders) analyze institutional level data to serve as the basis for policy and practice decisions about instructional and other changes to support student success.
This paper describes lessons learned by a data facilitator as he helped colleges build their capacity to use data to make institutional level decisions. The paper addresses the broad policy issues both for colleges and for funders and leaders of large scale national initiatives. The paper looks at the complementary nature of the use of institutional level data with the use of self-organizing groups such as Communities of Learning, Inquiry, and Practice (CLIPs).
|
|
Faculty Engagement Through Communities of Learning, Inquiry, and Practice (CLIPs)
|
| Beverly Parsons,
InSites,
beverlyaparsons@aol.com
|
|
Communities of Learning, Inquiry and Practice (CLIPs) are self-selected groups of faculty and staff who collaboratively investigate questions about their work. They undertake a three step evaluative inquiry process and operate under a set of Guiding Principles that integrate research on complex adaptive systems, inquiry processes, learning theory, communities of practice, appreciative inquiry, and change processes. The testing of the CLIP process at a California community college under a National Science Foundation grant showed that the process 1) builds collaborative relationships, 2) enhances evaluative inquiry skills, and 3) leads to changes in professional practice.
This paper provides the theoretical basis for the CLIPs and the key findings from a three year action research study. It looks at how the CLIP process complements a more structured analysis of institutional level data to build a sustainable culture of inquiry and evidence based decision making.
|
|
Session Title: Shaping Policy and Practice: Evaluative Contributions to the delivery of Education Technology and Professional Development in Jordan
|
|
Multipaper Session 507 to be held in Capitol Ballroom Section 4 on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Distance Ed. & Other Educational Technologies TIG
|
| Chair(s): |
| Audrey-Marie Moore,
Academy for Educational Development,
amoore@aed.org
|
| Discussant(s): |
| John Gillies,
Academy for Educational Development,
jgillies@aed.org
|
| Abstract:
The Education Reform for the Knowledge Economy in Jordan has prompted the development and implementation of new approaches to professional development and the teaching of information management in classrooms. The USAID-funded Jordan ERfKE Support Program (ESP) has been instrumental in assisting the Ministry of Education to shape policy into practice through the use of a multi-step approach to evaluation. To understand the impact of ESP activities on teacher and student learning, and institutionalization of the reform process within the MoE itself, the Monitoring and Evaluation Team in Jordan developed a developmental approach, which included experimental design, case study and measurement of systems change, to show how MoE policies impacted teacher and student learning in Jordan while leading to a more systemic understanding policy implementation in the field. The approaches also contributed to MOE conceptualization of future activities that shaped on-going improvements of the interventions.
|
|
Contributions to Policy Decision-making: The Effectiveness of Blended, On-line learning strategies in Jordanian Secondary Education
|
| John Rich,
Evaluvision,
drjohnrich@evaluvision.com
|
| Audrey-Marie Moore,
Academy for Educational Development,
amoore@aed.org
|
|
In Jordan, the current approach to teaching the Management Information (MIS) stream curriculum is teacher-centered. The USAID-funded, ERfKE Support Program developed a series of online modules that introduced students to the same MIS concepts and facts through a student-centered, project-based learning approach. In this approach, students were guided through the process of self-selecting a profit-making business enterprise, and creating their own business plan, based on commonly used principles in the real-life business world. In the process of working through this business plan, required students to master concepts without reliance on textbooks and lecture alone. This paper will discuss the effectiveness of the approach when compared to the traditional approach; identify the characteristics of students, teachers, and/or schools that amplified, moderated, or suppressed the effect; and show how the approach and results of the evaluation contributed to policy decisions made by Ministry officials to institutionalize the approach in Jordan.
|
|
Evaluative Contributions to Teacher Professional Development: An Analysis of Demand-Driven Training in Jordan
|
| David Balwanz,
Academy for Educational Development,
dbalwanz@aed.org
|
| Razan Al-Hadidi,
Academy for Educational Development,
ralhadidi@esp-jordan.org
|
| Ahmed Thawabieh,
Ministry of Education,
ahmadthawabieh@yahoo.com
|
|
This paper presentation will discuss evaluation findings that share teacher and student perceptions on the efficacy and impact of demand-driven training; the extent to which new content-knowledge and teaching strategies (co-planning, project-based learning, etc.) are used in the classroom; teacher and principal perceptions of learning team implementation in supporting teacher work and professional development, and all of the above in relation to the management and resource environments in which teachers work. The paper will also show how engagement of key Ministry personnel and the use of formative evaluation led to the evolution of the Ministry's professional development approach from a reform oriented intervention (to upgrade teacher skills for the introduction of a new curriculum) to one that could institutionalize a decentralized approach supporting the on-going professional development of teachers in Jordan.
|
|
Education Reform Support: Creating and Supporting Conditions for Success
|
| Clive Cavanagh,
Knowledge Economy Support Program,
ccavanagh@esp-jordan.org
|
| Audrey-Marie Moore,
Academy for Educational Development,
amoore@aed.org
|
|
This paper presentation will discuss institutionalization of reform components that include the strategies, approaches and capacities supporting the implementation of the management Information Stream (MIS) curriculum and the new professional development model for MIS teachers. To support discussion of sustainability and begin to measure the institutionalization of reform components, the ESP team used a set of rubrics to evaluate the development of institutions capable of providing technically and systematically sound support such as information systems, incentives, legal frameworks, and qualified staff members who have the ability to coordinate national priorities within the context of local and international cooperation. The framework allowed the project along with their Ministry counterparts to monitor changes in the conditions that allow for the integration of interventions ensure sustainable changes.
|
|
Session Title: The AEA Town Hall Meeting: Internal Scan Findings, Policy Governance, Actions, and Next Steps
|
|
Think Tank Session 508 to be held in Capitol Ballroom Section 5 on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the AEA Conference Committee
|
| Presenter(s):
|
| Leslie Goodyear,
Education Development Center Inc,
lgoodyear@edc.org
|
| Susan Kistler,
American Evaluation Association,
susan@eval.org
|
| William Trochim,
Cornell University,
wmt1@cornell.edu
|
| Discussant(s):
|
| Thomas Chapel,
Centers for Disease Control and Prevention,
tchapel@cdc.gov
|
| Thomas Schwandt,
University of Illinois Urbana-Champaign,
tschwand@uiuc.edu
|
| Mary Stutzman,
Florida State University,
mstutzma@fsu.edu
|
| Colleen Manning,
Goodman Research Group Inc,
manning@grginc.com
|
| Abstract:
Over 2000 members shared their background, input, concerns, and suggestions via the AEA Internal Scan, which included a membership survey, interviews and online discussion groups. The findings were useful and insightful and affected the association's strategic planning, programs, and services. We will highlight specific findings from the scan and tell how those findings have translated into plans and action for the association. We will also be sure that you know how to access the full scan, instruments, reports, and articles so that you may dig deeper into the scan results. Finally, we will examine our move to policy governance, share our plans for making AEA more engaged with the members, and more nimble and responsive, and open the floor for discussion and input as we plan for 2009 and beyond.
|
|
Session Title: The Science and Art of Creating Evaluation Standards That Stand the Test
|
|
Panel Session 509 to be held in Capitol Ballroom Section 6 on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the AEA Conference Committee
|
| Chair(s): |
| Hazel Symonette,
University of Wisconsin Madison,
symonette@bascom.wisc.edu
|
| Abstract:
Intended as a means to introduce the 3rd edition of The Program Evaluation Standards, this session goes well beyond simple descriptions of the product. Participants get an inside perspective on this extensive and difficult development process as well as the Joint Committee's focus on quality assurance. This session explores four facets of evaluation standards: (a) explanation of the operating procedures for developing standards why they are important, how they have changed, and why they changed; (b) the procedural changes employed by the Joint Committee and input received from the evaluation and practitioner professionals; (c) the fruits of the process that will serve as an American National Standard for the next decade or more, and (d) the steps the Joint Committee takes through an independent validation panel to assure quality in both the development process and the resulting product. Participant questions and comments are encouraged.
|
|
Key Elements of Developing Sound Evaluation and How the Joint Committee Works to Achieve Them
|
| Arlen Gullickson,
Western Michigan University,
arlen.gullickson@wmich.edu
|
|
Since its inception in 1975, the Joint Committee has become the nation's preeminent developer of education evaluation standards. Its first and foremost product,' The Program Evaluation Standards.' is certified as an American National Standard by the American National Standards Institute (ANSI) and has guided evaluation planning and implementation both nationally and internationally since its first publication in 1981.
This presentation focuses on the third edition of these standards. It describes the methods the Joint Committee employs to ensure high quality in its standards and the ways in which development methods have changed since the standards were first developed. This paper tracks the genesis of the standards development process, describes the forces that have shaped its development processes, and addresses the many practical issues that can derail best intentions to assure that the standards developed actually are principles commonly agreed to in the evaluation profession.
|
|
|
Stakeholder Contributions to the Development of The Program Evaluation Standards: 3rd Edition
|
| Don Yarbrough,
University of Iowa,
d-yarbrough@uiowa.edu
|
| Flora Caruthers,
National Legislative Program Evaluation Society,
caruthers.flora@oppaga.state.fl.us
|
| Rodney Hopson,
Duquesne University,
hopson@duq.edu
|
| Lyn Shulha,
Queen's University at Kingston,
shulhal@educ.queensu.ca
|
|
A primary goal of the Program Evaluation Standards development task force is that all interested stakeholders across the 16 sponsoring organizations as well as in the evaluation communities at-large have adequate opportunities to be involved, contribute suggestions, and make recommendations. This paper describes input received during development of The Program Evaluation Standards, 3rd Edition, including results of national and targeted surveys for needs assessment and format development, the 3-phased national and international review processes and results, the on-going hearings at national conferences, and the methodology and yields of the multi-stage field trials, some of which have been supported by the National Science Foundation. The full paper provides examples of feedback received via reviews, hearings, and field trials, and how that feedback has been put to good use. Remaining opportunities to contribute via field trials, reviews and national Web-based asynchronous 'hearings' will be discussed.
| |
|
An Introduction to the Program Evaluation Standards, Third Edition
|
| Lyn Shulha,
Queen's University at Kingston,
shulhal@educ.queensu.ca
|
| Rodney Hopson,
Duquesne University,
hopson@duq.edu
|
| Flora Caruthers,
National Legislative Program Evaluation Society,
caruthers.flora@oppaga.state.fl.us
|
| Don Yarbrough,
University of Iowa,
d-yarbrough@uiowa.edu
|
|
The development process for the revised Program Evaluation Standards, 3rd Edition is nearing completion as the final reviews, hearings and field trials are completed following Joint Committee and ANSI guidelines. The near final draft is expected to be approved in September 2008 and is scheduled for publication in 2009. The 3rd edition is organized into individual chapters addressing the five key dimensions of quality in evaluations: Feasibility, Propriety, Accuracy, Utility, and Metaevaluation. In addition, it contains supporting chapters addressing the history of the standards development and development processes, what is 'new' to this edition and why a revision of the standards is needed now (and will be needed again within the next decade), how to use the standards in practical situations, and extended cases describing how multiple standards are applied in real life settings. This paper provides an introduction to the revised standards and to the 3rd edition book.
| |
|
The New Program Evaluation Standards: A First Look
|
| Marvin Alkin,
University of California Los Angeles,
alkin@gseis.ucla.edu
|
|
This presentation describes the validation panel's analysis of the Joint Committee's work in developing the standards and the quality of the standards that resulted. This panel, comprised of leading educators and evaluators and led by Dr. Alkin who is one of this nation's leading evaluation practitioners and theorists, is invested with determining whether the Joint Committee has met its own procedural expectations for development of standards and produced strong standards that truly are evaluation principles commonly agreed to by the education profession. The makeup of the panel, the procedures used, and the data employed are all described as factors important in panel determinations. This presentation directly addresses findings such as the congruence between procedures mandated for development of standards and actual procedures followed in the development process, as well as other criteria it applied to make in determining the Joint Committee's success in reaching the intended goals.
| |
|
Session Title: Use of Indicators, Screeners, and Scaling Methods in Substance Use and Mental Health Evaluations
|
|
Multipaper Session 510 to be held in Capitol Ballroom Section 7 on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG
|
| Chair(s): |
| Saumitra SenGupta,
APS Healthcare Inc,
ssengupta@apshealthcare.com
|
|
Development of Evaluation Indicators for a Policy Framework for Children and Youth Mental Health Services
|
| Presenter(s):
|
| Evangeline Danseco,
Center of Excellence for Child and Youth Mental Health,
edanseco@cheo.on.ca
|
| Ian Manion,
Center of Excellence for Child and Youth Mental Health,
imanion@cheo.on.ca
|
| Abstract:
The Ontario Ministry of Children and Youth Services in Canada developed a policy framework for the children and youth mental health sector highlighting four broad goals: accessibility, integration, quality and evidence-based care, and accountability. Indicators to assess progress in achieving these goals were identified and selected using a modified Delphi approach. An expert panel of 21 stakeholders (parents, service providers and administrators, policy-makers, researchers, and advocates) reviewed candidate indicators based on relevance, validity, and feasibility. There were two rounds of online surveys and two panel meetings over six months. An initial 225 indicators were identified with seven of these being proposed for piloting and potential system-wide adoption. Through a series of online surveys and panel meetings over six months, the final list of recommended indicators included seven indicators based on validity, feasibility and relevance. This presentation will discuss the modified Delphi approach and the processes used to select the indicators.
|
|
Using Key Component Scaling to Evaluate the Implementation of a Community Crisis Bed Program
|
| Presenter(s):
|
| Purnima Sundar,
Carleton University,
purnimasundar@hotmail.com
|
| John Sylvestre,
University of Ottawa,
jsylvest@uottawa.ca
|
| Parastoo Jamshidi,
University of Ottawa,
pjams070@uottawa.ca
|
| Matthew Manion,
University of Ottawa,
mmani069@uottawa.ca
|
| Abstract:
The Community Crisis Bed (CCB) program in Ottawa, Ontario, provides community-based alternatives to hospitalization for people experiencing a mental health crisis. Based on work by Cousins et al. (2004), a Key Component Scaling tool was developed to evaluate program implementation. From a program logic model, key programmatic activities were identified. For each activity, descriptions of “full” and “low” program implementation were written based on program documentation and feedback from an evaluation steering committee. Staff members from three program stakeholder groups, as well as clients and family members, participated in focus groups or interviews during which they rated and discussed implementation of the various program components. Scale data were used to categorize program components as achieving full implementation, inconsistent/partial implementation, or low level implementation. Qualitative data showed that implementation challenges were attributable to differences in expectation among the partners, as well as difficulties in communication, information sharing, and coordinating partners.
|
|
Theory of Planned Behavior and Propensity Scores: Applications for Longitudinal Evaluation of Underage Alcohol Use Prevention Programs
|
| Presenter(s):
|
| Robert Seufert,
Miami University at Middletown,
seuferrl@muohio.edu
|
| Mark Carrozza,
University of Cincinnati,
mark.carrozza@uc.edu
|
| Abstract:
The research evaluates the utility of applying the Integrated Theory of Planned Behavior for understanding, predicting, and reducing underage alcohol use. The authors also assess the merit of using logistic regression-based propensity scores as a methodology to form statistically equivalent quasi-experimental groups during longitudinal program evaluations. Applications of other statistical procedures to evaluate program impact are assessed. Finally, we discuss and evaluate the policy implications of using media campaign messages to reduce underage drinking and related health and safety problems
|
|
Mental Health Performance Indicators to Measure Disparity: The California External Quality Review Experience
|
| Presenter(s):
|
| Saumitra SenGupta,
APS Healthcare Inc,
ssengupta@apshealthcare.com
|
| Hui Zhang,
APS Healthcare Inc,
hzhang@apshealthcare.com
|
| Sheila Baler,
APS Healthcare Inc,
sbaler@apshealthcare.com
|
| Abstract:
The role played by the External Quality Review Organizations (EQRO) in managed care waiver states in reviewing health and mental health plans are seldom discussed at the American Evaluation Association annual meetings. The proposed session will introduce the work of the California Mental Health EQRO in assessing statewide disparities in public mental health service access, expenditures and implications in policy initiatives and performance improvement at local and state levels.. The CA EQRO has utilized and improvised upon some commonly used performance indicators such as penetration rate, cost per beneficiary, and retention rate to examine racial and gender based disparities in the state. These indicators will be briefly described followed by some current findings and challenges of using this methodology in assessing disparity such as definitional issues, the EQRO role, as well as strengths of this approach including complementary data collection methods employed by the EQRO to enhance the review findings.
|
| | | |
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Blogging to the Beat of a New Drum: The Use of Blogs and Web Analytics in Evaluation |
|
Roundtable Presentation 511 to be held in the Limestone Boardroom on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Integrating Technology Into Evaluation
|
| Presenter(s):
|
| Cary Johnson,
Brigham Young University,
cary_johnson@byu.edu
|
| Stephen Hulme,
Brigham Young University,
byusnowboarder@yahoo.com
|
| Abstract:
Blogs are changing the face of marketing, business, and culture. How are they changing the practice of evaluation? This roundtable discussion will focus on how evaluators can use the combination of blogs and web analytics to assist in their evaluative processes both quantitatively and qualitatively. The facilitators will guide the participants in discussing some of the pros, cons, and potential hazards of using blogs in evaluations.
|
| Roundtable Rotation II:
Voicethread: A New Way to Evaluate |
|
Roundtable Presentation 511 to be held in the Limestone Boardroom on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Integrating Technology Into Evaluation
|
| Presenter(s):
|
| Stephen Hulme,
Brigham Young University,
stephen_hulme@yahoo.com
|
| Tonya Tripp,
Brigham Young University,
tonya.tripp@byu.edu
|
| Abstract:
Voicethread.com is a new website that allows users to comment on videos, images, documents, and presentations online. Not only can they make text comments, but they can also make audio and video comments. This is a rich resource for evaluators because they can gather far more data via a voicethread than a simple survey or interview. Because Voicethread.com is accessible to anyone with an internet connection, you can gather feedback from people outside your city, state, or even country. This roundtable will provide an open discussion about how this new technology could enhance evaluation in the future.
|
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Painting a Picture of Evaluating Trainee Needs: Content and Experience With Training Methodologies |
|
Roundtable Presentation 512 to be held in the Sandstone Boardroom on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Needs Assessment TIG
|
| Presenter(s):
|
| Sandra Naoom,
University of South Florida,
snaoom@fmhi.usf.edu
|
| Adrienne Cadle,
University of South Florida,
adriennewoodleycadle@msn.com
|
| Abstract:
There is an intuitive benefit to evaluating a need for training as it identifies the motives for delivering (or not delivering) training, the method of delivery, the content to be trained and appropriate trainees. Despite the benefits of evaluating training, it is seldom reported in peer reviewed publications (Arthur et. al, 2003). This evaluation was aimed at assessing two different levels of school district leaderships’ need for training. The management-oriented approach was used to help trainers tailor the training they will deliver. The purpose of this formative evaluation was to identify training needs, past experiences with training others, and knowledge of training content. Training needs were assessed using mixed methodology, including paper and pencil surveys, as well as interviews. Organizational and person level analyses were conducted as part of the assessment of training needs. The benefits and limitations to conducting a needs assessment for training will be presented further in this paper.
|
| Roundtable Rotation II:
Measuring Impact Beyond the Training Room: Executive Leaders Training Program Evaluation |
|
Roundtable Presentation 512 to be held in the Sandstone Boardroom on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Needs Assessment TIG
|
| Presenter(s):
|
| Jecky Misieng,
University of South Florida,
mjecky@gmail.com
|
| Abstract:
This evaluation was conducted to measure overall impact of executive leadership training on behavior changes and performance of district level administrators in a South Eastern school district. Approximately 100 participants, all new to their leadership positions, completed the training over a four year period (2004-2007). Kirkpatrick’s and Guskey’s evaluation models were used to provide insight on information at five different levels including reaction, learning, behavior, results, organizational support and change management. Several instruments were used to collect quantitative and qualitative data from participants, their supervisors and coaches in order to address the evaluation questions. These methods include post-training surveys, questionnaires, and interviews. Impacts at each of the above mentioned levels were examined based on the training evaluation criteria. Participant responses on the training, including logistic difficulties, expenses and limitations that provided useful information to improve the training will be discussed.
|
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Measuring Changes in Teachers’ Mathematical Conceptual Understanding for Evaluation of Math-Science Partnership Professional Development Effectiveness |
|
Roundtable Presentation 513 to be held in the Marble Boardroom on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Presenter(s):
|
| MaryLynn Quartaroli,
Northern Arizona University,
marylynn.quartaroli@nau.edu
|
| Hollace Bristol,
Coconino County Education Services Agency,
hbristol@coconino.az.gov
|
| Abstract:
The Mathematics and Science Partnership (MSP) competitive grant program encourages partnerships between local education agencies and universities to collaboratively engage in professional development activities aimed at increasing teachers’ content knowledge and improving pedagogical practices. To facilitate planning and implementation of effective K-5 mathematics teacher training activities, one MSP grant team substantially modified the Frayer model graphic organizer into a new instrument, the Teachers’ Conceptual Understanding and Pedagogical Practice (TCUPP), as a pre-post assessment of teachers’ conceptual understanding of fundamental mathematical concepts and appropriate pedagogical practices. This instrument supplements data from standardized mathematics content tests and teacher self-assessments. In addition to serving as a pre/post measure, the new instrument can also be used to inform specific curricular decisions for summer institutes or other professional development sessions.
|
| Roundtable Rotation II:
Developing Higher Education Institution and University Partnerships in Reform of Science and Mathematics Education: A Three-Year Evaluation Study |
|
Roundtable Presentation 513 to be held in the Marble Boardroom on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Presenter(s):
|
| Chad Ellett,
CDE Research Associates Inc,
cderesearch@charter.net
|
| Judith Monsaas,
University System of Georgia,
judith.monsaas@usg.edu
|
| Abstract:
This round table discusses the findings of a recent qualitative evaluation (individual and focus group interviews) of factors that contribute to changes in organizational culture in four higher education institutions in a southeastern state and traditional policies and practices framing faculty tenure and promotion decisions and incentive/reward structures. The results were synthesized from extensive interviews with university/college level and department level administrators and key faculty informants. The primary focus of the study was to examine whether concerted (four-year) attempts to change higher education norms, values, and policies resulted in changes in organizational cultures supportive of new policy directions in faculty rewards and the enhancement of partnerships between K-16 science and mathematics education faculty and public school teachers and student achievement. The program evaluation results are interpreted and discussed in view of implications for the evaluation of statewide program innovation initiatives in mathematics and science education.
|
|
Session Title: Innovation, Complex Research Teams and Problems of Integration: The Missing Link
|
|
Panel Session 514 to be held in Centennial Section A on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Jerald Hage,
University of Maryland,
jerryhage@yahoo.com
|
| Abstract:
The management of innovation literature has a panacea that complex research teams with high rates of communication are responsible for innovation. The problems with this panacea are that it: (1) ignores the difficulties in creating high rates of communication in complex research teams; (2) does not consider the variety of complex research teams that exist; and (3) ignores the variety of linkages or ways in which teams can be integrated into networks of technical information. Our intent is to provide evaluators with check lists for how to determine the missing links of integration and how these might be measured.
|
|
Kinds of Complex Research Teams
|
| Gretchen Jordan,
Sandia National Laboratories,
gbjorda@sandia.gov
|
|
The first problem and one that exists in all research teams is the amount of cognitive distance between the specialties within the complex team. As this distance grows, the possibility for radical innovation increases but communication tends to decline. To overcome this decline in communication, special mechanisms have to be used and policies instituted.
The second problem of integration occurs when we consider the differences between small and large complex research teams. In the latter case, not only is the likely decline in communication augmented but the problem of maintaining a balance between research project autonomy and coordination occurs.
The third problem of integration occurs when we consider the differences in the context of the small and large complex research teams. They are located in different kinds of idea innovation network contexts that can be distinguished by the kinds of gaps that are likely to occur with the growth in knowledge. The key is that the complex teams have to bridge these gaps.
|
|
|
Examples of Integrated and Non-integrated Research Teams in a Highly Innovative Research Organization' the Institut Pasteur
|
| Jerald Hage,
University of Maryland,
jerryhage@yahoo.com
|
|
What are the organizational characteristics that encourage the formation of integration complex research teams? A historical analysis of the Institut Pasteur highlights the following structural characteristics that facilitated the creation of complex research teams: complex charter, terrific trio of visionary leaders, recruitment of scientists from very different channels, and multiple sources of funds. In addition, the Institut Pasteur illustrates multiple mechanisms for creating integration: common training course, duo-team leadership, joint research projects and publications, and a cultural sense of belonging to a family. The departments or laboratories in the Institut Pasteur that had these integrative characteristics achieved radical breakthroughs, while those lacking in these characteristics did not achieve similar breakthroughs. Likewise studies across time indicate how important it is to have complex teams that are integrated.
| |
|
Critique of Current Network Studies: Not Measuring Complex Nodes, Project Integration and Gaps in the Idea Innovation Network
|
| Jon Mote,
University of Maryland,
jmote@socy.umd.edu
|
|
A flurry of network studies has recently been completed but many of these have important limitations and are not connected to the management of innovation literature. In general, these studies fail to identify the network structures and processes that are most appropriate for achieving various types of innovation. One key issue is the need to study network ties in real time rather than on the basis of joint publications, joint research projects, etc. The research environment survey provides a series of questions that solve this, and examines the kinds and extent of various networks connected to the research project. In other words, we should measure links from the bottom up. Another key problem is the failure to measure gaps in the idea innovation network. It is not just a question of links, but which ones and are they in the right location.
| |
|
Session Title: Core Quantitative Issues: New Developments in Outcome Measures
|
|
Panel Session 515 to be held in Centennial Section B on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Manuel C Voelkle,
University of Mannheim,
voelkle@rumms.uni-mannheim.de
|
| Abstract:
The session gives an overview of important new developments in constructing, analyzing and communicating outcome measures in evaluation research. It contains methodologically oriented presentations as well as examples from current evaluation studies. It is thus expected to be of interested to both the methodologically interested and applied researcher alike. The first presenter, Lee Sechrest, talks about the importance of properly calibrating outcome measures in order to make them meaningful for policy makers and practitioners. AndrTs Steffanowski proposes a new individualized outcome measure in rehabilitation research by combining information on status and change. Manuel Voelkle addresses the more general issue of emergent versus latent outcome measures in longitudinal designs. Mende Davis demonstrates the use of Rasch modeling to construct a measure of academic and professional success in mathematics, and illustrates its use in an outcome evaluation. Finally, Werner Wittmann discusses different approaches to synthesize outcome measures of large-scale evaluation studies.
|
|
Properly Calibrated Measures
|
| Lee Sechrest,
University of Arizona,
sechrest@u.arizona.edu
|
|
The measures used in social science are often expressed in metrics that have no intrinsic meaning. Even measures of effect size are often not interpretable in any direct way. Persistent efforts should be exerted toward calibrating measures for meaning so that the effects of interventions can be described in ways that make sense to policy makers and practitioners. Unfortunately, very few such efforts have been mounted, and none systematically. It is easy, however, to exemplify the need and to illustrate the possibilities by reference to existing work. Examples also show why calibrated measures would be more persuasive and likely to result in implementation of effective interventions. Calibration research is an activity distinct from the development of basic measuring tools and need not interfere in any way with the production of evaluation measurement tools based on sound theory, state of the art methods, and pragmatic concerns for their implementation in the field.
|
|
|
Individualized Outcome Measures in Pre-Post-Studies: Combining Information on Status and Change
|
| Andros Steffanowski,
University of Mannheim,
andres@steffanowski.de
|
| Manuel C Voelkle,
University of Mannheim,
voelkle@rumms.uni-mannheim.de
|
|
Social interventions try to achieve two goals: First, to cure (or at least ameliorate) dysfunctional states (rehabilitation) and, second, to maintain desirable states (prevention). Regarding these two aspects, how is a fair outcome evaluation possible? The traditional approach has been to compute pre-post-difference scores. This, however, addresses only the first goal but not the second. In other words, maintaining a dysfunctional state, as well as maintaining a healthy state, would result in a zero-effect size (d = 0), with zero-effects being typically interpreted as 'no success'. Accordingly, using only pre-post-difference scores in outcome evaluation, the overall effect can be severely underestimated. To deal with this problem, an alternative measure has been developed by combining z-standardized pre-post-difference scores with z-standardized post-status scores for each item. The procedure is illustrated using a dataset of N = 858 psychosomatic inpatients with results indicating high reliability and good validity of the new outcome measure.
| |
|
Emergent Versus Latent Outcome Measures in Longitudinal Analysis
|
| Manuel C Voelkle,
University of Mannheim,
voelkle@rumms.uni-mannheim.de
|
| Andros Steffanowski,
University of Mannheim,
andres@steffanowski.de
|
|
When designing and analyzing outcome criteria it is important to distinguish between emergent (i.e., formative) and latent (i.e., reflective) measures. While the former have typically been analyzed using methods of variance decomposition, the latter have a factor analytic tradition (Cole, Martin, & Steiger, 2005). In this presentation the distinction is reviewed for the analysis of longitudinal data. It is argued that approaching the distinction from the perspective of latent growth curve modeling as a general data analytic system for the analysis of change has several methodological, didactical and statistical advantages. All arguments are illustrated by our research on quality monitoring in ambulatory psychotherapy and results of a short Monte-Carlo simulation are presented to evaluate the underlying assumptions specific to the analysis of change.
| |
|
Measuring the Educational Pipeline
|
| Mende Davis,
University of Arizona,
mfd@u.arizona.edu
|
|
The measurement of change is often limited to single outcome variables, even when multiple measures have been collected. Relying on a single outcome measure can decrease power and reduce the likelihood of detecting the effect of an intervention. Combining multiple measures into a scale to measure change may result in greater sensitivity to intervention effects. This possibility is illustrated by the development and illustration of a scale to measure academic and professional progress in graduate school. An educational pipeline scale can incorporate multiple types of indicators, multiple sources of data, and even processes that play out over time. Rasch analyses are used in scale development, and the resulting scale is demonstrated in a program evaluation.
| |
|
Outcome Measurement and Meta-Analysis, What are the Most Adequate Effect Sizes?
|
| Werner Wittmann,
University Mannheim,
wittmann@tnt.psychologie.uni-mannheim.de
|
|
The majority of programs we evaluate are complex ones, i.e. intervention packages. Fair evaluations therefore need a set of different outcome measures. Applying meta-analysis to synthesize the effects of such programs we have several options. One is computing the effect sizes for each single outcome measure and average them or report them as an outcome profile to visualize level, scatter and shape what the program produced. Another strategy is considering the redundancy of the measures and reducing them via aggregation or factor analysis. The effect sizes resulting from the latter ones should be higher than the average of the single outcomes. Data from large-scale program evaluations using such multiple act outcome criteria are used to illustrate the differences. Meta-Analysis with no access to the intercorrelations of the outcomes used, will underestimate the effects of programs.
| |
|
Session Title: Methodology and Tools for Assessing the Quality of Evidence
|
|
Panel Session 517 to be held in Centennial Section F on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Research on Evaluation TIG
|
| Chair(s): |
| Nancy Kingsbury,
United States Government Accountability Office,
kingsburyn@gao.gov
|
| Discussant(s):
|
| Leslie J Cooksy,
University of Delaware,
ljcooksy@udel.edu
|
| Abstract:
Evaluators in all contexts are responsible for assuring that evaluation work follows good practice guidelines (e.g. AEA's Guiding Principles, Joint Committee Standards). Evaluators are also called upon to make judgments about the quality of others' studies. This may result in a metaevaluation of a single study, assessing study designs for their capacity to respond to particular policy questions, or, among other reasons, evaluating study quality for inclusion in a meta-analysis to determine the effectiveness of interventions from extant evaluations. Evaluators have devised heuristic tools such as checklists for making such judgments and scorecards to present summary snapshots of quality judgments to time-burdened decisionmakers. The first paper provides an overall framework within which the tools for assessing quality described in the other papers can be situated. The Discussant will focus on the challenges and opportunities evaluators' face when assessing quality through metaevaluation and its associated tasks.
|
|
Metaevaluation: A Conceptual Framework for Improving Practice
|
| Valerie Caracelli,
United States Government Accountability Office,
caracelliv@gao.gov
|
| Leslie J Cooksy,
University of Delaware,
ljcooksy@udel.edu
|
|
Metaevaluation, a mechanism for improving and informing stakeholders of an evaluation's strengths and weaknesses, is specifically called for in the Joint Committee's Program Evaluation Standards. A research program to clarify the current state of conceptual and technical development in metaevaluation was started in 2007. The Metaevaluation Project (Cooksy & Caracelli) has examined misunderstandings in terminology, purposes served, criteria of quality used, and its range of use in practice. The practice of metaevaluation as defined by the Joint Committee Standards remains limited, yet metaevaluative tasks requiring judgments of quality are frequently a part of other evaluation methodologies such as peer review, research synthesis, and meta-analysis. Drawing on the first phase of the project we will describe the theory and methodology associated with metaevaluation, including the criteria of quality (standards, principles, paradigm specific) used in practice. This paper sets the stage for the metaevaluative tools described in subsequent presentations.
|
|
|
Using Checklists to Assess Design Quality: Applications and Utility
|
| Cindy K Gilbert,
United States Government Accountability Office,
gilbertc@gao.gov
|
|
Evaluators at the U.S. Government Accountability Office are often asked to assess the quality of evaluation designs - those of federal agencies as well as other research entities - at times making recommendations for improvement. For example, a recent report outlined the strengths and weaknesses of a Department of Defense (DoD) pilot program evaluation design, highlighting specific changes that could improve the validity of the study. Several types of evaluation checklists are available to guide such efforts, and are used to varying degrees at GAO. This paper will discuss available evaluation checklists, the characteristics these checklists have (and do not have) in common, the extent to which these and other frameworks are used at GAO, and their utility in a government auditing and evaluation setting.
| |
|
Ensuring Quality in 'Score Card' Methodologies
|
| Martin de Alteriis,
United States Government Accountability Office,
dealteriism@gao.gov
|
|
A number of US Federal Government agencies use 'score cards' or 'report cards' to assess key aspects of performance. For example, this methodology can be used to assess whether selected agencies have fully implemented, partially implemented, or taken no steps to implement a set of good practices. While 'score cards' allow for succinct presentations and definitive findings, there are also some potential limitations with this methodology, including possible oversimplification, and the 'shoehorning' of some issues into categories for which they are not a particularly good fit. Based on a review of reports that have used score cards, and input from evaluation specialists both inside and outside of government, this presentation will: 1) discuss the advantages and limitations of the score card methodology; 2) lay out key decisions that need to be made when constructing and using a score card; and 3) provide guidance on how quality can be ensured.
| |
|
Shaping a Quality Product: A Balanced Approach to Assessing the Quality of Performance Audit in the United Kingdom
|
| Jeremy Lonsdale,
National Audit Office United Kingdom,
jeremy.lonsdale@nao.gsi.gov.uk
|
|
Being seen to produce 'high quality' work is essential to the credibility of all evaluators. This paper will examine the quality assurance arrangements of the National Audit Office in the United Kingdom for its performance audit work, a form of evaluation that plays an important role in assessing the success of government programs. In particular, it will consider: How does the NAO define the quality of its products? Where do the criteria it uses come from? How does the NAO measure and monitor quality? And how is it trying to raise the quality of its work? The paper will assess how the NAO's approach to 'quality' has developed over the years, influenced in part by other disciplines. It will draw on a range of evidence, including 15 years of independent quality reviews, and highlight the current 'balance scorecard' arrangements designed to meet the expectations of different audiences.
| |
|
Session Title: Excellence in Extension Evaluation: Views From Extension Education Evaluation Topical Interest Group Award Winners
|
|
Panel Session 518 to be held in Centennial Section G on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Extension Education Evaluation TIG
|
| Chair(s): |
| Mary Arnold,
Oregon State University,
mary.arnold@oregonstate.edu
|
| Discussant(s):
|
| Mary Arnold,
Oregon State University,
mary.arnold@oregonstate.edu
|
| Abstract:
Panelists in this session will provide observation and reflection on Extension evaluation, from where it has been, to where things are currently, to where we are headed in the future. Each panelist will speak from the perspective of one of the five areas in which the EEE-TIG offers awards annually: 1) Sustaining excellence in Extension evaluation; 2) providing administrative leadership for evaluation; 3) promoting excellence in evaluation; 4) demonstrating excellence in evaluation teaching; and 5) providing outstanding leadership and service to the Extension evaluation profession. Presenters on this panel are past or current recipients of the EEE-TIG award in the category in which the panelist will speak. The collective wisdom and experience of this panel will provide great insight for any evaluator working as part of a complex organization such as Extension.
|
|
Building Evaluation Capacity: Excellence in Evaluation Training
|
| Lisa Guion,
North Carolina State University,
lisa_guion@ncsu.edu
|
|
This award is given for evaluation training of outstanding quality. Criteria for the excellence in evaluation training include creativity, enablement of trainees in evaluation practice, and the use of training to further extension evaluation. The panelist is a past award winner with over 12 years of working in Extension evaluation. She has witnessed significant progress in evaluation capacity building efforts within Extension and within the two Extension systems in which she worked. Today, more in-depth, ongoing training is being provided that is being coupled with timely, user-friendly evaluation resources and tools. These efforts are producing Extension educators who have more positive/favorable attitudes towards evaluation, and are more knowledgeable and skilled in evaluation design and methods. This panelist will provide a framework and discussion of some of the critical components necessary for Extension organizations that want to engage in a holistic, comprehensive, multi-faceted evaluation capacity building effort within their state.
|
|
|
Sharing Extension's Impact: Excellence in Program Evaluation
|
| Murari Suvedi,
Michigan State University,
suvedi@msu.edu
|
|
This award is given for an evaluation of outstanding quality. Criteria for this award are accuracy, feasibility, propriety, and utility, with particular emphasis placed on utility in improving Extension programs, policies and/or accountability. The panelist is a past award winner and professor of program evaluation at a major land-grand university. His professional background is varied, bringing a unique perspective to the evaluation of agriculture, environment and natural resources programs and projects. He conducts evaluation of Extension programs, teaches undergraduate and graduate courses in program development and evaluation, and offers short courses or workshops for development professionals focusing on evaluation. He has conducted evaluation of Extension programs and initiatives in Mexico, India, Nepal and the United States. The panelist will outline major factors influencing the quality of program evaluation in Extension and discuss ways to address them.
| |
|
Supporting Evaluation Efforts: The Mary Nell Greenwood Extension Evaluation Award
|
| Scott Reed,
Oregon State University,
scott.reed@oregonstate.edu
|
|
This award is given for administrative or program leadership toward outstanding organizational, policy and resource contributions to Extension evaluation. The panelist is currently a vice-provost of outreach and engagement at major land-grant university and will be nominated for this award in 2008. As vice-provost, this panelist also serves as the director of the Extension Service. The unique structure and emphasis of Extension in his state system has provided a rich context for the creative development and support of Extension Evaluation. The panelist will provide insight into the important ways that Extension administration can support and value evaluation efforts. And explore ways such efforts are instrumental in fulfilling the land grant mission of teaching, research and outreach.
| |
|
Leading the Way: Leadership and Service to the Extension Evaluation Profession
|
| Michael Lambur,
Virginia Polytechnic Institute and State University,
lamburmt@vt.edu
|
|
This award is given for leadership and service in creating networks and gaining visibility for Extension evaluators. A recent award winner in this category, this panelist will examine the evolution of Extension evaluation as a profession over the twenty-two years that Extension Education Evaluation Topical Interest Group (EEE TIG) has formally been in existence. The panelist will comment on the growth Extension evaluation as a profession through an exploration of the types and numbers of Extension evaluators, the role of the EEE TIG, professional development opportunities including AEA sessions, the role of our federal partner, and the structure for program evaluation within Extension organizations. Based on this, the presenter will provide perspectives on the future of Extension evaluation as a profession.
| |
|
Building Evaluation Capacity That Lasts: Sustained Excellence in Extension Evaluation
|
| Roger Rennekamp,
Oregon State University,
roger.rennekamp@oregonstate.edu
|
|
This award is given for conceptual, methodological and evaluation accomplishments making outstanding contributions to Extension evaluation. The panelist has a long career in Extension. His contributions to sustaining evaluation are well-recognized across the country, and he is frequently acknowledged by his peers as a leader in Extension evaluation. The panelist spent the majority of his career at one university, and during that time was instrumental in the evolution of evaluation throughout the organization. His work includes all aspects of developing a positive evaluation culture- from building individual capacity to implementing systems, processes and tools to enhance and sustain evaluation practice. Letters in support of this panelist's nomination for the sustained excellence award universally commented on the significant impact he made in changing the practice of evaluation. The panelist will share his knowledge and understanding of creating change within in a system that allows for evaluation success and sustainability over time.
| |
|
Session Title: Community Organizing Evaluation: Crossing the Next Frontier
|
|
Panel Session 522 to be held in Mineral Hall Section C on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Advocacy and Policy Change TIG
|
| Chair(s): |
| Astrid Hendricks Smith,
The California Endowment,
ahendricks@calendow.org
|
| Abstract:
Community organizing is an area of advocacy work where there is considerable demand for more evaluation information and where there are new attempts to fill that void. This session is designed to provide the audience with an overview of numerous community organizing evaluation resources in existence, key themes in those resources, related tools that are available to the public and new models being tested. Case examples of recent evaluations done of community organizing work, as well as efforts to build organizations' capacities to do that work, will be presented and will bring several theories and models to life. The panel will explore the effectiveness of the tools and models used in the examples and the unresolved challenges faced. A group discussion with the audience will follow.
|
|
Resources and Themes for Community Organizing Evaluation
|
| Susan Hoechstetter,
Alliance for Justice,
shoech@afj.org
|
|
There are a variety of different theories used and approaches taken by community organizing groups, evaluators, and funders in order to evaluate community organizing work. In an effort to identify resources for this work in a centrally available location, Alliance for Justice spent six months compiling electronic and hard copy evaluation models for an electronic compendium available to all. A review of resources demonstrated that some groups focus most heavily on empowering the community, others on achieving policy change, and yet others have a focus that combines these two areas and/or incorporate others. This presentation will provide an overview of the resources for informational purposes and will, at the same time, provide a basis for critiquing the evaluation examples in the ensuing presentations.
|
|
|
Evaluating Local and Statewide Community Organizing Efforts - What Does It Look Like?
|
| Justin Louie,
Blueprint Research and Design Inc,
justin@blueprint.com
|
|
Moving beyond the theory, what does evaluation of community organizing look like on the ground? How do you engage organizers in developing their theories of change? What do you measure in community organizing and how do you measure it? How do you build organizers' own capacity to do evaluation? How do you feed back evaluation information to help improve organizing? We will present examples and lessons learned from our work with community organizers across the country, evaluating issues ranging from education reform to healthcare access to smart growth. We will discuss the local, regional, and statewide organizing efforts of individual organizing groups and organizing collaboratives, and provide real-world guidance to evaluators engaging or interested in evaluating community organizing.
| |
|
Developing an Approach to Evaluating Community Engagement Advocacy Efforts
|
| Jane Reisman,
Organizational Research Services,
jreisman@organizationalresearch.com
|
| Sarah Stachowiak,
Organizational Research Services,
sstachowiak@orginaizationalresearch.com
|
| Kendall Guthrie,
Bill and Melinda Gates Foundation,
kendall.guthrie@gatesfoundation.org
|
|
The Bill & Melinda Gates Foundation funds three community organizations to support a community engagement strategy in concert with high school redesign efforts in Austin, Texas. A key part of the learning for the Gates Foundation, as well as the Austin community partners and school district, has been an effort to develop theory of change models for each partner and for the effort overall based on the Foundation's comprehensive community engagement framework. Organizational Research Services (ORS) has been serving as an evaluation consultant to facilitate the theory of change development and build the capacity for designing tools, and collecting, analyzing and using data to evaluate the shorter-term advocacy changes identified in their models. This presentation will share the overall community engagement model and use the Austin experience as a case illustration to share models and tools and reflect on how this process has been working.
| |
|
Session Title: Services, Quality, and Outcomes in Mental Health and Substance Use Evaluations
|
|
Multipaper Session 523 to be held in Mineral Hall Section D on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG
|
| Chair(s): |
| Phillip Graham,
RTI International,
pgraham@rti.org
|
|
Does Quality Matter Too?: Evaluating A Tool to Assess Quality of Implementation in a Parenting-Based Program to Improve Children’s Postdivorce Mental Health
|
| Presenter(s):
|
| Cady Berkel,
Arizona State University,
cady.berkel@asu.edu
|
| Sarah Jones,
Arizona State University,
sarahjp@asu.edu
|
| Melissa Hagan,
Arizona State University,
melissa.hagan@asu.edu
|
| Irwin Sandler,
Arizona State University,
irwin.sandler@asu.edu
|
| Sharlene Wolchik,
Arizona State University,
wolchik@asu.edu
|
| Tim Ayers,
Arizona State University,
tim.ayers@asu.edu
|
| Erin Schoenfelder,
Arizona State University,
erin.schoenfelder@asu.edu
|
| Anne Mauricio,
Arizona State University,
anne.mauricio@asu.edu
|
| Abstract:
While many contend program quality increases participants’ engagement, we lack standard measures for evaluating implementation, beyond fidelity checklists, which evaluate implementation quantity, but not quality. We present reliability and predictive utility of the QIM, an implementation quality measure rating seven domains of implementation (including fostering group cohesion, personalizing material). Data comes from video recordings of New Beginnings, a parenting program to improve children’s post divorce mental health, lead by two co-facilitators. For each session, we collect:
3 independently-rated QIMs (2 facilitators, 1 overall)
2 QIM self-assessments (1 per facilitator)
Fidelity
Engagement (participant attendance, homework completion)
Using MLM with our nested, repeated measures data, we evaluate the following:
Can adequate interrater reliability be achieved on the QIM?
How do independent QIM ratings relate to fidelity, QIM self-assessments, and participant engagement?
For independent ratings, how does an overall QIM score compare to a combination of the two facilitator QIM scores as predictors of engagement?
|
|
An Assessment of Substance Abuse Training and Technical Assistance Needs in Frontier States: Implications for the Adoption of Evidence-Based Practices
|
| Presenter(s):
|
| Joyce Hartje,
University of Nevada Reno,
jhartje@casat.org
|
| Nancy Roget,
University of Nevada Reno,
nroget@casat.org
|
| Wendy Woods,
University of Nevada Reno,
wwoods@casat.org
|
| Joanne Brosh,
University of Nevada Reno,
jbrosh@casat.org
|
| Abstract:
The purpose of this project was to assess the training and technical assistance needs of substance abuse and allied health professionals in the six-state Mountain West Addiction Technology Transfer Center (MWATTC) region, and to examine their attitudes and level of interest in adopting specific evidence-based practices (EBPs). A web-based survey design and a snowball sampling procedure were used to facilitate collecting data from a representative sample in a short amount of time. Results from the needs assessment were used at the MWATTC Advisory Board meeting to guide state-level discussions regarding the training and technical assistance activities needed by substance abuse treatment professionals in their state. By getting stakeholder participation in this process, the MWATTC is able to more effectively implement stakeholder-driven training and technical assistance activities, thereby increasing the likelihood of events being relevant to participants, as well as enhancing event attendance and attitudes towards implementing EBPs.
|
|
Evaluating Findings from Nebraska's State Incentive Grant (SIG)
|
| Presenter(s):
|
| Phillip Graham,
RTI International,
pgraham@rti.org
|
| Jason Williams,
RTI International,
jawilliams@rti.org
|
| Abstract:
The CSAP State Incentive Grant (SIG) program was the Center’s flagship initiative to reduce substance use among youth age 12-17 through the promotion and adoption of evidence-based prevention strategies. From 2004-2007, the State of Nebraska funded 27community coalitions to implement and evaluate comprehensive prevention strategies to reduce substance use among its youth. However, evaluation of this initiative has been hampered by several methodological challenges (e.g., lack of sound comparison data).
Longitudinal Growth Modeling (LGM) will be applied to three waves of data collected in 2003, 2005, and 2007 to determine the overall impact of the SIG initiative. Results from latent profile analysis (LPA) will also be presented to identify key underlying characteristics of the most effective community coalitions. Findings will provide important insight regarding characteristics associated with program effectiveness in real world settings and will provide a critical test of federal policies requiring implementation of evidence-based strategies via community coalitions.
|
|
Linking Services to Outcomes in the Evaluation of San Francisco's Mental Health Services Act Full Service Partnerships
|
| Presenter(s):
|
| Thomas Bleecker,
Research, Evaluation, and Quality Management,
tom.bleecker@sfdph.org
|
| Deborah Sherwood,
Research, Evaluation, and Quality Management,
deborah.sherwood@sfdph.org
|
| Abstract:
San Francisco’s implementation of Mental Health Services Act Full Service Partnerships (MHSA FSPs) targets underserved mental health and dual diagnosis clients. The treatment approach relies on an intensive case management model along with a commitment to do “whatever it takes” to support clients’ recovery and integration into the community.
Client outcomes are tracked through the Data Collection and Reporting (DCR) system, a web-based application that captures initial assessments in life domains (e.g., residential, employment), changes in those domains (Key Events), and quarterly summaries. FSP mental health services are tracked in the county billing system.
This presentation demonstrates our evaluation approach, which ties together the processes of service provision (from the billing system) to client outcomes (from the DCR). The presentation will demonstrate how differences in FSP client populations, treatment models, and outcomes were visually presented to program managers and staff, and how those results were used in ongoing program operations.
|
| | | |
|
Session Title: It's About the Relationship! Crosscutting Organizational policies That support Program Evaluation and Institutional Learning in Social Services.
|
|
Multipaper Session 525 to be held in Mineral Hall Section F on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Social Work TIG
|
| Chair(s): |
| Benjamin Kerman,
Annie E Casey Foundation,
bkerman@caseyfamilyservices.org
|
| Discussant(s): |
| Diane Purvin,
Annie E Casey Foundation,
dpurvin@caseyfamilyservices.org
|
| Abstract:
Social Services evaluation policy can be conceptualized as a series of formal and informal contracts developed among members of an organization to learn from its collective experience. The contract metaphor points to the intrinsic role of relationship in using evaluation effectively. Shared principles and goals underscore the contract parameters, such as the direct connection between evaluation and shared mission, valuing the commitment and capacity to apply feedback, the delineation of roles, confidentiality and the protection of participant rights. This session features three papers describing diverse evaluation activities with social services staff and managers illustrating how evaluation policies and activities manifest explicit and implicit contracts. The three projects involve the routine review of performance indicators, an implementation evaluation of a program redesign, and an outcome evaluation using a longitudinal follow-up. Evaluation policies need to adhere to common principles while remaining flexible enough to allow for adaptation to questions and context.
|
|
The Role of Evaluation Policies in Integrating Agency-wide Performance Measurement
|
| Ellen Ross,
Annie E Casey Foundation,
eross@caseyfamilyservices.org
|
|
This paper examines how evaluation policy can facilitate agency wide performance measurement. In 2002, Casey Family Services adopted results-based accountability across the agency's core goals. This framework provides a simple, disciplined rubric, common language, and common ground for implementing new ideas. Operational tasks included how integrating evaluation into practice, recruiting and engaging social workers from the onset, and institutionalizing the framework. By involving all organizational levels in routine indicator review, leadership and staff were alerted to areas of concern and accomplishment. Critical in this process was the involvement and buy-in from senior management, as well as collaboration across research, field operations, information technology, and direct care staff. Challenges included: timing, funding, buy-in from stakeholders, and organizational culture. Strategies used to address these challenges included building departmental relationships, development of policies and infrastructure to embed the data collection and review in ongoing activities and structures, and attending to diverse stakeholder's needs.
|
|
Policies, Pressures, and Expectations: Managing Relationships During a Multifaceted Implementation Evaluation
|
| Judy Lee,
Casey Family Services,
judymlee@msn.com
|
| Benjamin Kerman,
Annie E Casey Foundation,
bkerman@caseyfamilyservices.org
|
|
Casey's Move to Greater Permanency initiative reflects an intensified focus on providing permanency planning and foster care services to youth in effort to increase legal and relational permanence. While some program refinements were similar across service sites (e.g., use of Permanency Teaming), additional refinements were site specific. As expectations for a swift and complete integration of Permanency Teaming practice increased among agency administrators, pressures also mounted among field leaders and staff who were experiencing increased uncertainty given the major shifts in practice and procedures. Amidst these changing demands and social worker anxiety, the evaluators used an array of existing policies to conduct a multifaceted implementation and short-term outcome evaluation. Close coordination between internal evaluators and the reliance on external consultants helped ensure that the relationships between evaluator and evaluand were appropriate and corresponded optimally to the particular evaluation question and component.
|
|
Follow-up Evaluation and the Long View: Preparation for the Casey Longitudinal Study of Foster Youth Development
|
| Gretta Cushing,
Casey Family Services,
gcushing@caseyfamilyservices.org
|
| Diane Purvin,
Annie E Casey Foundation,
dpurvin@caseyfamilyservices.org
|
|
Longitudinal and follow-up projects may require additional policies and structures that go beyond those of either performance measurement or concurrent program evaluation designs. This paper will discuss how researchers and social workers collaborated in the development of a longitudinal study aiming to provide the agency leadership with an unbiased view of outcomes, as well as to examine applied developmental questions concerning vulnerable youth and their families. All youth who were placed in a foster home after December 31 1993 and spent one year with Casey are included. Data collection includes record reviews and young adult interviews at ages 19, 22 and 25. A variety of agency policies and structures are essential to the conduct of longitudinal service outcome research, including location of former clients and response to service requests, internal communication and confidentiality, and handling reports of maltreatment, as well as the time investment inherent in longitudinal research.
|
|
Session Title: Collaborative Evaluations: Successes, Challenges, and Lessons Learned
|
|
Multipaper Session 526 to be held in Mineral Hall Section G on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Chair(s): |
| Susan Hibbard,
University of South Florida,
hibbard@coedu.usf.edu
|
|
An Accountability Evaluation Supportive of Stakeholder Learning
|
| Presenter(s):
|
| Cheryl Poth,
University of Alberta,
2cnp@queensu.ca
|
| Abstract:
Using the two-year evaluation of the Queen’s University Inter-Professional Patient-centred Education Direction (QUIPPED) program as a case study, this paper describes a participatory approach supportive of learning and ongoing program development while meeting the evaluation’s primary purpose for accountability to government funders. Our current understandings of the participatory approach describe it as useful to support individual and organizational learning. However, studies of evaluation use have yet to examine specifically how evaluators create opportunities for stakeholders to learn and at the same time, how evaluators produce outcomes that are useful for external accountability purposes. This paper reports the analysis of evaluator behaviour as it endorsed, modified, and extended current notions of the participatory approach.
|
|
Empowerment evaluation: Did The Approach Work with a National School Breakfast Program In Australia?
|
| Presenter(s):
|
| Wayne Miller,
Avondale College,
wayne.miller@avondale.edu.au
|
| Heather Yeatman,
University of Wollongong,
hyeatman@uow.edu.au
|
| Abstract:
The empowerment evaluation approach was used in an evaluation of a national school breakfast program in Australia. Preliminary findings have been reported previously (Miller and Lennie, 2005). The purpose of this presentation is to report key evaluation outcomes associated with employing the empowerment evaluation approach rather than program outcomes as a result of the evaluation.
Evaluation outcomes were found to be of two types - those that are clearly linked to the steps of the empowerment evaluation approach, and others linked more generally to the principles of empowerment evaluation. The three steps provided a simple, progressive framework for the evaluation. This built early confidence in the approach which contributed to the achievement of significant evaluation and program outcomes. However, participants reported both alignment and misalignment with the ten principles.
The presentation will discuss findings in relation to the application of the empowerment evaluation approach. Strengths and limitations with respect to its use with a community-based, dispersed and diverse program will be highlighted. Some signposts will be provided for those looking to use empowerment evaluation in similar applications.
|
|
Strategies for Using Stakeholder Input to Improve Evaluation Designs
|
| Presenter(s):
|
| Miles McNall,
Michigan State University,
mcnall@msu.edu
|
| Lauren Lichty,
Michigan State University,
lichtyla@msu.edu
|
| Abstract:
Participatory and collaborative evaluation approaches call for meaningful involvement of stakeholders in nearly every aspect of evaluation, including evaluation design. When stakeholders participate in decisions about evaluation design, they impart critical information about the context in which the evaluation will be implemented and their information needs. In the absence of such information, evaluations are unlikely to meet standards of feasibility, utility, or accuracy. In our experience, discussions with stakeholders about evaluation design surface issues such as the political and logistical feasibility of various methods and the kinds of evidence that will have credibility for various stakeholder groups. In this paper, we present a framework to assist evaluators and stakeholders in considering these issues and discussing their implications for evaluation design. The framework is designed to promote evaluation designs that meet stakeholders information needs (utility) and are responsive to local realities (feasibility) while upholding methodological integrity (accuracy).
|
|
Walking a Fine Line: The Role of the Evaluator in Monitoring Program Implementation
|
| Presenter(s):
|
| Rebeca Diaz,
WestEd,
rdiaz@wested.org
|
| Abstract:
This presentation will discuss the role of evaluators in monitoring the implementation of programs in school districts. The impetus for this topic comes from an evaluation study of a federal Smaller Learning Communities (SLC) grant implemented in a school district in northern California. The implementation of the grant was virtually void of district leadership, and as a result, the program contributed to negative student outcomes. The evaluator, who was subcontracted by the district, found this experience to be particularly challenging as she witnessed low socio-economic, ethnic minority students at a low performing high school expressing feelings of inferiority as a result of their participation in the program. She began to question her own responsibility in the situation. This presentation will explore questions such as: What is the responsibility of the evaluator in the implementation of school programs? How does one conduct a fair and meaningful evaluation while respecting both the client and the goals of the program?
|
|
Collaborative Evaluation Strategies that Promote Culturally-Responsive Evaluations
|
| Presenter(s):
|
| Monifa Beverly,
University of Central Florida,
mbeverly@mail.ucf.edu
|
| Karyl Askew,
University of North Carolina Chapel Hill,
karyls@email.unc.edu
|
| Michelle Jay,
University of South Carolina,
jaym@gwm.sc.edu
|
| Abstract:
Collaborative evaluation is an orientation to evaluation that promotes culturally-responsive practice. The goal of both collaborative evaluation and culturally-responsive approaches is to use an understanding of the cultural context to develop and execute evaluations that accurately capture program impact and provide maximum benefit to stakeholders. This presentation provides three examples of evaluations that demonstrate the use of four collaborative evaluation techniques resulting in practices that were culturally-responsive: collaborative planning, evaluation technical assistance, evaluation capacity building, and evaluation fairs. The evaluands include single-site and multi-site programs focused on delivery of services to ethnically diverse populations. The presenters, five individuals trained in the collaborative evaluation approach, offer critical reflection on using collaborative evaluation methods to promote culturally responsive practices. The evaluators will present successes, challenges, and practical recommendations for using collaborative evaluation techniques that can be widely applied to enhance the cultural-responsiveness of evaluations and evaluators.
|
| | | | |
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Issues, Challenges and Solutions of Evaluating International Scholars/Participants Training in the United States |
|
Roundtable Presentation 527 to be held in the Slate Room on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Multiethnic Issues in Evaluation TIG
and the International and Cross-cultural Evaluation TIG
|
| Presenter(s):
|
| Deborah Oh,
California State University at Los Angeles,
doh2@calstatela.edu
|
| Abstract:
Twenty-six (26) university senior administrators from Guangdong University of Foreign Studies completed the Guangdong University of Foreign Studies Senior Administrators Training Program, a comprehensive institute at California State University, Los Angeles (CSULA). The scope of this three week institute held from July 30, 2007 to August 18, 2007 was two-fold: focused training on various topics, conducted primarily by faculty at CSULA, and fieldwork visits to various university campuses here in Southern California and in the East Coast, thereby featuring a hands-on intensive orientation, and familiarization of current organizational structures and governance, curriculum, student affairs, development, infrastructure support, and partnerships of institutions of higher learning in the US. The issues and challenges and possible solutions related to evaluating participants in a cultural sensitive manner, namely, the manner in which the surveys were conducted, the survey results are understood, and the manner in which the surveys uncovered unanticipated answers which were not part of the program or evaluation design will be discussed.
|
| Roundtable Rotation II:
Evaluating the Impacts of Bolsa Escola Programs on Student Performance in Brazil |
|
Roundtable Presentation 527 to be held in the Slate Room on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Multiethnic Issues in Evaluation TIG
and the International and Cross-cultural Evaluation TIG
|
| Presenter(s):
|
| Amy Lora,
Florida International University,
alora001@fiu.edu
|
| Abbas Tashakkori,
Florida International University,
tashakko@fiu.edu
|
| Abstract:
Beginning in 1997, conditional cash transfer programs started throughout Brazil, giving children the opportunity to choose school over work. Currently, these programs (known as Bolsa Escola) support over eight million children through the distribution of R$127.2 million to 5,512 municipalities per month. Thus far, evaluations and investigations of Bolsa Escola programs have focused on enrollment, attendance, and dropout rates, indicating that the programs have been largely successful in achieving these goals. However, studies have largely neglected to investigate, or have failed to demonstrate, impacts on students' academic performance. This paper presents the results of an evaluation project, examining the possible effects of the program on student performance in Mathematics and Language Arts in grades 4 and 8.
|
|
Session Title: Conducting Evaluation With a Focus on Stakeholders
|
|
Multipaper Session 528 to be held in the Agate Room Section B on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Graduate Student and New Evaluator TIG
|
| Chair(s): |
| S Lory Hovsepian,
University of Montreal,
sarine.lory.hovsepian@umontreal.ca
|
|
Crossing Country and Cultural Boundaries for Work
|
| Presenter(s):
|
| Lennise Baptiste,
Kent State University,
lbaptist@kent.edu
|
| Abstract:
The development co-operation strategy adopted in 1996 by the Organization for Economic Cooperation and Development, an international organization focused on economic and social issues. This strategy shaped two partnerships formed between one European and two Caribbean non-governmental organizations. The lessons learned from the evaluation of the partnerships centered on, cross-cultural relations, access, project plan logistics, and of negotiation of the Terms of Reference which required qualitative methodology. One key finding was that routine tasks can become monumental when evaluators are in unfamiliar terrain even though language was not a barrier. This presentation can inform evaluators who undertake projects spanning cultural and country boundaries.
|
|
Exploring Connections Between Programs, Their Evaluation and Those Involved: An Example of the Implementation of a Community-based Pilot Program for Women Living with HIV
|
| Presenter(s):
|
| S Lory Hovsepian,
University of Montreal,
sarine.lory.hovsepian@umontreal.ca
|
| Astrid Brousselle,
University of Montreal,
astrid.brousselle@umontreal.ca
|
| Joanne Otis,
University of Quebec Montreal,
otis.joanne@uqam.ca
|
| Abstract:
Background: Community-university collaborators developed and implemented “Pouvoir Partager/Pouvoirs Partagés”, a community-based program (CBP) helping women living with HIV in Montreal, Canada to manage the disclosure of their HIV status. Process and outcome evaluations were undertaken in the context of a formative evaluation. Objective: We explore the interconnections between the CBP, its evaluation and participatory processes. Methods: Interviews with stakeholders, focus groups with program participants and committee meeting observations were undertaken. Results: Extent of participatory evaluation varied according to stakeholder expertise and interests, evaluation type and other practical considerations. The evaluation process seemed to facilitate participation in program development and the program, and aid its continuation. Although evaluation contributed to improving the program and its implementation, some perceived it as strain, restricting implementation flexibility and limiting program impact. Conclusion: Planning, implementing and evaluating CBPs require that stakeholders address issues emerging from the interrelations between the program, its evaluation and participatory processes.
|
|
Evaluation Decision Making as Authoritative Allocation of Value
|
| Presenter(s):
|
| A Rae Clementz,
University of Illinois,
clementz@uiuc.edu
|
| Jeremiah Johnson,
University of Illinois Urbana-Champaign,
jeremiahmatthewjohnson@yahoo.com
|
| Abstract:
Evaluation resources are limited. There is rarely enough time, funding, people, etc., to accomplish everything we might need and want to. David Easton (1954) defined policy as "an authoritative allocation of value." In this same way, the decisions made during the course of evaluation regarding evaluation design and implementation reflect value stances and are actualized through the allocation of the limited evaluation resources at our disposal. Thus, our practical and functional decisions in evaluation practice carry value implications. Evaluation decisions are never made solely by the evaluator. They are made in context and reciprocally with the group of individuals involved in conducting the evaluation. This session discusses these value decisions and their role in evaluation practice. Separate from method or approach, this framework is intended to engage the practical implications of holding various value stances in the conduct of an evaluation.
|
|
Evaluability Assessment: What Makes a Program Ready for Evaluation?
|
| Presenter(s):
|
| Gargi Bhattacharya,
Southern Illinois University at Carbondale,
gargi@siu.edu
|
| Meghan Lowery,
Southern Illinois University at Carbondale,
meghanlowery@gmail.com
|
| Alen Avdic,
Southern Illinois University at Carbondale,
alen@siu.edu
|
| Abstract:
Evaluation of program impact should guide the development of programs; however, in real-life organizational situations, clients are often unaware of this essential program development process. Professional evaluators often encounter clients who are unsure of how they want to evaluate a program or how they want to measure program impact. Applied Research Consultants (ARC), a vertical practicum built within the doctoral program of Applied Psychology at Southern Illinois University at Carbondale, is a graduate student-run consulting firm. ARC conducts professional evaluations and provides other services (e.g., job analysis, survey research, data analysis etc.) to clients in and outside the University community. Different programs evaluated by ARC were analyzed and evaluators were interviewed to identify components of programs that made them easier to evaluate or become evaluation-ready.
|
| | | |
|
Session Title: Evaluation Policy and Practice: Its Impact on African American Women's Sexual and Reproductive Health
|
|
Multipaper Session 529 to be held in the Agate Room Section C on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Feminist Issues in Evaluation TIG
|
| Chair(s): |
| Jenny L Jones,
Virginia Commonwealth University,
jljones2@vcu.edu
|
| Abstract:
This multipaper presentation will include three studies that highlight the need for sound evaluation policies which influence practice. Each paper focuses on African American women and health disparities as they relate to HIV/AIDS as well as adverse birth outcomes. Quantitative and qualitative findings will be presented for Young Black College Women: HIV/AIDS Secreted 'At Risk' Group and An Exploration of Factors That Influence the Sexual Decisions of Young Black Women: What Are They Really Thinking respectively. Exploring Adverse Birth Outcomes Among African American Women with Means: Revealing a New Population will include a conceptual discussion regarding the adverse birth outcomes of African American women. Topics related to black feminist as well as gender and power theories will be discussed.
|
|
HIV Risk Behaviors among a Sample of Young Black College Women
|
| Binta Alleyne,
University of Illinois Chicago,
balleyne@uic.edu
|
|
The relationship between certain factors associated with the theory of gender and power including: sexual relationships, condom use self-efficacy, and substance use, to young Black college women's condom use was studied. A sample of 189 young Black women between the ages of 18 and 24 from a Historically Black University in the South was selected to complete a HIV/AIDS Risk Behavior Survey.
An ordinal logistic regression revealed that self-efficacy and type of sexual relationship were positively associated with condom use, whereas substance use was negatively associated with condom use. This discussion will focus on the need to identify young Black women in college as a non-disenfranchised 'at risk' group, understand how the type of sexual relationships in which young Black women are involved influence their condom usage, and the need to evaluate current HIV prevention practice and policy with this population.
|
|
An Exploration of Factors That Influence the Sexual Decisions of Young Black Women: What are they Really Thinking?
|
| Aisha Tucker-Brown,
University of Georgia,
aktb@bellsouth.net
|
|
This study explored the factors that influence the decision making of 25 African American women between the ages of 18 and 25. They were purposefully selected and interviewed. Interviews and focus groups served as the sole source of data for this study. This was a basic interpretive study and a qualitative research design was used to explore the factors that influence sexual decision making among these participants. Data were analyzed using the constant comparative method.
An analysis of the data revealed that participants were in fact quite knowledgeable about HIV/AIDS and prevention strategies. This raises the question, what are we evaluating regarding HIV prevention programs? Are we allowing practice to dictate policy or vice versus. Conclusions also address the intersection of race, class, and gender and its relevance to the development of sound evaluation policy and practice with regards to HIV prevention initiatives.
|
|
Exploring Adverse Birth Outcomes Among African American Women with Means: Revealing a New Population
|
| Kimberly Farris,
University of Illinois at Chicago,
kimberlydfarris@gmail.com
|
|
Health researchers have long noted racial disparities in birth outcomes, including pre-term delivery, low birth weight, and infant mortality. Recent literature has shown that African American women in higher SES categories are more likely to have adverse birth outcomes including low birth weight, pre-term deliveries, and fetal death. While researchers have focused on socioeconomic and behavioral explanations, few have examined psychosocial factors among this population.
The issue warrants further investigation through the initiation of dialogue to identify potential methods for researchers/evaluators to reach women not traditionally considered as disenfranchised in terms of SES factors but still experience adverse outcomes. Furthermore, the discussion will focus on conceptualizing potential psychosocial factors that may predict adverse outcomes among this population, developing a model examining the extent to which these factors affect birth outcomes, and exploring methods on recruitment strategies of participants and issues related to understanding the role of evaluation practice and policy.
|
|
Session Title: Evaluating Early Childhood Programs: Methodological Challenges and Lessons Learned
|
|
Multipaper Session 530 to be held in the Granite Room Section A on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Barbara J Helms,
Education Development Center Inc,
bhelms@edc.org
|
|
Avoiding the Ethical and Methodological Deficiencies Found in Widely Acclaimed Early Childhood Education Studies
|
| Presenter(s):
|
| Morris Lai,
University of Hawaii Manoa,
lai@hawaii.edu
|
| Susan York,
University of Hawaii Manoa,
yorks@hawaii.edu
|
| Abstract:
Although many have praised the High/Scope Perry Preschool and the Abecedarian Preschool studies, we seriously question whether such studies strongly generalize today and meet prevailing ethical, evaluation, and other professional standards. We are disturbed by the devastating fate of the control-group children; we have also noted other types of inappropriateness in published studies on the effects of prekindergarten early childhood education (ECE).
After delineating our concerns about several research or evaluation studies in the ECE literature, we develop recommendations that address the numerous inadequacies. Evaluations of ECE initiatives should (a) accommodate allowing all children to participate in the main treatment if their families so choose, (b) include nonformal/noncenter-based programs, (c) use data-collection instruments designed to specifically measure children’s readiness for kindergarten, (d) regard participant feedback as primary evaluation data, (e) analyze the data focusing on effect size rather than on statistical significance, and (f) ensure cultural appropriateness.
|
|
Pre-Kindergarten Teachers' Literacy Knowledge: Instrument Development and Lessons Learned
|
| Presenter(s):
|
| Melissa Chapman,
University of Iowa,
melissa-chapman@uiowa.edu
|
| Abstract:
This proposal is contextualized in this evaluator’s experiences working as an external consultant for an Early Reading First project and the problems encountered when federal site visitors requested that project teachers be evaluated on their knowledge of the mandated early reading areas: oral language (expressive and receptive), phonological awareness, print awareness, and alphabet knowledge (Whitehurst & Lonigan, 1998; USDOE, 2006). Given the absence of widely available, technically adequate assessments that would serve this purpose, an open-ended instrument was first created and used. This proposal is focused on improving the technical adequacy of a Language and Literacy Assessment (LLA) by conducting cognitive interviews (Willis, 1999, 2005) with expert teachers and researchers and then assessing a large sample of pre-K teachers in a Midwestern state. Further item level statistics and other technical adequacy issues will be examined following the statewide distribution.
|
|
The Use of Parent and Teacher Task Scenarios in the Evaluation of a Literacy Website
|
| Presenter(s):
|
| Margaret Vaughan,
Goodman Research Group Inc,
vaughan@grginc.com
|
| Margaret Tiedemann,
Goodman Research Group Inc,
tiedemann@grginc.com
|
| Abstract:
The paper is based on a comparative study, one component of a comprehensive website evaluation. The study involved recruiting parent and teacher participants to examine the strengths and weaknesses of a targeted early childhood literacy site in comparison to its primary competitors. A unique aspect of the evaluation was the inclusion of hypothetical task scenarios. A task scenario, often part of usability testing, is a representation of work a user would likely perform while visiting a website. These task scenarios were designed to gain feedback from the website’s primary audiences and to be meaningful to their perspectives and roles. The scenarios provided participants with a child-based situation to consider rather than having them follow a step-by-step approach. The paper will report on methodology and findings with a focus on parents and teachers overall impressions of the website as well as specific details about their experiences while completing the task.
|
|
Development of an Interview Protocol for Families of English Language Learners in Head Start Programs
|
| Presenter(s):
|
| Marc Winokur,
Colorado State University,
marc.winokur@colostate.edu
|
| Victoria Buchan,
Colorado State University,
buchan@cahs.colostate.edu
|
| Abstract:
The Head Start English Language Learner Project (HELLP) is one of eight Head Start Innovation and Improvement Projects funded by the U.S. Department of Health and Human Services to strengthen Head Start programs nationwide. Specifically, HELLP is focused on supporting Head Start programs to better serve English language learner (ELL) children and their families. Through evidence-based curricula, hands-on training, and programmatic assistance, HELLP is designed to build the capacity for Head Start personnel and families to assist ELL children in their literacy development. To evaluate this ambitious initiative, a comprehensive program evaluation was fashioned. A major component of the evaluation was a parent-family interview designed to explore the level of engagement of ELL parents in supporting their children’s literacy development in the home and at school. The proposed paper will present the unique approach used in the protocol development and administration process, while reporting on findings from the interviews.
|
| | | |
|
Session Title: HIV/AIDS Prevention at the Community Level: Innovative Evaluation Approaches
|
|
Multipaper Session 531 to be held in the Granite Room Section B on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Health Evaluation TIG
|
| Chair(s): |
| Ann Zukoski,
Oregon State University,
ann.zukoski@oregonstate.edu
|
|
Evaluating the Effectiveness of the Baby and Me Tobacco Free Program
|
| Presenter(s):
|
| Anne Gadomski,
Bassett Research Institute Computing Center,
anne.gadomski@bassett.org
|
| Laurie Adams,
Tri County Tobacco Control Program,
lauriea@windstream.net
|
| Nicole Krupa,
Bassett Research Institute Computing Center,
nicole.krupa@bassett.org
|
| Patrick Burdick,
Bassett Research Institute Computing Center,
patrick.burdick@bassett.org
|
| Nancy Tallman,
Bassett Research Institute Computing Center,
nancy.tallman@bassett.org
|
| Nancy Weissflog,
Bassett Research Institute Computing Center,
nancy.weissflog@bassett.org
|
| Paul Jenkins,
Bassett Research Institute Computing Center,
paul.jenkins@bassett.org
|
| Abstract:
The Baby & Me Tobacco Free Program is a multi-pronged intervention that includes cessation counseling, biomarker feedback and an incentive program to facilitate prenatal and post-partum smoking cessation. The intervention includes four face-to-face contacts prenatally with a cessation specialist who does counseling and carbon monoxide (CO) testing. For 1 year post-partum, mothers are CO tested every 3 to 4 weeks and, if negative, receive a diaper voucher. Outcomes include biochemically validated monthly abstinence rates and self reported quit rate at 12 months post-partum ascertained by telephone interview. To date, 22 sites (WIC offices and obstetric clinics) in 9 counties in New York State have enrolled 400 women. Preliminary results include 289 women who have completed, on average, between 5 and 6 (mean=5.63) follow-up tests (CO and/or saliva), with a mean passing/quit rate of 93.8%. Among the 45 women who have completed 12 follow-up tests, the mean passing/quit rate is 96.9%.
|
|
Examination of Peer Advocates in HIV/AIDS Care and Treatment
|
| Presenter(s):
|
| Marcia Dutcher,
Kansas City Free Health Clinic,
marciad@kcfree.org
|
| Brenda Loscher Hudson,
Kansas City Free Health Clinic,
brendal@kcfree.org
|
| Abstract:
The Peer Education Training Site (PETS) project is a Health Resources and Services Administration (HRSA) multiyear funded project to increase the number of trained HIV peer educators and the number of organizations with the capacity to utilize peers in their HIV service delivery. Three sites located in California, Missouri, and New York have developed peer and organization training programs tailored to the assets, needs, and preferences of their individual target populations. Training sessions have been offered since 2006. A cross-site quantitative longitudinal design is used to measure research questions for peers and organizations while a qualitative longitudinal design is used to address research questions for clients of peer advocates. The paper will discuss the research methodologies to evaluate the emerging role of peer advocates in HIV/AIDS care and treatment and how the implementation of a multi-dimensional evaluation design utilized in non-profit health care settings impact evaluation practice and theory.
|
|
Evaluating a Community Level HIV and Substance Use Prevention Program: The Fortaleciendo la Comunidad Project
|
| Presenter(s):
|
| Kathryn DeRoche,
Mental Health Center of Denver,
kathryn.deroche@mhcd.org
|
| Antonio Olmos,
Mental Health Center of Denver,
antonio.olmos@mhcd.org
|
| Lydia Prado,
Mental Health Center of Denver,
lydia.prado@mhcd.org
|
| Hollie Granato,
University of Colorado Denver,
hollie.granato@email.cudenver.edu
|
| Shara Shaw,
University of Northern Colorado,
sshaw_dance@yahoo.com
|
| Abstract:
The Fortaleciendo la Comunidad project is a HIV, sexual transmitted infections, and substance abuse prevention program designed for and by the Latino community in West Denver. The project includes four programs designed for selected sub-populations in the community deem to have the highest need for prevention including: adolescents, parents, heterosexual adult females, and a community outreach program for injecting drug users and men who have sex with men. In the evaluation, we have determined the progress of the four individual prevention programs, while also evaluating the cumulative affects of the project within the community. The presentation will discuss the evaluation results, the process of increasing evaluation capacity of service organizations within the community, and the process of conducting a community-wide, culturally competent evaluation.
|
|
Innovations in HIV/AIDS Prevention and Education: Reining in the Evaluation After We Got Creative
|
| Presenter(s):
|
| Jennifer Camacho Catrambone,
Ruth M Rothstein CORE Center,
jcamacho@corecenter.org
|
| Rebecca Goldberg,
Ruth M Rothstein CORE Center,
rgoldberg@corecenter.org
|
| Jackie Boone,
Ruth M Rothstein CORE Center,
jboone@corecenter.org
|
| Chester J Kelly,
Ruth M Rothstein CORE Center,
ckelly@corecenter.org
|
| Peter McLoyd,
Ruth M Rothstein CORE Center,
pmcloyd@corecenter.org
|
| Abstract:
A multidisciplinary team based at a midwestern infectious disease clinic developed this innovative presentation designed to educate its audience (primarily community college students in Chicago) by shocking them into paying attention and taking HIV/AIDS prevention seriously. This approach was developed to combat “prevention fatigue,” a phenomenon in which people have heard HIV/AIDS prevention in the background so often that it seems to have become easier to ignore. This session will first describe the multifaceted and innovative approach our team took and contrast it with more traditional HIV/AIDS prevention messages. Subsequently, presenters will discuss the problems the unique presentation style created for keeping the evaluation methodologically sound, missed evaluation opportunities and their causes, as well as two years worth of evaluation data. Finally, the team will list future directions of the evaluation.
|
| | | |
|
Session Title: Evaluation Apprenticeship: Teaching Evaluation by Doing
|
|
Panel Session 532 to be held in the Granite Room Section C on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Teaching of Evaluation TIG
|
| Chair(s): |
| Cynthia Tananis,
University of Pittsburgh,
tananis@education.pitt.edu
|
| Abstract:
The Collaborative for Evaluation and Assessment Capacity (CEAC) at the University of Pittsburgh's School of Education provides collaborative workspace for evaluation work in education (preK-16), child advocacy and support, and related initiatives through faculty and graduate students. Graduate students come to CEAC with no or little evaluation experience and often faculty serve as content-specialist consultants rather than bringing any particular evaluation expertise.
This panel will explore the ways in which apprenticeship (specifically for graduate students) can provide for their own field-based training in evaluation to augment their primary studies in other areas of education such as comparative studies, international development, educational leadership, and research methodology and simultaneously provide high-quality evaluation services for CEAC clients.
The panel will consist of two faculty members (one who serves as the Director of CEAC and one who works in collaboration with CEAC) and three graduate student evaluators, one who serves as the full-time Project Manager for CEAC and two others who serve as graduate assistants. Lessons learned and dilemmas from practice will be explored from each perspective.
|
|
From the Evaluation Director Perspective
|
| Cynthia Tananis,
University of Pittsburgh,
tananis@education.pitt.edu
|
|
As the director of the evaluation collaborative, I am ultimately responsible for contracts and grants, evaluation planning and development, hiring and supervision of staff and graduate students and all management and leadership aspects of the Collaborative. As such, I offer this perspective on the issues identified in the panel description and rationale. My role on the panel is both to manage the conversation (to assure ample coverage from each of the presenters' perspectives and the audience) and to present the issues and dilemmas of managing a large evaluation group with staff and students who have no formal evaluation training or experience.
|
|
|
From the Perspective of Full-time Staff: The Project Manager
|
| Cara Ciminillo,
University of Pittsburgh,
ciminill@education.pitt.edu
|
|
As the Project Manager for CEAC and doctoral student in educational leadership (with the Director as my academic advisor), I wear numerous hats. I am the first-line supervisor for the other graduate students who work for CEAC while also being a peer student and evaluator with them on projects. I serve as a peer evaluator with my academic advisor and employer/CEAC Director. I supervise and hire undergraduate students who work as support staff for our unit. While I had no formal evaluation experience prior to working with my advisor and CEAC, I find my continuing academic pursuits and interests being influenced by my work and my work being equally influenced by my academic interests. My perspective is varied and unique among CEAC staff and students.
| |
|
From the Perspective of a Doctoral Student: Graduate Assistant
|
| Keith Trahan,
University of Pittsburgh,
kwt2@pitt.edu
|
|
I serve as a graduate assistant at CEAC and am a full-time doctoral student in social and comparative analysis in education. I am lead evaluator on a number of projects and also work as a team member on a number of large projects as well. My academic interests include history and philosophy and their interplay in how educational issues are framed and studied. I have worked with CEAC for two years now and offer a perspective of how my work with CEAC has influenced my academic work and how my academic focus has also brought certain features to the evaluation work I do individually and as a team member.
| |
|
From the Perspective of a Masters Student: Graduate Assistant
|
| Tracy Pelkowski,
University of Pittsburgh,
tlp26@pitt.edu
|
|
I have worked as a graduate assistant for CEAC for one year and am completing a masters degree in social and comparative analysis in education. I have been a K-12 teacher in an urban district for a number of years as well. As both a novice graduate student (hoping to move directly into the doctoral program at the conclusion of my masters degree) and a novice evaluator, my first year at CEAC has been all about learning. Often I come to projects that are well underway and, as such, need to catch up quickly with both an understanding of context as well as evaluation. I bring the perspective of a true novice to CEAC as well as the panel.
| |
|
From the Perspective of a Collaborating Faculty Member
|
| Junlei Li,
University of Pittsburgh,
jul@pitt.edu
|
|
I am an Assistant Professor in the School of Education and also serve as the Evaluation Director for the Office of Child Development, a department in the School. I have worked with CEAC to consider avenues of professional collaboration and development for our staff to share expertise and learning. Additionally, we have collaborated on a grant proposal and a number of shared projects across our units. I offer the perspective of a faculty colleague as well as a manager of evaluation services.
| |
|
Session Title: Reviving the Evaluability Assessment (EA) Method: EAs of Interventions to Prevent Childhood Obesity
|
|
Multipaper Session 535 to be held in Room 102 in the Convention Center on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Health Evaluation TIG
|
| Chair(s): |
| David Cotton,
Macro International Inc,
david.a.cotton@macrointernational.com
|
| Discussant(s): |
| Debra Rog,
Rockville Institute,
debrarog@rockvilleinstitute.org
|
| Laura Kettel Khan,
Centers for Disease Control and Prevention,
ldk7@cdc.gov)
|
| Nicola Dawkins,
Macro International Inc,
nicola.u.dawkins@macrointernational.com
|
| Abstract:
In 2007, the Robert Wood Johnson Foundation funded the Centers for Disease Control and Prevention (CDC) Foundation, which provided funds to the CDC and Macro International to conduct 25 evaluability assessments (EAs) on selected initiatives to improve the eating habits and physical activity levels of children. EAs provide a systematic way to determine if an initiative is ready for rigorous evaluation. Developed by Joseph Wholey and colleagues, EAs received significant attention 30 years ago but later fell into disuse (Rog 1985; Trevisan 2007). Today the method is seeing renewed interest, particularly for quickly and cheaply gauging whether an initiative is plausible and feasible, and whether investment of evaluation resources is appropriate. This session will share the method for selecting initiatives and implementing the EA process, process for identifying and training 33 professionals to conduct EAs, and lessons and products of the process. Discussants will reflect on the project to date.
|
|
Evaluability Assessments of Interventions to Prevent Childhood Obesity: What, Why, and How
|
| Holly Wethington,
Centers for Disease Control and Prevention,
eyk5@cdc.gov
|
| Nicola Dawkins,
Macro International Inc,
nicola.u.dawkins@macrointernational.com
|
| Seraphine Pitt-Barnes,
Centers for Disease Control and Prevention,
spe6@cdc.gov
|
| Diane Dunet,
Centers for Disease Control and Prevention,
dol0@cdc.gov
|
| David Cotton,
Macro International Inc,
david.a.cotton@macrointernational.com
|
| Leah Robin,
Centers for Disease Control and Prevention,
ler7@cdc.gov
|
| Jo Anne Grunbaum,
Centers for Disease Control and Prevention,
jpg9@cdc.gov
|
| Laura Leviton,
Robert Wood Johnson Foundation,
llevito@rwjf.org
|
| Laura Kettel Khan,
Centers for Disease Control and Prevention,
ldk7@cdc.gov)
|
|
Evaluability assessments (EAs) provide a systematic way to determine if an intervention is ready for rigorous evaluation and identify refinements that may be needed prior to formal evaluation. This method gauges an initiative's plausibility and feasibility, stakeholder agreement on objectives, data systems and capacity to collect data, and consistency of key elements with program theory. In 2007, the Early Assessment of Programs and Policies to Prevent Childhood Obesity project conducted 25 EAs on programs and policies that address the physical, economic, or social environment to prevent childhood obesity. Three focus areas were selected: programs or policies in the after school/daycare settings; programs or policies to increase access to healthier foods in urban, low-income communities; and school district local wellness policies. This presentation will describe the EA methodology and will demonstrate the value of using a systematic process to conduct rapid assessments to identify those most suited for full-scale evaluation.
|
|
Teaching Evaluability Assessment Methodology
|
| Thearis Osuji,
Macro International Inc,
thearis.a.osuji@macrointernational.com
|
| Nicola Dawkins,
Macro International Inc,
nicola.u.dawkins@macrointernational.com
|
| Holly Wethington,
Centers for Disease Control and Prevention,
eyk5@cdc.gov
|
| Seraphine Pitt-Barnes,
Centers for Disease Control and Prevention,
spe6@cdc.gov
|
| Diane Dunet,
Centers for Disease Control and Prevention,
dol0@cdc.gov
|
| David Cotton,
Macro International Inc,
david.a.cotton@macrointernational.com
|
| Leah Robin,
Centers for Disease Control and Prevention,
ler7@cdc.gov
|
| Jo Anne Grunbaum,
Centers for Disease Control and Prevention,
jpg9@cdc.gov
|
| Laura Leviton,
Robert Wood Johnson Foundation,
llevito@rwjf.org
|
| Laura Kettel Khan,
Centers for Disease Control and Prevention,
ldk7@cdc.gov)
|
|
In 2007, the Early Assessment of Programs and Policies to Prevent Childhood Obesity project conducted evaluability assessments (EAs) of programs and policies with the potential to improve the eating habits and physical activity levels of children. One goal was to develop a network of evaluation professionals with the skills to conduct evaluability assessments. This presentation will focus on the process of identifying and training 33 professionals in the EA methodology, results from a post EA survey of the evaluators, and insights from the process of coordinating a large number of evaluation professionals in 3 months. The training process highlighted: the benefit of a diverse team approach; the distinction between EA and full evaluation; the critical nature of strong qualitative interview skills; the significant time commitment for evaluators; and the need for centralized oversight for methodologic integrity among evaluators.
|
|
Lessons Learned: Practical Tips for Conducting Evaluability Assessments
|
| Seraphine Pitt-Barnes,
Centers for Disease Control and Prevention,
spe6@cdc.gov
|
| Holly Wethington,
Centers for Disease Control and Prevention,
eyk5@cdc.gov
|
| Karen Cheung,
Macro International Inc,
karen.cheung@macrointernational.com
|
| Nicola Dawkins,
Macro International Inc,
nicola.u.dawkins@macrointernational.com
|
| Diane Dunet,
Centers for Disease Control and Prevention,
dol0@cdc.gov
|
| David Cotton,
Macro International Inc,
david.a.cotton@macrointernational.com
|
| Leah Robin,
Centers for Disease Control and Prevention,
ler7@cdc.gov
|
| Jo Anne Grunbaum,
Centers for Disease Control and Prevention,
jpg9@cdc.gov
|
| Laura Leviton,
Robert Wood Johnson Foundation,
llevito@rwjf.org
|
| Laura Kettel Khan,
Centers for Disease Control and Prevention,
ldk7@cdc.gov)
|
|
The Early Assessment of Programs and Polices to Prevent Childhood Obesity project offers key lessons learned from quickly conducting large numbers (25-30 annually) of evaluability assessments (EAs). These lessons are offered to inform others interested in or using this methodology. Among the lessons learned are: (1) clear communication is critical with evaluators and sites about roles, expectations, and the EA site visit process; (2) site documents often do not represent the program accurately; (3) involving the site in the selection of stakeholder interviewees is key to ensure comprehensive perspectives; (4) logic models are an appropriate and enlightening method for examining program goals and activities; (5) standard reporting formats provide clarity and consistency by evaluators; and (6) brief technical assistance calls and formal site reports are highly valued, low cost incentives for participation and useful tools for program/policy improvement.
|
|
Impact of the Evaluability Assessment Process: Early Returns
|
| Laura Leviton,
Robert Wood Johnson Foundation,
llevito@rwjf.org
|
| Holly Wethington,
Centers for Disease Control and Prevention,
eyk5@cdc.gov
|
| Seraphine Pitt-Barnes,
Centers for Disease Control and Prevention,
spe6@cdc.gov
|
| Diane Dunet,
Centers for Disease Control and Prevention,
dol0@cdc.gov
|
| Leah Robin,
Centers for Disease Control and Prevention,
ler7@cdc.gov
|
| Jo Anne Grunbaum,
Centers for Disease Control and Prevention,
jpg9@cdc.gov
|
| David Cotton,
Macro International Inc,
david.a.cotton@macrointernational.com
|
| Nicola Dawkins,
Macro International Inc,
nicola.u.dawkins@macrointernational.com
|
| Laura Kettel Khan,
Centers for Disease Control and Prevention,
ldk7@cdc.gov)
|
|
The Early Assessment of Programs and Polices to Prevent Childhood Obesity project had four notable consequences in its first year of operation. (1) The process revealed more than ten policies and programs worthy of more rigorous evaluation. (2) Three such policies and programs rose to the top of the list of priorities: a day care policy for prevention of childhood obesity, a program designed to bring supermarkets back to underserved inner city environments, and an incentive program for poor families to buy fruit and vegetables at farmers' markets. (3) Most sites visited found the feedback helpful for program improvement. (4) Based on site visitor reports and expert panel discussions, themes and research issues emerged for all policy and program types that were assessed. This process might easily be emulated for other policies and programs to find out "what works."
|
|
Session Title: Evaluation Policy and Training Evaluation in Child Welfare
|
|
Panel Session 536 to be held in Room 104 in the Convention Center on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Human Services Evaluation TIG
|
| Chair(s): |
| Henry Ilian,
New York City Children's Services,
henry.ilian@dfa.state.ny.us
|
| Abstract:
This panel will address evaluation policy's influence on evaluation practice within child welfare systems. It posits that the absence of evaluation policy creates structural barriers to conducting training evaluation. In the absence of policy, training and training evaluation compete with all other organizational priorities for resources and managerial attention. This competition is governed by such factors as organizational structure, type of leadership, leadership turnover, organizational culture, and caseload size. Thereby the ability of evaluators to get data and to show how training is or is not influencing practice is frequently impeded. This conclusion is based on two sources of evidence: (1) an overview of the training evaluation literature; (2) lessons drawn from training evaluation within child welfare agencies in three localities, the state of Kentucky and the cities of Los Angeles and New York.
|
|
Effects of Leadership Change on Evaluation Efforts in Kentucky
|
| Anita Barbee,
University of Louisville,
anita.barbee@louisville.edu
|
|
Dr. Barbee was selected to participate in this panel because she has worked with one state child welfare agency conducting training evaluation over the past 16 years under four different governors, three different configurations of the agency and eight different commissioners. These changes have had an impact on organizational structure and culture, leadership style and expertise as well as case load size. All of these variables have affected access to decision makers, whether or not research and evaluation is valued, the evaluator's ability to obtain data from stressed and overworked employees, the quality of that data, the usefulness of the data to the agency, and the willingness of the agency to make changes based on the reported results. One solution to the problems would be the establishment of a formal evaluation policy so that such changes would not undermine the evaluation enterprise. This and other possible solutions will be discussed.
|
|
|
Evaluation Policy and Child Welfare Training Evaluation: Lessons from Los Angeles
|
| Todd Franke,
University of California Los Angeles,
tfranke@ucla.edu
|
|
Dr. Franke was selected for this panel as a result of his extensive work at the intersection of evaluation and child welfare issues. He has been involved with agencies that serve thousands of families representing unique geographic and cultural communities in California, particularly southern California counties. He is the PI for the evaluation of the First 5 Los Angeles funded Partnership for Families Initiative. Partnerships for Families is a five year secondary prevention initiative that is designed to prevent child maltreatment by creating opportunities for families, communities, and governmental entities to meaningfully partner in the prevention of child abuse and neglect throughout Los Angeles County. It is designed to assist at-risk families including pregnant and parenting teens. Dr. Franke also is the PI for the Small County Initiative for the past six years, which is designed to systematically evaluate the State of California's efforts to build and enhance child abuse and neglect prevention efforts in 11 rural counties in northern California. This effort required extensive collaboration with multiple county agencies and employed a mixed model approach to data collection. For the past several years he has been working with the Department of Children and Family Services (DCFS-Los Angeles) in evaluating their pre-service and in-service training for Child Protective Service (CPS) workers. In this capacity he has had the opportunity to work collaboratively with the Training Director and supervisors.
| |
|
Training Evaluation and Operational Imperatives at New York City's Children's Services
|
| Henry Ilian,
New York City Children's Services,
henry.ilian@dfa.state.ny.us
|
|
Dr. Ilian was selected to participate on this panel because of his work over twenty years in evaluating training at Children's Services, the New York City child welfare agency. He is strongly interested in the organizational dynamics--including change and persistence in organizational culture and the role of leadership--that permit or inhibit the evaluation of training. He will address lessons learned from efforts undertaken between 2002 and 2005 to conduct a multidimensional evaluation of a large-scale supervisory training program. The evaluation ultimately had to be abandoned, but the experience proved to be rich in insights into the conditions under which training evaluations are and are not successfully conducted within large child welfare organizations. Implications point to an evaluation policy that mediates between the competing imperatives of conducting an agency's ongoing work and ensuring the success of efforts to instill a mode of practice that meets nationally recognized standards.
| |
|
Session Title: Evaluating Programs in Times of Complexity and Chaos
|
|
Panel Session 537 to be held in Room 106 in the Convention Center on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Systems in Evaluation TIG
|
| Chair(s): |
| Daniel Folkman,
University of Wisconsin Milwaukee,
folkman@uwm.edu
|
| Abstract:
How do we evaluate programs that operate in complex, rapidly changing, and indeterminate environments? Join our panel presentation to learn how this one evaluation strategy incorporates principles from complexity theory along with more traditional approaches. The utility of our evaluation strategy is demonstrated through a case study of the Healthier Wisconsin Partnership Program. This program brings together a group of community partners including an urban middle school, a school based health clinic, a community based agency, and a family leadership development program. The goal is to address the academic, health, safety and wellness issues facing low-income, high-risk, middle school girls. The challenge is that each partner operates within highly complex environments where implementing strategies are continuously changing and outcomes emerge that were not anticipated by the program model. Learn how the evaluation design documents the emerging outcomes and facilitates stakeholder decision making while operating in complexity and near chaos.
|
|
Evaluating Programs That Operate in Complex and Chaotic Environments
|
| Daniel Folkman,
University of Wisconsin Milwaukee,
folkman@uwm.edu
|
|
My presentation is an overview of the evaluation design and its implementation. A central component involves creating a learning community among the partners, which meets monthly to review the progress and challenges encountered as the program unfolds. One tool that is used to document the implementation process is called 'Headlines.' Here each partner records the major accomplishments, challenges, and critical learning that has occurred over the previous month. These headlines provide the basis for constructing a narrative on how the program is evolving as it is happening. A second tool involves creating an outcome database called 'good stuff' This is literally a database of unique accomplishments that the girls are producing as they participate in the program but could never be anticipated. This tool is highly sensitive to the emergent patterns and outcomes that the girls are producing. These introductory comments set the stage for the next two presentations.
|
|
|
Creating Partnership in the Face of Complexity
|
| Devarati Syam,
University of Wisconsin Milwaukee,
devasyam@uwm.edu
|
|
Creating a community-based partnership model for addressing the health, safety and wellness issues facing high-risk, African American middle school girls is a central goal of this project. Therefore, the collaborative endeavor requires multiple partners to coordinate their services. The partners belong to different complex systems (both institutional and situational) and are attempting to develop a program by managing different expectations, values, roles and responsibilities that are part of their institutional commitments.
My presentation will report the progress we have made in this partnership building attempt and how the partners have experienced their participation in this process so far - their learning as well as their struggles and challenges. The partners meet once a month to problem solve around the different issues that crop up in the implementation. This presentation will provide the highlights from these conversations and draw implications of evaluating the partnership building project in this context.
| |
|
Facilitating Teen Support Groups Where Complexity is the Norm and Chaos is Just Around The Corner
|
| Yvette Dobson,
PEARLS for Teen Girls Inc,
yvette@pearlsforteengirls.com
|
|
This presentation is focused on implementing the PEARLS program. The school serves some of the most 'high-risk' youth in the district making the environment highly charged and ready to erupt at any moment. Therefore, my programming context is characterized by a high degree of uncertainty and unpredictability. My primary responsibility is to work closely with the teen girls and develop a relationship with them. But it also entails working with teachers, nursing staff, parents, and community residents.
The discussion will include critical incidents that demonstrate the complexity of working with teen girls within a middle school setting. The incidents will also show the connections with other partners including school nurses, parents, and community influences. The school principal captures my challenge nicely when she described by role as being like the cartoon character Plastic Man. To be successful I must bend and shape to the changing conditions within my environment.
| |
|
Session Title: Evaluation Strategies and Assessment in the Arts
|
|
Multipaper Session 538 to be held in Room 108 in the Convention Center on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Evaluating the Arts and Culture TIG
|
| Chair(s): |
| Ching Ching Yap,
University of South Carolina,
ccyap@gwm.sc.edu
|
| Discussant(s): |
| Rob Horowitz,
Columbia University,
artsresearch@aol.com
|
|
Developing a Theory-Based Evaluation Strategy Within a University-Based Arts Education Program
|
| Presenter(s):
|
| Mark Hansen,
University of California Los Angeles,
hansen.mark@gmail.com
|
| Noelle Rivera,
University of California Los Angeles,
novari320@hotmail.com
|
| Abstract:
ArtsBridge at the UCLA School for the Arts and Architecture provides high-quality arts education experiences for youth and teachers in local schools who would otherwise have little or no access to such opportunities. Recently, the Social Research Methodology Evaluation Group of the UCLA Graduate School of Education and Information Studies began consultation with ArtsBridge. Our purpose was to assess the program’s current approach to evaluation and to identify opportunities for enhancing these efforts. By engaging a broad base of stakeholders, the assessment process clarified objectives related to multiple target audiences. Logic models were developed to illustrate hypothesized links between these objectives and program activities. Using these models and input from stakeholders concerning areas of inquiry that should be prioritized, draft questionnaires were created and pilot-tested. This paper will describe the consultation process, share preliminary findings and demonstrate the appropriateness of theory-based approaches to evaluation within the field of arts education.
|
|
Exploring the Nonformal Education Frontier: Using Cluster Analysis to Investigate Children’s Museum Programming for Special Learner Groups
|
| Presenter(s):
|
| Deborah Chapin,
University at Albany - State University of New York,
debchapin50@juno.com
|
| Dianna Newman,
University at Albany - State University of New York,
dnewman@uamail.albany.edu
|
| Abstract:
This study presents the findings of a study on the perceptions of eighty education directors of children’s museums responding to a paper-pencil survey examining patterns of planning, designing, and implementing nonformal education programs for minority youth and children with disabilities. K-means cluster analysis, applied as an exploratory tool, demonstrated that three major children’s museum types exist, modest-sized urban “dedicated” museums, small suburban “elite” museums, and large urban “generic” museums. Dedicated and elite museums planned their curriculum using State Standards, teachers, and community cultural leaders more than generic museums. Dedicated and generic museums followed Universal Design strategies when serving children with disabilities; however, only dedicated museums used elements of culturally relevant education with minority youth. “Dedicated” children’s museums charged the least per admission, devoting fewer resources to out-reach to special learner groups and offering less variety in their current programming, but had staff holding more specialized degrees creating specific science content.
|
|
Examining Arts Educators Use of Arts Assessment Results
|
| Presenter(s):
|
| Tara Pearsall,
University of South Carolina,
tmcpearsall@yahoo.com
|
| Kate Halladay,
University of South Carolina,
kate.halladay.art@gmail.com
|
| Ashlee Lewis,
University of South Carolina,
scaap@gwm.sc.edu
|
| Ching Ching Yap,
University of South Carolina,
ccyap@gwm.sc.edu
|
| Abstract:
In 1999, the South Carolina Department of Education, University of South Carolina, and South Carolina arts agencies collaborated to develop a comprehensive program that encourage standards-based arts instruction and measure standards-based arts achievement. To achieve those goals, collaborators created professional development programs for arts educators and developed standardized arts assessments in all four arts areas. The arts assessments provide school-level information regarding student arts achievement and the professional development programs enable arts educators to use the results to inform their instruction. Although numerous arts educators involved in this comprehensive program have informally communicated accounted their experiences with the program, including their use of the assessment results, no formal attempts has been made to collect that information. The purpose of this study is to collect information regarding teachers’ experiences with this comprehensive program.
|
| | |
|
Session Title: Modeling the Logic and Measuring the Performance of Government Programs
|
|
Multipaper Session 539 to be held in Room 110 in the Convention Center on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Government Evaluation TIG
|
| Chair(s): |
| David J Bernstein,
Westat,
davidbernstein@westat.com
|
| Discussant(s): |
| David J Bernstein,
Westat,
davidbernstein@westat.com
|
|
Assisting the Local Program Level with Performance Measurement
|
| Presenter(s):
|
| Leanne Kallemeyn,
Loyola University Chicago,
lkallemeyn@luc.edu
|
| Abstract:
A critical evaluation policy, the Government Performance and Results Act of 1993, has contributed to the growth of performance measurement. While some evaluators have criticized performance measurement, others have contributed to its development and incorporated it into their evaluations. The purpose of this paper is to further contribute to this conversation about how evaluators can engage performance measurement. I use a case example of how I, as an evaluator, assisted a local Head Start program in using mandated school readiness assessments. Based on the literature and case, I identified five areas for an evaluator to work within and challenge performance measurement: What is assessed, how it is assessed, how results are interpreted and used, what is the purpose, and what relationships are facilitated. I also provided example activities that I completed with the local program to illustrate how evaluators can assist local programs with performance measurement.
|
|
Practical Limitations to Gathering Data in Support of GPRA and PART
|
| Presenter(s):
|
| William Scarbrough,
Macro International Inc,
william.h.scarbrough.iii@macrointernational.com
|
| Herbert Baum,
Macro International Inc,
herbert.m.baum@macrointernational.com
|
| Renee Bradley,
United States Department of Education,
renee.bradley@ed.gov
|
| Andrew Gluck,
Macro International Inc,
andrew.gluck@macrointernational.com
|
| Abstract:
Providing the Office of Management and Budget with data to support a program’s Program Rating Assessment Tool (PART) represents an unfunded mandate for the program. As such, the program often can only gather data on a “shoestring” budget. Macro International has worked for 4 years with the Research to Practices Program, Office of Special Education Programs, US Department of Education, to develop program measures and collect data in support of these. In this paper we briefly describe how we have used expert panels to review material, and the challenges associated with this approach.
|
|
Partnership Contribution to Program Outcomes: Logic Models as Tools for Developing Measures for Partnerships
|
| Presenter(s):
|
| Valerie Williams,
RAND Corporation,
valerie@rand.org
|
| Lauren Honess-Morreale,
RAND Corporation,
laurenhm@rand.org
|
| Abstract:
Partnerships are a common strategy for many programs. Government programs, in particular, rely on partnerships because they offer the potential to deliver more effective programs as well as increased efficiency in producing and delivering outputs. Despite the ubiquitous nature of partnerships, there is very little information on developing measures to assess the value of partnerships, particularly as it relates to the contribution of partnerships to achieving program outcomes. In this paper, the authors describe the use of a logic model template as a means for developing measures that can be applied to partnerships. The logic model provides a framework for describing partnership inputs, activities, outputs and outcomes, and as such, serves as a basis for developing measures to assess 1) specific elements of partnerships; 2) the cost efficiency and effectiveness of the partnership; and 3) the extent to which “partnership,” rather than simply two entities working together, contribute to program outcomes.
|
| | |
|
Session Title: Process Use-Methods and Measures
|
|
Multipaper Session 542 to be held in Room 105 in the Convention Center on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Evaluation Use TIG
|
| Chair(s): |
| Sandra Ortega,
National Data Evaluation Center,
ortegas@ndec.us
|
|
Promoting the Use of Evaluation Information: Early Reading First in Chicago Charter Schools
|
| Presenter(s):
|
| Tania Rempert,
University of Illinois Urbana-Champaign,
trempert@uiuc.edu
|
| Lizanne DeStefano,
University of Illinois Urbana-Champaign,
destefan@ad.uiuc.edu
|
| William Teale,
University of Illinois Chicago,
wteale@uic.edu
|
| Abstract:
This presentation is organized via alignment of evaluative questions, measurement tools, and reporting strategies. Comments from program staff highlight the methods, strategies, and tools that have been most useful to them: observations, teacher portfolios, assessment data, and PD questionnaire. This evaluation has been designed to be longitudinal in nature in order to capture changes that occur over time and across settings. Formatively, the evaluation routinely monitors progress toward specific objectives, assesses the quality of implementation, and gauges the short-term impacts of the project. This information is reported to program staff regularly at project management meetings so that it could be used to guide program development and improvement. Summatively, the evaluation employs a quasi-experimental pre-test, post-test, non-equivalent comparison group design to document the impact of the project on teaching, learning and family. Finally, the evaluation is designed to provide information on effective strategies for replication of the program in other settings.
|
|
Initial Results from a Planned Approach to Process Use: Using Evaluation to Influence Public School Policy and the Classroom Practice of Formative Assessment
|
| Presenter(s):
|
| Debra Heath,
Albuquerque Public Schools,
heath_d@aps.edu
|
| Nancy Carrillo,
Albuquerque Public Schools,
carrillo_n@aps.edu
|
| River Dunavin,
Albuquerque Public Schools,
dunavin_r@aps.edu
|
| Abstract:
This paper describes how evaluators in an urban school district employed strategies to promote evaluation use, foster organizational alignment and influence evaluation policy.
Findings from an evaluation of district-wide formative assessment practices suggest that teachers under-utilized formative assessment techniques and that district support systems offered inconsistent formative assessment definitions, strategies and resources. We wanted to do more than provide the standard written report and set of presentations; we wanted to help optimize the district’s support and practice of proven formative assessment strategies. We used a collaborative evaluation approach with the intention of solidifying commitment to formative assessment, increasing communication between stakeholders, creating shared understandings regarding formative assessment, and fostering collaborations between departments.
We outline and critique our methods for achieving process use, and describe the techniques and tools we used to measure process use, including group processes, questionnaires, observations and interviews. Finally we share lessons learned.
|
|
Evaluation as Change Agent: The Precarious See-Saw Between Helping and Judging and How it Affects Process Use
|
| Presenter(s):
|
| Gale Mentzer,
University of Toledo,
gmentze@utnet.utoledo.edu
|
| Abstract:
This paper tracks the precarious path of program improvement that began with a required program evaluation for a federally funded project. It illustrates the complexity of working with “reluctant” stakeholders through the example of the evolution of a summer program designed to introduce secondary school students to teaching. A mixed methods evaluation was used to measure stated outcomes and to discover whether other, unanticipated factors were affecting program goal attainment. Application of the evaluation model to the program revealed not only that students’ misconceptions about teaching were being cleared up but also discovered a major unanticipated factor—the program instructors (university faculty) held misconceptions as to the goals of the program. This paper shows how, over the course of four years, group process techniques were used to engage the stakeholders in the evaluation in order to implement necessary programmatic changes as indicated by program evaluation results.
|
| | |
|
Session Title: Communities in Schools (CIS) Evaluations
|
|
Multipaper Session 543 to be held in Room 107 in the Convention Center on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG
|
| Chair(s): |
| Pamela Bishop,
Oak Ridge Institute for Science and Education,
pbaird@utk.edu
|
|
Determining the Impact of CIS Programs: Student-Level Randomized Controlled Trials
|
| Presenter(s):
|
| Clawson Heather,
ICF International,
hclawson@icfi.com
|
| Allan Porowski,
ICF International,
aporowski@icfi.com
|
| Felix Fernandez,
ICF International,
ffernandez@icfi.com
|
| Christine Leicht,
ICF International,
cleicht@icfi.com
|
| Abstract:
Communities In Schools, Inc. (CIS) is a nationwide initiative to connect community resources with schools to help at-risk students successfully learn, stay in school, and prepare for life. CIS is currently in the midst of a comprehensive, rigorous three-year national evaluation, culminating in a two-year multi-site randomized controlled trial (RCT) to ascertain program effectiveness. In this presentation, we will draw from our experience working with Austin, TX and Jacksonville, FL public schools, present our overall study design, and the process involved in conducting a student-level RCT. Preliminary results from Year One will also be discussed.
|
|
The Communities in Schools Inc (CIS) National Evaluation: Conducting Case Studies of Large, Diverse Community-Based Organizations
|
| Presenter(s):
|
| Sarah Decker,
ICF International,
sdecker@icfi.com
|
| Kelle Basta,
ICF International,
kbasta@icfi.com
|
| Susan Siegel,
Communities In Schools Inc,
ssiegel@cisnet.org
|
| Abstract:
Communities In Schools, Inc. (CIS) is a nationwide initiative to connect community resources with schools to help students in need. As part of the national student- and school-level evaluation, we have conducted eight case studies in an effort to gain an in-depth understanding of the processes that contribute to successful CIS affiliate- and site-level outcomes. This presentation will provide a two-fold description of case study implementation within diverse, community-based organizations. First, we will examine the methodology behind our case study of CIS sites. We will include a discussion of challenges encountered while conducting the studies in elementary, middle, and high schools located in rural, suburban, and urban areas around the country, and the solutions we developed to circumvent these issues. Second, we will use this presentation as an opportunity to present our findings from the case studies, and will discuss their value-added to the overall national evaluation framework.
|
|
Findings from the Communities In Schools (CIS) School-Level Quasi-Experimental Study
|
| Presenter(s):
|
| Allan Porowski,
ICF International,
aporowski@icfi.com
|
| Julie Gdula,
ICF International,
jgdula@icfi.com
|
| Susan Siegel,
Communities In Schools Inc,
ssiegel@cisnet.org
|
| Abstract:
Communities in Schools, Inc. (CIS) is a nationwide initiative to connect community resources with schools to help at-risk students successfully learn, stay in school, and prepare for life. As part of a five-year comprehensive national evaluation of CIS, a school-level quasi-experimental study was conducted to determine whether CIS sites effected school-level change on a range of outcomes, including dropout rates, graduation rates, attendance, and academic performance. A brief description of our matching procedures, which included a technique called propensity score matching, will be included, along with a general overview of how this component fits into our comprehensive national evaluation plan. The final results from this study will also be presented, which covers a sample of 604 CIS sites and 604 matched comparison sites. The presentation will conclude with lessons learned about the effectiveness of community-based integrated student supports (CBISS) in general, and the CIS program in particular.
|
|
Typology Results from the National Evaluation of Communities In Schools Inc (CIS)
|
| Presenter(s):
|
| Jing Sun,
ICF International,
jsun@icfi.com
|
| Katerina Passa,
ICF International,
kpassa@icfi.com
|
| Allan Porowski,
ICF International,
aporowski@icfi.com
|
| Susan Siegel,
Communities In Schools Inc,
ssiegel@cisnet.org
|
| Abstract:
Communities In Schools, Inc. (CIS) is a nationwide initiative to connect community resources with schools to help at-risk students successfully learn, stay in school, and prepare for life. To capture the diversity of the CIS network as part of our multi-component, national evaluation of the program, threshold analysis was employed to develop a typology, which identifies “high implementers” and “partial implementers” according to an ideal implementation model from the CIS Total Quality System (TQS). By comparing program outcomes (including dropout indicators and academic achievements) across typologies, we found that CIS “high implementers” outperformed their similar non-CIS comparison schools in most outcomes, while “low implementers” did not. We used this methodology to gain a deep understanding of how and why specific program models work. In this presentation, we will describe our methodology and results, as well as the processes that can be linked to positive outcomes. Implications for CIS will also be discussed.
|
| | | |
|
Session Title: The Challenges of Conducting Experimental and Quasi-Experimental Small-Budget Evaluations of Teacher Professional Development Programs: The Case of Teaching American History Grants
|
|
Multipaper Session 545 to be held in Room 111 in the Convention Center on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Sally Bond,
The Program Evaluation Group LLC,
usbond@mindspring.com
|
| Abstract:
Experimental and quasi-experimental designs are becoming customary in RFPs from federal agencies including the U.S. Department of Education, National Science Foundation, and others. This session will explore the application of those designs in small-budget evaluations. Presenters will draw upon their years of experience using such designs in Teaching American History (TAH) grant project evaluations to illustrate key features of design and implementation. Of particular emphasis will be negotiating program and evaluation design with program staff, matching assessment to content (including design and validation of locally developed instrumentation), and challenges of implementing experimental/quasi-experimental evaluations (including factors necessitating changes in evaluation design and strategies for recruiting control or comparison groups). Presenters will discuss their own experiences and will engage attendees in discussing lessons learned in this and other evaluation contexts.
|
|
Negotiating Project Design to Permit Experimental and Quasi-Experimental Evaluation Designs
|
| Sally Bond,
The Program Evaluation Group LLC,
usbond@mindspring.com
|
|
Negotiating program design elements to accommodate a robust experimental or quasi-experimental design is often overlooked when developing proposals or designing programs. In such cases where experimental or quasi-experimental evaluation designs are desirable and/or required, it is the responsibility of the evaluator to collaborate with program staff to (1) educate them about the demands of the desired evaluation design and (2) design a program that is consistent with the aims of the project and the needs of the evaluation design. The presenter will illustrate the iterative process of program and evaluation design in a TAH program, including selection of the linchpin assessment of teacher learning which provided the best fit with the resulting design and content to be conveyed.
|
|
The Critical Role of Assessment in Experimental and Quasi-Experimental Designs
|
| Michael Herrick,
Herrick Research LLC,
herrickresearch@aol.com
|
|
Teaching American History grants are required to be evaluated with experimental or quasi-experimental designs. The expected outcomes of these projects are often expressed as gains in student achievement. This session will explore two different TAH evaluations, each using different types of assessments for measuring student achievement. One project used nationally validated NAEP items, while the other used locally developed and locally validated test items. The pre to post test gains in the experimental group using NAEP items were not statistically different than those of the control group. For the project using locally development and validated items, the gains of the experimental group over the control group were statistically significant at the .01 level. The primary conclusion was that the NAEP items were not strongly aligned to the content of the project, whereas the locally developed assessments were aligned and, therefore, sensitive to instruction.
|
|
The Challenges of Implementing Experimental and Quasi-Experimental Designs in Teaching American History Grant Evaluations
|
| Kristine Chadwick,
Edvantia Inc,
kristine.chadwick@edvantia.org
|
| Kim Cowley,
Edvantia Inc,
kim.cowley@edvantia.org
|
| Georgia Hughes-Webb,
Edvantia Inc,
georgia.hughes-webb@edvantia.org
|
|
Through six projects, TAH program evaluators at Edvantia have taken on the challenge of implementing mandated experimental or quasi-experimental designs on shoestring budgets. Working with slightly different designs and methods, these evaluators have learned a few lessons about how to adhere to the rigorous evaluation mandate in the program solicitation, soothe client fears, partner with clients to implement these designs, collect data for both formative and summative purposes, stay within budget, and successfully manage at least a few threats to validity. Three main challenges will be discussed. Evaluations were often downgraded from true experimental designs to quasi-experimental either in the initial design phase or later in the project implementation. Negotiating and recruiting teacher comparison groups required different solutions in every project. Comparison group data collection has required evaluator flexibility and extensive use of persuasion skills, thus highlighting both the technical and people skills evaluators must have to be effective.
|
|
Session Title: Evaluation Planning Incorporating Context (EPIC): A Model and Case Examples of Practice
|
|
Panel Session 546 to be held in Room 113 in the Convention Center on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the Theories of Evaluation TIG
|
| Chair(s): |
| Debra Holden,
RTI International,
debra@rti.org
|
| Abstract:
This session will present an overview of a Sage book (release date: 10/08), entitled A Practical Guide to Program Evaluation Planning. The book provides a step-by-step process to guide evaluators as they begin developing a program evaluation with a number of settings. Our five step conceptual framework, called Evaluation Planning Incorporating Context (EPIC), includes 1) assessing context (e.g., stating the purpose of the evaluation); 2) gathering reconnaissance (e.g., determining the evaluation uses); 3) engaging stakeholders (e.g., ensuring stakeholders' buy-in); 4) describing the program (e.g., stating the theoretical underpinnings of the program); and 5) focusing the evaluation (e.g., assessing feasibility of proposed measures). We will first provide an overview of our model, including an explanation of each step and corresponding issues to anticipate for each, and then invited presenters, who are authors of chapters in the book, will present case examples of their planning process for evaluations conducted in differing contexts.
|
|
Evaluation Planning Incorporating Context (EPIC) Model Overview
|
| Debra Holden,
RTI International,
debra@rti.org
|
| Marc A Zimmerman,
University of Michigan,
marcz@umich.edu
|
|
In this presentation (and book), we introduce an evaluation planning process model we call the Evaluation Planning Incorporating Context (EPIC) model. We define evaluation planning as the initial or background processes that go into the final design and implementation of a program evaluation. We describe five general steps in this process assess context, gather reconnaissance, engage stakeholders, describe the program, and focus the evaluation that provide a guideline for evaluators as they develop an evaluation plan. The EPIC model provides a heuristic for evaluation planning rather than a specified set of steps that are required for all evaluations. Some parts of the model may be more or less applicable depending on such issues as the type of evaluation, the setting of the evaluation, the outcomes of interest, and the sponsor's interests. Thus, the EPIC model can be used as a kind of instruction guide to prepare for a program evaluation.
|
|
|
Planning for an Education Evaluation
|
| Julie Marshall,
University of Colorado Denver,
julie.marshall@uchsc.edu
|
|
In this presentation, contextual factors identified in planning the evaluation of a school-based nutrition curriculum in a rural, low-income community will be described. The curriculum delivery included fun, hands-on food preparation, cooperative learning, and activities that were tied to content standards in math, literacy and science. The evaluation focused on understanding long term curriculum effectiveness and factors that influence curriculum adoption and delivery. Evaluation planning considered local and state stakeholders; the stress surrounding high-stakes testing and the burden placed on schools for health related activities. Including teachers as part of the evaluation team was critical for informing the evaluation questions, for informing the context within which teachers and students operate that may modify curriculum delivery and impact, and in developing evaluation tools. Strategies for planning meaningful evaluations in a school setting and lessons learned will be highlighted.
| |
|
Evaluation Planning for a Service Agency Program
|
| Mari Millery,
Columbia University,
mm994@columbia.edu
|
|
This case study represents an evaluation planning process in a service program context. The EPIC model is applied to describe the planning for a study at the Leukemia & Lymphoma Society's Information Resource Center (LLS IRC), which responds to nearly 80,000 annual telephone inquiries from cancer patients and their family members. The study focused on a patient navigation intervention consisting of follow-up calls LLS IRC made to its clients. The author, who was contracted as an external evaluator, worked closely with LLS IRC managers and staff throughout the 3-month planning process which resulted in a fairly rigorous study design. The presentation will describe each planning step and discuss lessons learned. Issues of particular importance to service programs will be highlighted, including the complexity of the context, importance of stakeholders, process vs. outcome evaluation, and use of tools, conceptual frameworks, and evaluation research concepts while working with service program stakeholders.
| |
|
Evaluation Planning for a Community-Based Program
|
| Thomas Reischl,
University of Michigan,
reischl@umich.edu
|
| Susan P Franzen,
University of Michigan,
sfranzen@umich.edu
|
|
Planning an evaluation of a new community-based program required successful partnerships with the project's coordinating agency and other community-based organizations. In the presentation we describe these relationships and the role the evaluators played in developing and evaluating the new program. We adopted a "responsive predisposition" (Stake, 2004) to focus on key issues, problems, and concerns experienced by the program's stakeholders in plan development. We also adopted principles that engaged key stakeholders in the evaluation planning and in the implementation of the evaluation study. We describe the development of the evaluation plan for a new telephone information and referral service focused on serving African American families and reducing infant mortality among African American mothers. Finally, we discuss the utility of using an Evaluation Planning Matrix to help focus the evaluation. The evaluation plan focused on process evaluation goals and attempted an outcome evaluation study using baseline data from a previous study.
Stake, R. E. (2004). Standards-based and responsive evaluation. Thousand Oaks, CA: Sage.
| |
|
Planning for a Media Evaluation: Case Example of the National Truth Campaign
|
| W Douglas Evans,
The George Washington University,
sphwde@gwumc.edu
|
|
Media evaluation is an overarching subject area that includes the study of marketing campaigns intended to promote or change consumer behavior, as well as assessments of educational and entertainment media and the effects of news media on public discourse and policy. In this presentation, we describe evaluation planning strategies and research methods for health communication and marketing campaigns designed to affect consumer health behavior.
Media evaluation is distinct from other forms of program evaluation. It focuses on media effects on healthy behaviors or avoidance of unhealthy behaviors, as opposed to broad evaluation strategies that cross-cut multiple venues and approaches. Media evaluations measure four key process and outcome dimensions of campaign effectiveness: 1) exposure and recall, 2) message reactions and receptivity, 3) behavioral determinants, and 4) behavioral outcomes. After describing media evaluation methods, we describe the truth campaign evaluation, the largest antitobacco media campaign conducted in the United States.
| |
