|
Session Title: Get Engaged! Using Adult Learning Principles to Enhance Evaluation Learning
|
|
Skill-Building Workshop 734 to be held in International Ballroom A on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Presidential Strand
|
| Presenter(s):
|
| Ellen Taylor-Powell,
University of Wisconsin,
ellen.taylor-powell@ces.uwex.edu
|
| Christine Kniep,
University of Wisconsin,
christine.kniep@ces.uwex.edu
|
| Abstract:
Application of adult learning theory and appreciation of different learning styles can enhance our ability to enable people to learn from and about evaluation. Through a hands-on, interactive session, participants will be exposed to the foundations of adult learning theory through activities that use movement, music, brain energizers, and reflection to facilitate learning. Participants will practice specific activities and reflect on their application and appropriateness to their own work. We will simulate two situations common in our practice of working with community practitioners and organizations - a training workshop; a consultation about evaluation planning - to contextualize the activities. While the examples are drawn from our community-based practice, the concepts and principles are applicable for many evaluation contexts. This is a creative, active session, not a sit and listen session, which engages the mind and body in learning.
|
|
Session Title: Learning From the Consequences of No Child Left Behind on Evaluation (Part 1 of 2)
|
|
Multipaper Session 735 to be held in International Ballroom B on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Rebecca Eddy,
Claremont Graduate University,
rebecca.eddy@cgu.edu
|
| Abstract:
The primary purpose of the proposed sessions (Parts 1 and 2) is to explore what the field of evaluation has learned since the inception of No Child Left Behind (NCLB). Specifically, we will discuss how the practice of evaluation within the context of NCLB has itself been changed, and as a result, influences the purpose and the design of educational evaluation. In light of the reauthorization of NCLB set to occur in 2007, it is clear (for better or worse) that this legislation has substantively impacted schools, districts, and communities. It is also clear, but not often discussed, that NCLB has impacted the evaluation community. These panels explore NCLB and educational evaluation through the lens of professional evaluation. In the first panel, we hope to identify key issues of the legislation and then discuss case studies of how NCLB impacted a program, a school, and one district in Texas.
|
|
An Overview of No Child Left Behind
|
| Jack Mills,
Claremont Unified School District,
jackmillsphd@aol.com
|
|
Despite concerns over its lofty goals, implementation challenges and uneven results, No Child Left Behind (NCLB) is not going away. In fact, the sweeping educational reform President George W. Bush signed into law in 2002 is up for reauthorization in 2007 and the commission reviewing potential modifications in the law has proposed increasing the stringency of some provisions. The law affects evaluators working in public education in fundamental ways. Its requirements for 'rigorous scientific evidence' determine which types of research proposals receive government funding and which practices may be officially designated as 'effective'. The manner in which NCLB's accountability requirements are defined and measured affect the types of questions evaluators may be called upon to help school administrators answer and the type of evidence to be considered. This session will provide evaluators with an overview and update of the critical provisions of NCLB and how they affect our field.
|
|
Consequences of No Child Left Behind from a Local After School Evaluator
|
| Maura Harrington,
Lodestar Management/Research Inc,
mharrington@lmresearch.com
|
| Blanca Flor Guillen-Woods,
Lodestar Management/Research Inc,
bfguillen@lmresearch.com
|
|
Recent emphasis on impact evaluation has generated greater interest in outcome evaluation in K-12 settings. While seemingly favorable, several negative consequences have been discovered. While increased program accountability is essential, the indicators chosen by funders are often misleading. Some, particularly after school programs, are not fully compatible with the program's intent, compelling the program to change format and service delivery with some of the outcomes outpacing the program intensity to realize such results in a given time period. The question becomes: How should one address funders' expectations for academic improvement while ensuring that the program continues to improve in addressing youth development and engaging hard-to-reach students? Part of this challenge is related to how evaluation data are collected and used, particularly when trying to balance program and evaluation demands. The experiences with an Out-of-School Time (OST) evaluation of the program conducted by Woodcraft Rangers agency in Southern California are described.
|
|
No Child Left Behind and Evaluative Thinking From a Local School Perspective
|
| Meta Nelson,
Bassett Unified School District,
mnelson@bassett.k12.ca.us
|
| Rebecca Eddy,
Claremont Graduate University,
rebecca.eddy@cgu.edu
|
|
We have learned a great deal about how No Child Left Behind (NCLB) can impact the educational process at local schools as well as the field of evaluation. Schools have been challenged to meet Adequate Yearly Progress (AYP) goals, and educators have been required to take on more intensive roles as evaluators to succeed in the current system. Similarly, the evaluator's role in assisting schools can now be described as more focused on evaluation capacity building. This paper will describe a case study of a middle school in California where strategic changes in planning and teaching through increased evaluation capacity building contributed to improved student achievement. We will describe a model for macro-planning in which strong principal leadership guided structural changes and professional development activities such as peer coaching and the use of data from state and local assessments to evaluate the effectiveness of current practice.
|
|
Using Evaluation Data to Strike a Balance Between Stakeholders and Accountability Systems: Consequences of No Child Left Behind From a District Evaluator
|
| Maria Elise Defino Whitsett,
Austin Independent School District,
mwhitset@austinisd.org
|
| Lisa Schmitt,
Austin Independent School District,
lschmitt@austinisd.org
|
|
From the perspective of a district evaluator in a large Texas district, new challenges have arisen since the implementation of No Child Left Behind. Most notably, those challenges relate to
(1) navigating state and federal accountability systems with, at times, competing requirements;
(2) evaluating the effectiveness of the sanctions (i.e., Supplemental Service Providers and School Choice) districts are required to address;
(3) using Scientifically-Based Research (SBR) as a basis for selecting effective programs/interventions to implement; and
(4) initiating SBR given high student mobility, inefficient data management systems, and competing priorities of local schools.
The purpose of this presentation is to detail these challenges that district level evaluators deal with on a daily basis, to highlight ways in which they can put processes into place to strike a balance between supporting decision-making in the district and conducting rigorous research on its own programs.
|
|
Session Title: Lessons for Methodology and Fieldwork in International Development Evaluations: The Heifer International Impact Evaluations in Africa, Asia, and the Americas
|
|
Panel Session 736 to be held in International Ballroom C on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Chair(s): |
| Carlisle Levine,
Catholic Relief Services,
clevine@crs.org
|
| Abstract:
Since 2005, the Evaluation Center at Western Michigan University (EC) has been working with Heifer International as an independent agency to evaluate their projects in different countries. In 2005 and 2006, EC teams have evaluated 37 projects supported by Heifer in Thailand, Nepal, Albania, Peru and the US. In 2007, four EC teams will evaluate 32 projects in China, Kenya, Tanzania, and Cameroon. The main focus of those evaluations is to assess Heifer's impact; however the evaluators have been using the Key Evaluation Checklist to look also at aspects related to process, cost, comparisons and generalizability. An evaluation approach-the Heifer Hoofprint Model-was developed and is being improved over the years. This panel will bring together the Evaluation Director, the Evaluation Manager, and the Evaluation Client to discuss the main lessons learned from these complex evaluations, especially the ones concerned with methodology and fieldwork.
|
|
Setting the Stage: The Heifer Hoofprint Evaluation Model
|
| Thomaz Chianca,
Western Michigan University,
thomaz.chianca@wmich.edu
|
|
The main focus for this presentation will be to describe the Heifer Hoofprint Model, emphasizing the evaluation methodology and fieldwork strategies developed by the WMU Evaluation Center (EC) to evaluate the work done by Heifer International in nine countries in different parts of the world. Mr. Chianca has been managing the external evaluations conducted by the EC for Heifer for the past three years, and has led the evaluation teams in Peru, Thailand, and China.
|
|
|
The Client's Learning: How the Impact Evaluations have Affected the Agency
|
| Rienzzie Kern,
Heifer International,
rienzzie.kern@heifer.org
|
|
Mr. Kern is the Director for Planning, Monitoring and Evaluation for Heifer International and is the primary client for the external evaluations conducted by the WMU Evaluation Center. His contribution to the panel will be to discuss the impact of the evaluations in various levels of his agency (from fundraising to agency's senior leadership and country offices), and also to present the client's perspectives on the main learning for methodology and fieldwork from the impact evaluations.
| |
|
Learning for Methodology and Fieldwork from the Perspective of the External Evaluation Team
|
| Michael Scriven,
Western Michigan University,
scriven@aol.com
|
|
Dr. Scriven has been the project director and main brain power behind the design and implementation of the impact evaluations for Heifer International. He has also participated in fieldwork in many of the countries included in these evaluations. His presentation will focus mainly in providing a critical reflection about the three rounds of impact evaluations with special attention to the main learning for methodology and fieldwork.
| |
|
Session Title: Findings From the Cross-site Evaluation of the Safe Schools, Healthy Students Initiative
|
|
Panel Session 737 to be held in International Ballroom D on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Crime and Justice TIG
|
| Chair(s): |
| James Trudeau,
RTI International,
trudeau@rti.org
|
| Abstract:
The Safe Schools/Healthy Students (SS/HS) Initiative is a landmark interdepartmental effort supported by an unprecedented collaboration among the Departments of Education, Health and Human Services, and Justice. The Initiative was developed in response to increasing public concern over school violence. Sites funded through the Initiative are required to establish comprehensive, integrated strategies to promote the healthy development of students and families in a safe school and community environment. This panel presents findings from the cross-site evaluation of the SS/HS Initiative that assessed the implementation and effects of the Initiative. Following an overview of the Initiative and the evaluation, presentations will address SS/HS partnership characteristics and functioning; school violence and safety; school and classroom climate; and student substance use. The panel will present final results of the evaluation, updating interim results previously presented.
|
|
Overview of the Safe Schools/Healthy Schools (SS/HS) Initiative and Cross-site Evaluation
|
| James Trudeau,
RTI International,
trudeau@rti.org
|
|
James Trudeau, the Principal Investigator of the SS/HS Cross-site Evaluation, will provide an overview of the SS/HS Initiative and the cross-site evaluation. Topics will include the Initiative rationale and site characteristics; evaluation design and logic model; and analytic techniques.
|
|
|
SS/HS Partnerships, Programs, and Policies
|
| Phillip Graham,
RTI International,
pgraham@rti.org
|
|
This presentation will examine changes in implementation and service integration as a function of how Safe Schools Healthy Students partnerships were structured and how well they functioned. Partnership-related indicators of structure and function included management and leadership, sector diversity, member participation, perceived leadership, perceptions of operations, barriers to key partner involvement and barriers to accomplishing goals. Data were collected from multiple sources including project directors; partnership members; key partners in education, law enforcement, and mental health; principals; and coordinators in violence prevention, substance abuse, and mental health promotion across all 97 SS/HS sites. Changes were tracked over three consecutive years beginning with the first year in which the SS/HS initiative was implemented in each site. These data were analyzed using hierarchical linear modeling (HLM) to assess change among indicators of structure and functioning; and to determine if those changes are associated with changes in implementation and service integration.
| |
|
School Violence and Safety in the Safe Schools/Healthy Students Initiative
|
| Angela Browne,
RTI International,
abrowne@rti.org
|
| James Trudeau,
RTI International,
trudeau@rti.org
|
|
This presentation examines school violence and safety in SS/HS sites, including change and associations with implementation. Outcomes include student victimization, witnessing, and fighting; teacher reports of bullying and fighting in the classroom, feelings of safety at school, and abuse by students (verbal abuse, threats, and physical attacks); and principal reports of serious violent infractions and fighting, and ratings of the severity of bullying problems and problem behaviors toward teachers. The presentation also identifies characteristics of implementation associated with school differences in outcomes. Measures of implementation, which are based on surveys of principals and violence prevention coordinators, assess many aspects of violence prevention, including comprehensiveness of curricula, staff professional development, parent training, partnering with external agencies, student exposure to prevention services, comprehensiveness of school rules, monitoring of building and grounds, use of surveillance and communication devices, use of school resource officers and other security staff, and consequences for violence-related infractions.
| |
|
School and Classroom Climate in the Safe Schools/Healthy Students Initiative
|
| Steve Murray,
RMC Research Corporation,
smurray@rmccorp.com
|
| Jason Williams,
RTI International,
jawilliams@rti.org
|
|
This presentation will examine changes in school and classroom climate across 97 Safe Schools/Healthy Student (SS/HS) Initiative sites. Changes are tracked over three consecutive years in each site. Climate variables derived from both student surveys and teacher surveys include orderliness, perceived safety, opportunities for student participation in decision making, and openness of communication between students and staff. In addition to examining changes in climate outcomes, the presentation will report findings from the application of statistical models to identify characteristics of implementation associated with school differences in climate changes. Measures of implementation, which are based on surveys of school principals and service coordinators, assess the availability and comprehensiveness of violence prevention, substance use prevention, and mental health services.
| |
|
Student Substance Use in the Safe Schools/Healthy Students Initiative
|
| James Trudeau,
RTI International,
trudeau@rti.org
|
| Jason Williams,
RTI International,
jawilliams@rti.org
|
|
This presentation examines student substance use in SS/HS sites, including change and associations with implementation. Student self-reported outcomes include use of alcohol, tobacco, marijuana and inhalants, at school and anywhere; binge drinking; availability of substances at school; and perceptions of normative use and peer attitudes. Teacher-reported outcomes include seeing students in possession of or under the influence of substances in the classroom. Principal-reported outcomes include the rate per 1,000 students of infractions involving tobacco, alcohol, and illegal drugs. Measures of implementation, which are based on surveys of principals and substance use prevention coordinators, assess many aspects of substance use prevention, including comprehensiveness of curricula, staff professional development, parent training, partnering with external agencies, student exposure to prevention services, comprehensiveness of school rules, monitoring of building and grounds, use of surveillance and communication devices, use of school resource officers and other security staff, and consequences for substance use-related infractions.
| |
|
Session Title: Can These Checklists Work? Simplifying Cost and Sustainability Evaluations
|
|
Think Tank Session 738 to be held in International Ballroom E on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Graduate Student and New Evaluator TIG
|
| Presenter(s):
|
| Lori Wingate,
Western Michigan University,
lori.wingate@wmich.edu
|
| Discussant(s):
|
| Nadini Persaud,
Western Michigan University,
npersaud07@yahoo.com
|
| Daniela C Schroeter,
Western Michigan University,
daniela.schroeter@wmich.edu
|
| Abstract:
Serious and sophisticated cost analyses and sustainability evaluations are two fundamental but often neglected aspects of professional evaluation. However, knowing that a program is responsible for certain outcomes is of little value in a political environment. The quintessential question at the end of the day pertains to "costs" and "sustainability." To thwart the neglect of these components of evaluation, this think tank introduces two checklists that can be used by novice and experienced evaluators to conduct cost and sustainability evaluations. After providing the audience with a brief overview of both checklists, attendees break into small groups to explore the strengths and weaknesses of each checklist. At the end of discussion both groups reconvene to share their enhanced understanding by providing constructive feedback to the checklist authors.
|
|
Session Title: Exploring Evaluation Expectations for Nonprofits, Foundations and Government: A Preview of an Upcoming New Directions for Evaluation Volume
|
|
Multipaper Session 739 to be held in Liberty Ballroom Section A on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Chair(s): |
| Joanne Carman,
University of North Carolina, Charlotte,
jgcarman@uncc.edu
|
| Discussant(s): |
| Kimberly Fredericks,
Indiana State University,
kfredericks@indstate.edu
|
| Abstract:
In recent years, many evaluators have observed that nonprofit organizations are under increasing pressure to demonstrate their effectiveness and document their program outcomes, as the current political and funding environment continues to stress the importance of accountability and measuring performance. Foundations, government agencies, and other funders are asking nonprofit organizations for more evaluation and performance measurement data. Yet, most nonprofit organizations continue to struggle with these demands, and many lack the capacity to implement evaluation and performance measurement in comprehensive or meaningful ways. This panel will bring together the authors of an upcoming NDE volume to discuss the current state of evaluation practice among nonprofit organizations, as well as the different expectations for nonprofit evaluation from the perspective of nonprofit leaders and various types of funders, including foundations, the federal government, and the United Way. The authors will discuss their findings and offer recommendations for reconciling these differing expectations.
|
|
Nonprofits and Evaluation: Empirical Evidence From the Field
|
| Joanne Carman,
University of North Carolina, Charlotte,
jgcarman@uncc.edu
|
| Kimberly Fredericks,
Indiana State University,
kfredericks@indstate.edu
|
|
During the last fifteen years, nonprofits have faced increasing pressures from stakeholders to demonstrate their effectiveness, document program outcomes, and improve their accountability. The purpose of this paper is to provide an empirical description about what evaluation practice looks likes among today's nonprofit organizations and to provide empirical evidence of current thinking and practice within the field. This paper presents a review of the empirical data that has been gathered and published within the literature about the current program evaluation practices of nonprofit organizations; presents descriptive data about evaluation practice gathered from a large mail survey of nonprofit organizations; and discusses a three-pronged typology about what nonprofit organizations think about evaluation, based upon a factor analysis of questions from the survey data. Finally, the paper offers recommendations about how evaluators can work with each type of organization to expand their thinking about evaluation and build evaluation capacity.
|
|
Nonprofits and Evaluation: Managing Expectations From the Leader's Perspective
|
| Sal Alaimo,
Indiana University,
salaimo@comcast.net
|
|
The call for greater accountability in the nonprofit sector continues to impact funders, board members, individual donors, evaluators and executive directors of nonprofit organizations. While this call has predominantly been framed around issues of fiscal responsibility, programmatic responsibility through evaluation has garnered an increasing role in an organization's overall accountability. Funders, accreditation organizations, government agencies and other stakeholders are increasingly asking for nonprofit organizations to provide information about the effectiveness of their programs. This request or requirement comprises an external pull emanating from the organization's stakeholders. Variance in expectations from these stakeholders results from different perceptions, beliefs, norms and parameters of their relationships with nonprofit organizations. This analysis uncovers whether leaders respond to the external pull of their important stakeholders, help drive an 'internal push' for evaluating programs where it is an intrinsic value embedded in their organization's culture, or attempt to balance the external pull with the internal push.
|
|
Foundations' Expectations (and Capacity) to Support, Conduct, and Use Evaluations
|
| Thomas Kelly,
Annie E Casey Foundation,
tkelly@aecf.org
|
|
The philanthropic sector has always included a variety of types of donors and endowments with many different grant making approaches and expectations across community foundations, private grant making foundations, private operating foundations, and other endowments. Within the past 10 years, we have seen an expansion both in the numbers and types of foundations and in the approaches they take to grant making. The diversity in structure, mission, and operations within the sector mirrors the diversity in approaches to and expectations for evaluation. Most importantly, foundations and other funders have affected the practice of evaluation, including its vocabulary, budgets, and methodology, through the expectations they communicate via their funding behaviors and requirements. This paper addresses these recent developments that have raised or altered expectations for the purposes and uses of evaluation as well as focus on the challenges that foundations face in implementing relevant evaluations.
|
|
Evaluation and the Federal Government
|
| David Introcaso,
United States Department of Health and Human Services,
david.introcaso@hhs.gov
|
|
This paper will provide an overview of the current state of evaluation and evaluation practice methods within the executive branch of the federal government. The paper will: discuss current expectations for and results from evaluating government program performance; discuss inherent problems or assumptions associated with performance expectations; and, describe specifically how the field is currently being practiced across various federal agencies. A main issue of focus will be on the implementation of the Office of Management and Budget's Program Assessment Rating Tool (PART). The PART's rationale and inherent assumptions will be examined and will be challenged by arguing that this model of planning frequently or largely conflicts with lived experience or with a complex, political environment where high levels of goal ambiguity are inherent. Suggestions for what can be done to avoid this trap or better align planning expectations with performance will be discussed.
|
|
United Way Experiences in Measuring Program Outcomes and Community Impact
|
| Michael Hendricks,
Independent Consultant,
mikehendri@aol.com
|
| Margaret Plantz,
United Way of America,
meg.plantz@uwamail.unitedway.org
|
|
Over the past ten years, the United Way (UW) system has been an influential force for greater evaluation and accountability within the nonprofit sector. More than 400 local United Ways (UWs) have helped approximately 19,000 local nonprofit agencies to identify the outcomes they intend for their clients to achieve as a result of their services, measure those outcomes, analyze the results, and use that information to improve their effectiveness and to document their accomplishments. This influence has spread beyond the UW system itself through the use of the popular manual Measuring Program Outcomes: A Practical Approach. Despite this overall success, however, local UWs and agencies still face undeniable challenges in measuring outcomes and using the findings. It requires skills not readily available, time and energy not always present, and local funders use different concepts and reporting requirements. Making funding decisions based on outcome data can be difficult for UW staff and allocations volunteers. This paper discusses these issues and presents future directions in the field.
|
|
Session Title: Practical Arguments, Checklists, and Meta-Evaluation: Tools for Improving Evaluation Theory and Practice
|
|
Multipaper Session 740 to be held in Liberty Ballroom Section B on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Theories of Evaluation TIG
|
| Chair(s): |
| Bernadette Campbell,
Carleton University,
bernadette_campbell@carleton.ca
|
|
The Relation Between the Application of the Process Specific to Program Evaluation and the Quality of Judgments and Recommendations
|
| Presenter(s):
|
| Marthe Hurteau,
Université du Québec à Montréal,
hurteau.marthe@uqam.ca
|
| Stéphanie Mongiat,
Université du Québec à Montréal,
smongiat@hotmail.com
|
| Sylvain Houle,
Université du Québec à Montréal,
houle.sylvain@uqam.ca
|
| Abstract:
Based on Scriven's (1980) “Logic of Evaluation”, Fournier (1995) and Stake's (2004) works, Hurteau & Houle's (2005) and Hurteau, Lachpelle & Houle's (2006) researches has led to the conception and validation of a modeling of the process specific to program evaluation. The results also emphasize on the fact that evaluators don't often refer to standards or rely on implicit ones to elaborate their claims, which has an impact on the quality of judgments and recommendations. In another context, and using a different methodology, Arens (2006) reaches the same conclusion concerning the use of standards. Further researches, based on the analysis of 40 evaluation reports, allow Hurteau, Mongiat & Houle to conclude that evaluators don't systematically refer to the modeling and that these considerations seem to have an impact on the quality of judgments and recommendations. Researchers will present the modeling in question and the results of their work.
|
|
The Logic of Practical Arguments in Evaluation
|
| Presenter(s):
|
| Nick L Smith,
Syracuse University,
nlsmith@syr.edu
|
| Abstract:
Although evaluation logic usually refers to the logic of research designs, the real logic of evaluation concerns the construction of convincing evaluation arguments that assist clients in making decisions and taking action. This paper presents a strategy for constructing local, context-sensitive, and case-specific evaluation arguments. Five aspects are considered: (1) the essential characteristics of client-centered evaluation practice, (2) the types of claims comprising evaluation arguments, (3) the levels of evidence associated with various claims, (4) how claims form lines of argument, and (5) the criteria for comparative evaluation of multiple lines of argument. Collectively, these aspects provide a logic enabling evaluators to produce persuasive, case-specific evaluation arguments. Through a closer analysis of the nature of evaluation practice, and the influence of evaluation context on evaluation arguments, this paper contributes to the development of a fundamental form of logic for the development of practical arguments across any type of client-centered evaluation.
|
|
An Evaluation Checklist: Educative and Meta-evaluation Opportunities
|
| Presenter(s):
|
| Jennifer Greene,
University of Illinois at Urbana-Champaign,
jcgreene@uiuc.edu
|
| Lois-ellin Datta,
Datta Analysis,
datta@ilhawaii.net
|
| Jori Hall,
University of Illinois at Urbana-Champaign,
jorihall@uiuc.edu
|
| Jeremiah Johnson,
University of Illinois at Urbana-Champaign,
jeremiahmatthewjohnson@yahoo.com
|
| Rita Davis,
University of Illinois at Urbana-Champaign,
-
|
| Lizanne DeStefano,
University of Illinois at Urbana-Champaign,
destefano@uiuc.edu
|
| Abstract:
Evaluation checklists offer succinct statements of the ambitions, intentions, and commitments of a given approach to evaluation, as translated into guidelines for evaluation practice. Checklists can be used to plan and guide evaluation implementation and to assess the quality of a particular evaluation (meta-evaluation). Evaluation checklists also have educative and capacity building potential, both as a general resource and as implemented in tandem with an evaluation study, e.g., through self-conscious attention to checklist items throughout an evaluation study. This paper offers a window into both the educative and meta-evaluative functions of evaluation checklists. As part of a field test of an educative, values-engaged approach to STEM education evaluation, a checklist representing the key features of this approach is being used to guide both a meta-evaluation and self-reflective practice by the evaluation team. The paper will highlight the contributions of the checklist to the quality and meaningfulness of evaluation practice.
|
|
Using the Metaevaluation Synthesis to Improve the Quality of State-level Evaluations
|
| Presenter(s):
|
| Paul Gale,
San Bernardino County Superintendent of Schools,
ps_gale@yahoo.com
|
| Abstract:
The presenter will briefly illustrate the rationale and methods to conduct a Metaevaluation Synthesis. The intent is for professionals to learn about a systematic, replicable method of integrating several metaevaluations of a single program's series of evaluations to create five indicators of quality. These indicators are standards-driven and based on the Program Evaluation Standards (1994) attributes of utility, feasibility, propriety, and accuracy. As such, they may be used to guide changes in evaluative practice and requirements as they provide the basis to identify systemic strengths and weaknesses of a program's evaluations. Although the specific context was predicated on the Comprehensive School Reform Program (CSRP), the method may be applied to multiple summative evaluations of any program.
|
| | | |
|
Session Title: Unintended Consequences of Program Action: When are They Problematic for Evaluation, and What Can We Do About Them?
|
|
Think Tank Session 741 to be held in Mencken Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Systems in Evaluation TIG
|
| Presenter(s):
|
| Jonathan Morell,
NewVectors LLC,
jamorell@jamorell.com
|
| Discussant(s):
|
| Glenda Eoyang,
Human Systems Dynamics Institute,
geoyang@hsdinstitute.org
|
| Deborah Wasserman,
The Ohio State University,
wasserman.12@osu.edu
|
| Abstract:
Unintended consequences of program action are ubiquitous. However, these consequences do not always threaten the integrity of evaluation, or the ability of evaluators to serve their clients and meet the needs of the program's stakeholders. This session will present a framework for identifying when unexpected program activity is problematic for evaluation, and will outline tactics that may ameliorate the difficulty. The intent is to engage as wide a range of opinion as possible in discussion to advance our ability to deploy tactics that can buffer the effectiveness of evaluation against unanticipated program activity or impact.
|
|
Session Title: Raising the Bar: What Role for Accreditation and Certification in Improving the Quality and Accountability (Q&A) of International Non-government Organization (INGO) Humanitarian Work?
|
|
Think Tank Session 742 to be held in Edgar Allen Poe Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Disaster and Emergency Management Evaluation TIG
|
| Presenter(s):
|
| Guy Sharrock,
Catholic Relief Services,
gsharroc@crs.org
|
| Pauline Wilson,
CARE USA,
pauline_wilson@wvi.org
|
| Discussant(s):
|
| Jock Baker,
CARE International,
baker@careinternational.org
|
| Daryl Martyris,
Save the Children USA,
dmartyris@dc.savechildren.org
|
| Malaika Wright,
CARE International,
mwright@care.org
|
| Abstract:
While many humanitarian sector standards emphasize the importance of accountability to beneficiaries, INGOs tend to focus on donor accountability. Recent efforts of Q&A networks have been striving to ensure standards keep aid recipients at the heart of disaster response interventions. Despite this, beneficiaries consistently cite dissatisfaction with the quality of humanitarian aid.
There are now pressing demands for the international relief community to establish an accreditation system to improve the quality of their humanitarian response. There are advantages and disadvantages to this. Nonetheless participants at a recent conference noted that, "the trend towards certification [in other sectors] means that it will be impossible to prevent it in the humanitarian sector." Many speak of INGOs needing to act collectively to develop and control a regulatory process before it is imposed upon them. Teasing out the accreditation issues and challenges that await the humanitarian community is the focus of the Think Tank.
|
|
Session Title: Navigating Subjectivity in the Evaluation Process
|
|
Multipaper Session 743 to be held in Carroll Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Qualitative Methods TIG
|
| Chair(s): |
| Jennifer Jewiss,
University of Vermont,
jennifer.jewiss@uvm.edu
|
| Discussant(s): |
| Jennifer Jewiss,
University of Vermont,
jennifer.jewiss@uvm.edu
|
|
What do you Mean When I say…: A Serendipitous Lesson in Evaluation Design
|
| Presenter(s):
|
| Dan Kaczynski,
University of West Florida,
dkaczys@uwf.edu
|
| Leigh Wood,
Macquarie University,
leigh.wood@mq.edu.au
|
| Abstract:
This paper explores the unexpected significance of terminology in evaluation design. The study was an external formative evaluation sponsored by an Australian university interested in critically examining their implementation of enrollment changes in undergraduate course delivery. The evaluation focused on documenting changes in instructional delivery related to enrollment reduction and establish baseline data on student performance and satisfaction.
During question construction of the qualitative interview guide the external evaluator developed a translation key of academic vocabulary terms. This step was necessary because the external evaluator was from the United States. Although the external evaluator and the stakeholders spoke English, significant differences in meanings of Australian English and United States English became immediately apparent. This design issue was initially considered necessary but insignificant. To the contrary, the translation key revealed more complex meanings during data collection and led to insightful serendipitous results.
|
|
Using Qualitative Methods to Negotiate Racial Identity and Insider/Outsider Status With Stakeholders and Participants
|
| Presenter(s):
|
| Corrie Davis,
Georgia State University,
cdavis@gsu.edu
|
| Sheryl Gowen,
Georgia State University,
sgowen@gsu.edu
|
| Syreeta Skelton,
Georgia State University,
snskelton@gsu.edu
|
| Abstract:
Evaluators that employ qualitative methods are charged with the responsibility of providing voice to participants. Before any interviews or observations are conducted, evaluators must first negotiate between their racial identity and the identity of the various stakeholders in the project. As the lines between an insider and outsider become blurred, it is imperative that a discussion of race enter the dialogue regarding quality evaluative methods. Utilizing examples and experiences from the authors' (with different racial identities) work on a large-scale, multi-site grant, this presentation will focus on how the dynamics of race and ethnic identity mediate qualitative evaluation process and findings.
|
|
Do You See What I See? Do You Hear What I Hear?: Researcher's Role and Subjectivity in Fieldwork Evaluation Experiences, a Student Researcher's Perspective
|
| Presenter(s):
|
| Quiana Cutts,
Georgia State University,
qcutts1@gsu.student.edu
|
| Janice Fournillier,
Georgia State University,
jfournillier@gsu.edu
|
| Sheryl Gowen,
Georgia State University,
sgowen@gsu.edu
|
| Abstract:
The lenses through which we view the world shape our perceptions, experiences, and ideas. Inasmuch as these lenses are created from varied classed, raced, and gendered identities, they under gird a researcher/evaluator's assumptions when conducting research and doing program evaluation. In this respect, researchers/evaluators must be willing to openly address issues of class, race, and gender; and scrutinize how those various identities affect perception of data collected in fieldwork experiences. As evident, researchers/evaluators are encouraged to be self-reflexive with self-reflexivity considered as a problematic, nuanced, complex, and critical entity. In this paper, we focus on the affects of researcher subjectivity on fieldwork experiences in program evaluation as well as the researcher/evaluator's self-reflective practices. By considering subjectivity within this methodological framework, we hope to provide a depiction of the complex nature of researcher self-reflexivity in evaluation research.
|
| | |
|
Session Title: Evaluating Customer Satisfaction: How Public and Private Sector Organizations and Web-managers Learn and Drive their Behavior From the American Customer Satisfaction Index (ACSI)
|
|
Panel Session 744 to be held in Pratt Room, Section A on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the AEA Conference Committee
|
| Chair(s): |
| Ted Kniker,
Federal Consulting Group,
ted.kniker@bpd.treas.gov
|
| Abstract:
Increasing customer satisfaction, trust and confidence is the driving force behind how private and public sector organizations measure results. However, often customer satisfaction evaluation is reduced to collecting and collating "smile sheets." This session will showcase the American Customer Satisfaction Index (ACSI), and how public and private sector organizations and websites are using it to predict how their actions will affect satisfaction and customer behavior in the future, to focus their resources on what matters most, and to benchmark against similar organizations and programs. The session will provide an overview of the ACSI and its methodology, how public and private sector organizations and websites are using the ACSI to evaluate their effectiveness, and how session participants can use it to improve their activities. Participants will specifically learn what drives customer satisfaction, how customer satisfaction differs between the public and private sector, how satisfaction can be used to fulfill performance measurement mandates, techniques to identify customers' priorities and the services they value, and how to analyze the data to make decisions on where to place resources for improvement efforts.
|
|
The American Customer Satisfaction Index: It's History and Findings
|
| David VanAmburg,
University of Michigan,
asc@umich.edu
|
|
David VanAmburg is the Managing Director of the American Customer Satisfaction Index (ACSI) at the Ross School of Business at the University of Michigan. Mr. VanAmburg has acted as Director of the federal government-wide satisfaction measurement since its inception in 1999 and of the ACSI as a whole since 2001. He has lectured extensively both at the University of Michigan and abroad on topics related to customer satisfaction, quality, customer loyalty and shareholder value. Mr. VanAmburg will present the history of the ACSI measurement 1999-2007 as it relates to the federal government, discussing how the ACSI government model was created and implemented in collaboration with individual federal agencies as well as a brief summary of findings from nearly a decade of government customer satisfaction research.
|
|
|
Customer Satisfaction Measurement and Financial Outcomes: Use of the American Customer Satisfaction Index in the Private Sector
|
| Sheri Teodoru,
Claes Fornell International Group,
steodoru@cfigroup.com
|
|
Sheri Teodoru is a Partner and Board Member at CFI Group USA, L.L.C., a research consulting firm dedicated to helping clients tie customer measurement to financial performance. Ms. Teodoru brings more than 20 years of research and consulting experience to the panel, and has worked with an array of public and private sector entities. Ms. Teodoru will be discussing the role of the customer satisfaction in the private sector. She will present how customer satisfaction, on a macro level, links to financial performance and to consumer spending, and for an individual firm how a robust satisfaction measurement program can be used to make resource allocation decisions and to predict the financial outcome of those decisions.
| |
|
Citizen Satisfaction: How the Federal Government is Using the American Customer Satisfaction Index (ASCI) to Create a Citizen-Centered, Results Oriented Government
|
| Susan Grow,
Federal Consulting Group,
susan.grow@bpd.treas.gov
|
|
Susan Grow is a Performance Management Consultant at Federal Consulting Group (FCG) a revolving fund franchise within the federal government focused on helping agencies develop and maintain a more customer-driven, results-oriented government. Susan brings more than 25 years of experience in performance and customer/employee satisfaction measurement, quality management and evaluation, strategic planning and performance management in both the public and the private sectors. She will present how the ACSI is customized for the federal government and how it has been used within the government to evaluate programs, improve program performance and meet government performance mandates.
| |
|
Beyond Page Views and Hits: Evaluating User Satisfaction with Websites
|
| Errol Hau,
ForeSee Results Inc,
errol.hau@foreseeresults.com
|
|
Errol Hau is the Senior Director of Public Sector Services at ForeSee Results, Inc. Mr. Hau is the business line manager for all Public Sector activity including quality control, solution development, channel management, and market growth. In this role, he manages a multi-disciplinary staff including sales, partner management, client services and research. Since 2002, Mr. Hau has been consulting with both public and private sector clients regarding their utilization of the ACSI methodology to evaluate and optimize the effectiveness of their websites. Mr. Hau will discuss the variety of ways in which organizations from government agencies to Fortune 2000 companies have benefited from their use of an attitudinal-based website measurement. Mr. Hau will also review specific initiatives in the area of web analytics that are further advancing the use of ACSI metrics.
| |
|
Session Title: Using a Protective Factor Framework to Measure Program Outcomes and Build Capacity in Community-based Family Support and Prevention Programs
|
|
Skill-Building Workshop 745 to be held in Pratt Room, Section B on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Human Services Evaluation TIG
|
| Presenter(s):
|
| Hallie Goertz,
Organizational Research Services,
hgoertz@organizationalresearch.com
|
| Maria J Gehl,
Children's Trust of Washington,
Maria@wcpcan.wa.gov
|
| Abstract:
How do you know if a community-based family support and prevention program is benefiting its participants? Measuring these efforts can be challenging. In this workshop participants will learn how to utilize a protective factor framework to guide program evaluation. Participants will: become familiar with individual or family outcomes that can result from family support and prevention programs, including short- and longer-term changes in knowledge, attitudes, skills, or behaviors, discuss how these outcomes connect to a protective factor framework and review measurement approaches. Participants will also discuss the connection between program evaluation and program improvement, and how evaluation data supports program capacity-building. The material presented will be based on real examples from family support and prevention programs in Washington state and Hawaii. Participants will be given opportunities to apply these techniques through a combination of individual and small group activities. This workshop will provide useful information both beginners and experienced practitioners.
|
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Using Indigenous Guidelines to Analyze Native Hawaiian Community-based Initiatives: How Does This Work? |
|
Roundtable Presentation 746 to be held in Douglas Boardroom on Saturday, November 10, 10:30 AM to 12:00 PM
|
| Presenter(s):
|
| Alice Kawakami,
University of Hawaii,
alicek@hawaii.edu
|
| Abstract:
The articulation of the Decolonizing Evaluation Framework and the Guidelines for Culturally Healthy and Responsive Learning Environments may be viewed as tools for describing and identifying key elements of successful indigenous community-based projects. A community-based early childhood needs assessment in Hawaiian communities and a pre-service teacher recruitment and education program will be discussed in terms of common characteristics that align with the framework and guidelines. This session will focus on the opportunities for clarity as well as the dilemmas resulting from blending perspectives of a holistic cultural nature with Western methods of qualitative analysis. Roundtable participants will be encouraged to engage in dialogue around these issues.
|
| Roundtable Rotation II:
How Tribal Colleges Help Struggling Students Online |
|
Roundtable Presentation 746 to be held in Douglas Boardroom on Saturday, November 10, 10:30 AM to 12:00 PM
|
| Presenter(s):
|
| Paule Steichen Asch,
Individual Building of Integrated Success International,
asch@ibisintl.com
|
| Abstract:
Tribal colleges are a main resource for (1) preserving Native Americans' heritage and (2) preparing individuals for success. Indeed education is the very key to achieving these goals; and such is the mission of tribal colleges.
This roundtable examines how students of tribal colleges, struggling with reading, writing, math and study skills in particular, are assisted through online help on their college's website. A detailed analysis of these websites will be presented. A comparative approach will encourage discussion and cross-utilization of resources.
|
|
Session Title: Participatory Evaluation Approaches With Latino and Indigenous Populations
|
|
Panel Session 747 to be held in Hopkins Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Indigenous Peoples in Evaluation TIG
|
| Chair(s): |
| David Dobrowski,
First 5 Monterey County,
david@first5monterey.org
|
| Discussant(s):
|
| Edward Kissam,
JBS International Inc,
ekissam@jbsinternational.com
|
| Abstract:
A growing segment of the farm worker and U.S. populations are from indigenous communities from the southern Mexican states of Oaxaca, Chiapas and Guerrero. These individuals often speak no English and limited Spanish. They also have cultures which differ from Spanish-dominant communities and from each other, which can create communications challenges that go beyond language alone. Providing and evaluating health, education and social services to this population requires culturally and linguistically appropriate approaches. Participatory evaluation approaches can help improve the ability of these communities to access services and provide feedback. This session will describe three participatory evaluation methods with indigenous and/or Latino immigrant populations in Northern California.
|
|
Inclusion of Indigenous Community Members in County-wide Participatory Strategic Planning Process
|
| David Dobrowski,
First 5 Monterey County,
david@first5monterey.org
|
|
First 5 Monterey County believes that community members hold the knowledge needed to define needs for resource allocation and capacity building. Therefore a participatory strategic planning process has defined the community visions and outcomes. Between March-December 2006 more than 1,000 individuals from diverse communities in Monterey County participated with their voices, experiences, and perspectives in the design of the new strategic plan. These included iterative large town hall meetings including Triqui and Mixteco community members. We will discuss ways in which funding processes have been created to be more inclusive of indigenous communities, how participation in community-wide events was facilitated and how programs were designed in culturally appropriate ways.
|
|
|
Participatory Evaluation Techniques Training Indigenous Community Members to Facilitate Focus Groups: Process, Key Findings, and Lessons Learned
|
| James Glasnapp,
JBS International Inc,
jglasnapp@jbsinternational.com
|
| Amparo Bustos-Navarro,
JBS International Inc,
abustos@jbsinternational.com
|
|
As the Mexican indigenous population continues to grow in Monterey County, California, so do the challenges facing organizations offering services to them. First 5 Monterey County (F5MC) wanted to evaluate the effectiveness of their services for indigenous Mexican immigrants of Triqui and Mixteco communities, using a participatory approach that capitalized on the skills of local indigenous community members. The goal of this evaluation was to identify the impact of services provided by two community based organization funded by F5MC that serve indigenous populations. This session will describe the participatory evaluation approach used, describe the training that occurred with local indigenous bilingual volunteers (in Spanish and their indigenous language of Mixteco or Triqui), present key results and lessons learned.
| |
|
The Use of Photovoice as a Participatory Evaluation Method in Diverse Communities
|
| Maricela Piña,
Harder & Company Community Research,
mpina@harderco.com
|
|
Evaluation operates in dynamic environments that are constantly reconfigured to adapt to the economic, social, political, and cultural realities. Evaluations must respond to current contextual factors and community needs. Use of participatory evaluation methods in working with culturally diverse populations is one approach that has been used to respond to such needs and contexts. Photovoice is an innovative participatory evaluation method pioneered by Caroline Wang in her work with underrepresented communities. This method actively engages participants to document their own realities including the strengths and concerns of their communities through the use of cameras and reflection. Photovoice offers evaluators a potential method for ensuring that all stakeholder's voices are captured. This presentation will discuss the use o fPhotovoice in a diverse urban community and highlight the lessons learned from implementing this method. Potential applications of Photovoice with rural indigenous and/or Latino immigrant populations will also be discussed.
| |
|
Session Title: Incorporating Development Evaluation Strategies in Education Reform Projects: Opportunities and challenges
|
|
Panel Session 748 to be held in Peale Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Chair(s): |
| John Gillies,
Academy for Educational Development,
jgillies@aed.org
|
| Abstract:
Monitoring and evaluation in USAID development projects has been undervalued for decades. The great majority of programs have limited M&E to donor reporting against standardized performance indicators. In complex challenges such as education reform, this has greatly limited the opportunity to learn and create a climate of continuous improvement.
The EQUIP2 program in AED has sought to incorporate evaluation as an integral part of the development process in several education sector reform projects. Consistent with Michael Quinn Patton's concept of development evaluation, AED has incorporated M&E in a comprehensive framework for sector reform. The panel will discuss the framework and the experience and challenges of promoting effective evaluation in USAID projects.
|
|
The Role of Monitoring and Evaluating (M&E) in a Framework for Education Reform
|
| John Gillies,
Academy for Educational Development,
jgillies@aed.org
|
| Audrey-Marie Moore,
Academy for Educational Development,
amoore@aed.org
|
|
The EQUIP2 framework for education sector reform identifies the key elements in three dimensions that affect system change; technical, political, and institutional. Monitoring and evaluation tools provide the linkages that inform decisions and help create space for change. This presentation will explore the role of evaluation in creating and promoting system change. This framework will be used in discussions of cases in the other panel presentations.
|
|
|
Empowerment Through Self Assessment Evaluation in Namibia
|
| Donna Kay Lezcel,
Academy for Educational Development,
dleczel@aed.org
|
|
Education reform in Namibia has effectively used self assessment tools to schools to support professional development, improve school performance, and strengthen supervisory support for schools. The evaluation data has informed national dialogue and resulted in policy innovations.
| |
|
Supporting System Reforms in Georgia
|
| Jessica Jester Quijada,
Academy for Educational Development,
jquijada@aed.org
|
|
Education reforms in Georgia are largely driven by ideology and national aspirations rather than foundation technical knowledge. Creative evaluation approaches are being used to create a learning environment to channel impetus for reform in productive directions.
| |
|
Education Reform in Egypt: The Role of Monitoring and Evaluation
|
| Robert Burch,
Academy for Educational Development,
rburch@aed.org
|
|
Egypt is implementing system-wide reforms in the education system, intended to improve education outcomes in the classroom. Evaluation tools that were developed to measure and report on progress have contributed to system learning and capacity development.
| |
|
Session Title: Evaluating Electronic Advocacy and Communications
|
|
Panel Session 749 to be held in Adams Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Advocacy and Policy Change TIG
|
| Chair(s): |
| Julia Coffman,
Harvard Family Research Project,
jcoffman@evaluationexchange.org
|
| Abstract:
Advocates increasingly are harnessing the power of technology to achieve policy change and communicate more effectively in general. Tools such as email messaging and blogging are offering the opportunity for many people to be meaningfully, but easily, engaged in social change efforts. Evaluators in the advocacy and policy field are challenged by how to utilize these electronic tools for evaluation purposes. In particular, we need to know how to assess their use by advocates. In addition, we need to know how we can take advantage of them in our evaluation methodology. This panel will focus on tested approaches for tracking and interpreting electronic advocacy and communications efforts, and on ways evaluators can use technology for advocacy evaluation. Panelists will focus specifically on useful evaluation data linked to website usage, email messaging, and blogging.
|
|
Gauging Nonprofit Online Effectiveness
|
| Tarek Rizk,
Global Interdependence Initiative,
tarek.rizk@aspeninst.org
|
|
All modern advocacy and social change efforts recognize the Internet's value for engaging and mobilizing large numbers of individuals to action. Nonprofits incorporate the Internet into their advocacy strategies and create dynamic websites and electronic forums that allow users to engage with issues and with each other for purposes of exchanging information and participating in the policy process. These efforts generate a wealth of potentially useful data about the effectiveness of Web-based strategies, but it can be difficult to sort out which statistics to pay attention to and how to use them. This presentation will demonstrate what data evaluators and nonprofits can collect and use to learn whether and how their websites and online strategies are working.
|
|
|
But What do the Numbers Mean? Benchmarks for E-Advocacy and Fundraising
|
| Karen Matheson,
M+R Strategic Services,
kmatheson@mrss.com
|
|
In the for-profit dot.com world, the bottom line is easy to measure-it comes down to dollars and cents. For nonprofit organizations using the Internet and online communications, success is more difficult to define. Even when nonprofit electronic performance is tracked, questions still exist about what those numbers actually mean. This presentation will describe The eNonprofit Benchmarks Study from M+R Strategic Services and the Advocacy Institute, which offers key metrics and benchmarks for nonprofit online communications-including email advocacy and fundraising-that evaluators can use to interpret nonprofit online performance data. The presentation also will review two additional studies on this topic-the Online Marketing (eCRM) Nonprofit Benchmark Index Study from Convio and the donorCentrics Internet Giving Benchmarking Analysis from the Target Analysis Group and Donordigital-to identify shared findings from all three studies.
| |
|
Tracking Blogs to Gauge Buzz
|
| Julia Coffman,
Harvard Family Research Project,
jcoffman@evaluationexchange.org
|
|
Blogs are a dynamic medium that allow any Internet user to have a voice and become an active information generator rather than a passive consumer. Of the 147 million Americans who now use the Internet, 39 percent read at least one blog regularly. Unlike professional journalists or reporters, almost no bloggers are paid to identify and cover issues. Rather, these 'Internet storytellers' develop blogs on their own time, covering the issues that are most important to them personally. As a result, the discussion of issues in the blogosphere is more than an indicator of whether bloggers are aware of certain issues; it is an indicator that bloggers perceive those issues as important enough to engage with and act on. This presentation will describe how to use blog tracking as a methodology for gauging whether policy issues (that advocates focus on) are gaining enough visibility to generate 'buzz' in the blogosphere.
| |
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
California Partnership for Achieving Student Success (Cal-PASS): Data-driven Faculty Decision Making to Improve Transitions Across Segments |
|
Roundtable Presentation 750 to be held in Jefferson Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
| Presenter(s):
|
| Jordan Horowitz,
California Partnership for Achieving Student Success,
jhorowitz@calpass.org
|
| Abstract:
The California Partnership for Achieving Student Success is the only informational resource that creates regional partnerships among K-12 schools, community colleges, and universities through the sharing of student transcripts and performance information. Educators develop continuity and align curricula to make instructional improvements that better prepare students as they move from kindergarten to the university. Since 1998, over 2,500 educational institutions across California joined Cal-PASS.
Cal-PASS has established 37 Regional Consortia to keep the focus local. Professional Learning Councils of faculty, within discipline and across segments, operate within each to identify student achievement and transition issues, implement action plans, and assess effects. PLCs have been established in mathematics, English, English learners, science, and counseling.
Interested institutions sign data sharing agreements within their consortium. They submit student-level data to a central, FERPA compliant Cal-PASS database. Each student receives an encrypted, unique identification number in the system to track student groups across segments.
|
| Roundtable Rotation II:
Evaluating the Retention of First Generation Students in a Community College: A Collaborative Approach |
|
Roundtable Presentation 750 to be held in Jefferson Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
| Presenter(s):
|
| Liliana Rodriguez-Campos,
University of South Florida,
lrodriguez@coedu.usf.edu
|
| John Hoye,
Western Michigan University,
john.hoye@wmich.edu
|
| Maxine Gilling,
Western Michigan University,
maxine.gilling@wmich.edu
|
| Abstract:
Collaboration between stakeholders and evaluators is essential for achieving and meeting key goals. This paper describes the strategies used to improve the quality of a retention program, while applying the Model for Collaborative Evaluations (MCE). According to Rodriguez-Campos (2005), the MCE is a comprehensive framework that transforms evaluation into a joint responsibility process. This model guides collaborative evaluations in a precise, realistic, and useful manner. The MCE includes a set of six interactive components that helps establish priorities in order to achieve a supportive evaluation environment. Therefore, this model transforms evaluation into a joint responsibility process where the evaluators examine the role of collaboration to improve the quality of a retention program. In addition, they analyze the attributes that constitute "effective retention" as brought to light by this investigation. The decision for the practitioners to collaborate on the evaluation is aimed to make the evaluation truly learner-centered.
|
|
Session Title: Constraints and Promising Practices in Evaluating Social Norm Change: Lessons Learned From Tobacco Control
|
|
Panel Session 751 to be held in Washington Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Health Evaluation TIG
|
| Chair(s): |
| Jenica Huddleston,
University of California, Davis,
jhuddleston@ucdavis.edu
|
| Discussant(s):
|
| Robert LaChausse,
California State University, San Bernardino,
rlachaus@csusb.edu
|
| Abstract:
Evaluating social norm change is challenging, with little comprehensive data collected and long time-lags before data is available. Using the context of tobacco control, this panel will identify challenges to evaluating social norm change and provide some specific approaches to address these challenges. The panel draws on several perspectives. An overview of California's approach to social norm change, including the role of the statewide Evaluation Center and the resources developed by this Center to assist evaluation at the local level, will be provided. A project conducting tobacco control work at a local level within California will then discuss local issues in evaluating social norm change. A third perspective will be provided by researchers who are evaluating social norm change at a national level. To conclude the session, a discussant will identify lessons learned across the panel presentations and raise ongoing issues that still need to be addressed in the future.
|
|
California's Approach to Evaluating Social Norm Change in Tobacco Control
|
| Leslie Cooksy,
University of California, Davis,
ljcooksy@ucdavis.edu
|
| Jenica Huddleston,
University of California, Davis,
jhuddleston@ucdavis.edu
|
| Julie Elmen,
University of California, Davis,
jdelmen@ucdavis.edu
|
| Jean Lamming,
University of California, Davis,
jlamming@ucdavis.edu
|
| Jeanette Treiber,
University of California, Davis,
jtreiber@ucdavis.edu
|
| Maria Tuccori,
University of California, Davis,
mttuccori@ucdavis.edu
|
|
California's Tobacco Control Program, which started in 1990, is an established social norm change program. It seeks to 'indirectly influenc[e] current and potential future tobacco users by creating a social milieu and legal climate in which tobacco becomes less desirable, less acceptable, and less accessible' (California Department of Health Services, 1998). To evaluate its tobacco control interventions the California program uses a combination of logic modeling, recording of outputs (e.g., local policies passed), local level evaluation, and state-wide surveillance data. The presentation about California's approach to evaluating social norm change will describe the strengths and weaknesses of this approach, providing some tools that other public and nonprofit agencies could adopt.
Reference:
California Department of Health Services. 1998. A Model of Change: The California Experience in Tobacco Control. Sacramento, CA: California Department of Health Services.
|
|
|
Evaluating Social Norm Change at the Local Level: Sacramento County's Tobacco Control Work
|
| Jenica Huddleston,
University of California, Davis,
jhuddleston@ucdavis.edu
|
| Carlene Henriques,
Sacramento County Dept of Health Services,
henriques@saccounty.net
|
| Kristina Clinton,
Sacramento County Dept of Health Services,
clintonk@saccounty.net
|
| Megan Trautman,
Sacramento County Dept of Health Services,
trautmanm@saccounty.net
|
|
Sacramento County will share a local perspective on evaluating social norm change in their tobacco control work. Two 'hot' topics within the field will be emphasized-- smoke-free multi-unit housing and tobacco retail licensing. Assessing norm change can be difficult at the local level with time and budgetary constraints. While the state funding agency focuses on impacting norm change through enacting policies, there is often only enough time available to the local project, and data collected, to convince decision-makers to adopt policies, not to assess the longer-term or larger picture of the impact these policies have on norms within the community. During the last three-year funding cycle (2004-2007), and in the current funding cycle (2007-2010), this project has worked on the two topics identified above. Changes in evaluating these topics from one cycle to another will be addressed.
| |
|
Social Climate Analysis as a Tool to Measure and Monitor the Social Penetration of Tobacco Control.
|
| Robert McMillen,
Mississippi State University,
robert.mcmillen@ssrc.msstate.edu
|
| Nell Valentine,
Mississippi State University,
nell.valentine@ssrc.msstate.edu
|
|
State and national tobacco control programs have targeted political and social intermediate objectives. To facilitate planning and evaluation, the Social Climate Survey of Tobacco Control (SCS-TC) provides timely, comprehensive data about the attitudes and practices of U.S. adults. The SCS-TC includes items to measure progress toward intermediate objectives such as policy changes, changes in social norms, reductions in exposure of individuals to environmental tobacco smoke, and rejection of pro-tobacco influences. Seven annual cross-sectional household telephone surveys of national probability samples of US adults were conducted in the summers of 2000 to 2006. Completion rates for eligible respondents contacted ranged from 75% in 2000 to 87% in 2006. Most measured indicators improved between 2000 to 2006, although not for all populations. Fewer people who live in rural areas, people with lower levels of education, and residents of the South Census Region endorsed many of the social climate indicators.
| |
|
Session Title: Evaluation Approaches that Support Nonprofit and Foundation Learning
|
|
Multipaper Session 752 to be held in D'Alesandro Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Chair(s): |
| Beverly Parsons,
InSites,
bevandpar@aol.com
|
|
"Don't Test Me, Let Me Play!" Evaluation Alternatives to the Standard Pre- Post-test for Latina/o Adolescents
|
| Presenter(s):
|
| Lyn Paleo,
Evaluation Research and Training,
paleo@igc.org
|
| Gina Malan,
Central American Resources Center,
gina@carecensf.org
|
| Abstract:
Cuerpo Sano: Avtivate! is an after-school program of exercise and nutrition that offers children in San Francisco's Mission District opportunities for outings, field trips, sports/games, and cooking classes. The evaluation uses innovative data collection strategies well suited to the youth's interests and the program's dynamic characteristics. These strategies for assessing nutrition knowledge, cooking safety skills, sports skills, and sportsmanship will be presented and discussed for their practicality, participatory and interactive features, and validity.
|
|
Using Outcome Measurement for Organizational Learning: From Rhetoric to Reality
|
| Presenter(s):
|
| Sandra Ortega,
Kent State University,
sortega@kent.edu
|
| James W Altschuld,
The Ohio State University,
altschuld.1@osu.edu
|
| Abstract:
Outcome measurement is an evaluation tool that has been touted as assisting service providers and funders in determining program effectiveness. Over the past ten years, outcome measurement has become a requirement for most non-profit organizations. One of the primary benefits associated with outcome measurement is that it can be used as an organizational learning tool. The presenters will discuss their research regarding the use and misuse of outcome measurement as an organizational learning tool and the reasons why it has not lived up to all the hype. The presentation is based on the dissertation research of the first presenter which investigated the differences in perceptions and understanding of outcome measurement by non-profit funders and service providers in social services agencies in a large state. In that study, gaps between rhetoric and reality were observed and will be presented during the session to spur discussion among the participants.
|
| |
|
Session Title: Concept Maps, Focus Groups and Rasch Analysis: Converting the Ineffable Qualitative into Quantitative
|
|
Panel Session 753 to be held in Calhoun Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Ann Doucette,
George Washington University,
doucette@gwu.edu
|
| Abstract:
Concept mapping was performed to conceptualize and organize measure development for financial exploitation of the elderly. A feasibility study of a financial exploitation measure was done based on the concept map and expert panel input. This consisted of focus groups and cognitive interviews at ten community sites (e.g., senior centers and case coordination units) to develop procedures for obtaining sensitive information concerning financial exploitation and to develop new items and refine existing items assessing these areas. This lead to the development of construct maps. Subsequently, a full-scale field test of the financial exploitation measure was conducted with data collection on 200 clients and with a subsequent psychometric analysis of the resulting data.
|
|
Concept Map of Elder Financial Exploitation
|
| John Ridings,
Metropolitan Family Services of Chicago,
jridin3@uic.edu
|
|
This presentation describes the procedures used to develop a concept map for financial exploitation of the elderly. This includes discussions of: expert panel members, brainstorming activities and results, sorting and rating procedures and the interpretation session.
The interpretation session included the following activities: Review the items, identify regions of meaning, give unique names to the regions, decide on the final number of clusters, give unique names to the clusters, review ladder graphs, review box plots, develop recommendations, identify next steps.
|
|
|
Using Qualitative Data in the Design of a Measure for Financial Exploitation of the Elderly
|
| Madelyn Iris,
Council for Jewish Elderly,
micki.iris@cje.net
|
|
Financial Exploitation of elderly individuals (EFE) is a hidden and pervasive problem. Conceptually sound measurement models that take into account the complex nature of elder abuse are needed, as the signs and symptoms of EFE may be insights into a larger system of abuse that includes more damaging consequences if left undiagnosed and untreated. Up to this time, no attempt has been made to conceptualize the individual components of EFE in order to guide the development of a measurement model. This presentation describes how focus group and interview data are being used in the development of a measure of EFE. Findings from focus groups and cognitive interviews will be discussed. Participants in these groups include: (1) staff at agencies that serve with victims of elder abuse and other professionals, and (2) older persons who have been financially exploited, as well as other seniors.
| |
|
Developing Theoretical Construct Maps and Testing Them with Rasch Person/Item Maps
|
| Kendon Conrad,
University of Illinois, Chicago,
kjconrad@uic.edu
|
|
Construct maps (Wilson, 2005) are different from concept maps (Trochim, 2003), although each may be used in the development of the other. Construct maps depict a theory-based hierarchy of both person characteristics and items along the same line representing the construct of interest. In this presentation, financial exploitation of elders will be depicted using a construct map that employs items developed with concept mapping techniqes. Subsequently, a Rasch person/item map based on data from elderly persons in adult protective services will be examined as to whether it confirms or denies the theory-based construct map. These techniques are sophisticated processes for developing measures essential to effective evaluation research.
| |
|
Session Title: Workgroup Solutions for Data Collection, Storage, Analysis, and Reporting With FileMaker Pro 8.5, Adobe Acrobat 8, and Microsoft Excel
|
|
Demonstration Session 754 to be held in McKeldon Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Integrating Technology Into Evaluation
|
| Presenter(s): |
| Susan Saka,
University of Hawaii,
ssaka@hawaii.edu
|
| Susan York,
University of Hawaii,
yorks@hawaii.edu
|
| Milton Shak,
1:1 Inc,
milts@earthlink.net
|
| Abstract:
What would be on your data management wish list? Ours includes the ability to develop new databases without having a dedicated expert on staff, in a program that is user-friendly across platforms and older operating systems; we need to input data using savable Adobe Reader forms that are rights-enabled, and then easily transfer the data between data-analysis programs. Our databases need to easily adapt for a variety of cultural considerations and peculiarities of location, while meeting the reporting needs of state and national funding agencies. We wanted to control the data "in-house" instead of paying for off-site storage and management, and to generate reports and form letters without having to merge data into a separate word-processing program. Experience four relational databases that cost-effectively fulfilled our wish list-presented first from the "non-techie" evaluators' perspective, followed by a closer look at the databases' design and structure presented by our FileMaker consultant.
|
|
Session Title: Strategic Approach to Developing Sustainable Infrastructure for Program Evaluation in an Organization: Multiple Perspectives
|
|
Multipaper Session 755 to be held in Preston Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Organizational Learning and Evaluation Capacity Building TIG
|
| Chair(s): |
| Bret Feranchak,
Chicago Public Schools,
bferanchak@cps.k12.il.us
|
| Abstract:
Building internal program evaluation capacity in organizations can be difficult. This panel presentation will examine one specific aspect of this capacity building-developing sustainable program evaluation infrastructure-through the multiple perspectives of actors comprising different roles (e.g., an organization's internal evaluation director, internal staff evaluator, external evaluator, graduate student intern) in one such effort. This effort is grounded in the context of a large urban school district, but the issues raised and strategies implemented would be applicable to many other types of organizations, especially those in the public or non-profit sectors. The panelists will discuss such issues as recruitment and development of evaluation staff; procuring and leveraging resources (human, fiscal, time, and space) for program evaluation activities; interactions with stakeholders; and approaches that increase the likelihood of longer term sustainability of the work. We will use the discussion period to engage other evaluators to reflect on their own strategies as a means to increase evaluation capacity in organizations.
|
|
The Internal Management Perspective
|
| Bret Feranchak,
Chicago Public Schools,
bferanchak@cps.k12.il.us
|
|
This presentation will focus on the theoretical program evaluation framework that guides the work of this organization. The presenter, who is the Director of Program Evaluation for the organization, will discuss key issues in the design of the theoretical framework, the strengths and limitations of the approach, and practical issues in the implementation of the framework including how to recruit, develop, and supervise internal evaluation staff; increase buy-in for program evaluation activities from key district stakeholders; and procure and leverage resources (human, fiscal, time, and space) for program evaluation activities. The presentation will be delivered through the lens of a management perspective.
|
|
The Internal Evaluator Perspective
|
| Kelci Price,
Chicago Public Schools,
kprice1@cps.k12.il.us
|
|
This presentation will focus on the role of the internal evaluator in building capacity within their own organization. The presenter is a staff evaluator in the Department of Program Evaluation and will discuss key issues around capacity building as experienced by the evaluation staff. These issues include the development of new and junior staff, balancing ad hoc data needs with longer term evaluation projects, and engaging different stakeholder groups. In addition, this presentation will address the very practical problems of creating systems for sharing skills and knowledge between evaluators, and of developing data management skills within stakeholder groups to facilitate on-going and formative evaluation.
|
|
The External Evaluator Perspective
|
| Stacy Wenzel,
University of Illinois, Chicago,
swenzel@uic.edu
|
|
This presentation will focus on the role that external evaluators play as part of building evaluation capacity within an organization. The presenter is part of an evaluation group who serves as an external evaluator for many of the district's evaluation projects. This presentation will focus on building evaluation capacity through responsive evaluation by external groups, including issues of providing timely data in sync with stakeholders' decision-making needs, promoting utilization through the engagement of multiple stakeholder groups, and adding capacity by bringing to the internal evaluation group a different set of skills.
|
|
The Graduate Student Intern Perspective
|
| Natalya Gnedko,
Chicago Public Schools,
ngnedko@cps.k12.il.us
|
|
This presentation will focus on the role of the graduate student intern in evaluation capacity building. The presenter is an intern in the Department of Program Evaluation whose position is funded by a local university with whom the department has established a working relationship. This mutually beneficial situation allows the university to place one of their students in an applied setting where they can gain practical research experience, while simultaneously benefiting the district with the services of an additional researcher. This presentation will focus on developing skills as a researcher through the internship experience, how the structure and vision of the department influence learning experiences, and the tradeoffs between gaining practical experience while still undertaking a formal education.
|
|
Session Title: Case Studies in Business and Industry Evaluation
|
|
Multipaper Session 757 to be held in Calvert Ballroom Salon B on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Business and Industry TIG
|
| Chair(s): |
| Otto Gustafson,
Western Michigan University,
ottonuke@yahoo.com
|
|
The Blind Men and the Elephant: Making Sense of Knowledge Management
|
| Presenter(s):
|
| Thomas E Ward,
United States Army Command and General Staff College,
tewardii@aol.com
|
| Abstract:
“Knowledge Management” means different things to different people. The lack of definitional agreement is due largely to different perspectives on the subject, and different expectations for outcomes. This paper presents a three-tier domain model that integrates perspectives of knowledge management. The domain model consists of an infrastructure layer, an information management layer, and a knowledge management layer, differentiating the stakeholders and their concerns in each layer. This three-tier domain model is the basis for a three-step knowledge management implementation model, highlighting the various foci of effort and types of benefits to be gained at each of the steps. Also included is a “knowledge management process model” that illustrates relationships between information management, new knowledge generation / capture / transfer, and decision making, highlighting the different processes for handling explicit and tacit knowledge, and a unique characteristic of an effective knowledge management environment. (Word count: 142)
|
|
Evaluating the Strategic Collaboration Model: Towards the Assessment of Mentoring–centered Succession Management Programs
|
| Presenter(s):
|
| Ray Haynes,
University of Louisville,
rk.haynes@louisville.edu
|
| Rajashi Ghosh,
University of Louisville,
rajashi.ghosh@louisville.edu
|
| Abstract:
This presentation focuses on the evaluation of the Strategic Collaboration Model. The Strategic Collaboration Model is one of the few succession management models that incorporates mentoring as a primary element in fostering leadership development and succession in organizational settings. The presentation provides a contextual discussion of organizational mentoring, and its role as a primary developmental mechanism within the succession management process. Additionally, the presentation addresses a systems-oriented program evaluation methodology for determining the efficacy of the Strategic Collaboration Model in creating effective succession management programs and developing future organizational leaders. Stufflebeam's context, input, process and product evaluation is applied to the Strategic Collaboration Model and specific program evaluative questions are proffered that can empirically test each phase of the model
|
|
Integrating Evaluation into 360-degree Feedback: Enhancing Executive Development Training and Program Impact
|
| Presenter(s):
|
| Zita Unger,
Evaluation Solutions,
zunger@evaluationsolutions.com
|
| Abstract:
This paper examines the role of evaluation and 360-degree feedback in executive development programs. As a long term initiative with multiple phases that must be worked into the culture of an organization. 360-degree feedback also plays a role in helping participants understand and self-reflect on strategic values and behaviors in their organization. The demonstration of leadership capability and performance, based upon the perceptions by others (such as managers, direct reports, colleagues and team members), has implications for self-awareness, professional development and measures of program effectiveness.
A case study example from a local government council will examine these issues and draw on the results of a meta-evaluation.
|
|
Putting Product Evaluation Theory into Practice: Lessons From Two Case Studies on Efficacy of Products Designed for Persons With Disabilities
|
| Presenter(s):
|
| Vathsala Stone,
University at Buffalo,
vstone@acsu.buffalo.edu
|
| Douglas Usiak,
Western New York Independent Living Project Inc,
dusiak@ilm.wnyilp.org
|
| Sajay Arthanat,
University at Buffalo,
arthanat@buffalo.edu
|
| Michelle Lockett,
Western New York Independent Living Project Inc,
mlockett@wnyilp.org
|
| Abstract:
This paper summarizes methods and findings from two case studies conducted by the Rehabilitation Engineering Research Center on Technology Transfer (T2RERC) on the efficacy of products brought to marketplace for persons with disabilities. Using a theoretical framework to benchmark product evaluation where its role is to enlighten decisions of design, development and commercialization, the paper compares the evaluative procedures followed by these two different cases, pointing out implications for both beneficiaries (end users) and industry stakeholders (manufacturers). While drawing lessons from inadequate, incomplete or out-of-sequence evaluations under analysis, the paper also acknowledges barriers to translating product evaluation theory into practice, among them limited industry capacity and expertise in evaluation and limited business experience at academic centers. Making a case for a collaborative framework with stakeholder involvement to improve practice, the paper proposes a collaborative venture to overcome these barriers through academic and business partnerships in future work at the T2RERC.
|
| | | |
|
Session Title: Frameworks of Evaluation Use and Empirical Assessments
|
|
Multipaper Session 758 to be held in Calvert Ballroom Salon C on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Evaluation Use TIG
|
| Chair(s): |
| Edward McLain,
University of Alaska, Anchorage,
ed@uaa.alaska.edu
|
|
Investing Stakeholders in the Process of Generating a Content-specific Evaluation
|
| Presenter(s):
|
| Susan Marshall,
Southern Illinois University, Carbondale,
smarsh@siu.edu
|
| Joel Nadler,
Southern Illinois University, Carbondale,
jnadler@siu.edu
|
| Nicholas Hoffman,
Southern Illinois University, Carbondale,
nghoff@siu.edu
|
| Jack McKillip,
Southern Illinois University, Carbondale,
mckillip@siu.edu
|
| Abstract:
Southern Illinois University's School of Music requested Applied Research Consultants' (ARC) services to revise three teacher evaluation forms: Classroom Instruction, Instruction of Applied Music, and Directors of Ensembles. Due to the three specific styles of teaching in the School, one standard evaluation form was not appropriate. ARC revised the original forms with the help of the Director of the School of Music and created four subscales across the three evaluation forms. The forms were further revised based on qualitative web survey responses from faculty and students. It was important for the instructors to be invested in the process of revising the existing forms to ensure the use of the new forms. Tenured faculty administered the revised evaluation forms to their students at the end of the Fall 2006 semester as a pilot test. The data from these evaluations were analyzed for internal consistency, factor analysis, discriminant validity, and reliability.
|
|
An Evaluation Use Framework and Empirical Assessment
|
| Presenter(s):
|
| Laura Peck,
Arizona State University,
laura_r_peck@hotmail.com
|
| Lindsey Gorzalski,
Arizona State University,
lindsey.gorzalski@asu.edu
|
| Abstract:
This proposed paper addresses the utilization of program evaluation. Substantial literature on evaluation utilization has focused the incorporation of evaluation recommendations into the program design. Prior theoretical research has examined evaluation use and has proposed varied frameworks for understanding the use (or lack thereof) of program evaluation results. Our research focuses on these frameworks and attempts to create a single, integrated framework. But more importantly, we argue, is the extent to which empirical research on evaluation use finds value in this framework. To this end, we rely on prior research regarding categories of utilization, typologies of recommendations, and factors affecting utilization to frame an empirical study of evaluation use. The empirical part of the paper draws on post-evaluation interviews with 19 agencies who have recently engaged in evaluation research. This work be of broad interest to AEA members because of its conceptual, empirical and applied foci.
|
|
The Continuous Quality Improvement (CQI) Benchmarking Initiative: Using Performance Measurement and Benchmarking to Support Organizational Learning
|
| Presenter(s):
|
| Brigitte Manteuffel,
Macro International Inc,
bmanteuffel@macrointernational.com
|
| Sylvia Fisher,
United States Department of Health and Human Services,
sylvia.fisher@samhsa.hhs.gov
|
| Gary Blau,
United States Department of Health and Human Services,
gary.blau@samhsa.hhs.gov
|
| Abstract:
The Continuous Quality Improvement (CQI) Benchmarking Initiative was implemented in 2004 by the Child, Adolescent and Family Brand of the Center for Mental Health Services to utilize evaluation data to support organizational learning and technical assistance planning for federally funded community-based children's mental health service programs. This presentation will provide an overview of a data-driven tool developed as part of this initiative that incorporates performance measures, benchmarks, a scoring index and a communication feedback process to support program development and improvement. The process and analytic techniques for identifying indicators, benchmarks and for developing the scoring index will be discussed. In addition, performance data will be presented to highlight progress that has been made in achieving program benchmarks. Attendees will learn how evaluation data can be used for performance measurement in a manner that is utility-focused and meets the needs of program administrators and evaluators.
|
|
Does Performance Measurement Facilitate Learning?
|
| Presenter(s):
|
| Leanne Kallemeyn,
University of Illinois at Urbana-Champaign,
kallemyn@uiuc.edu
|
| Abstract:
A notable expansion of evaluation is the 'performance measurement' movement, which emphasizes local level measurement of specified performance indicators or program outcomes. A defining characteristic of performance measurement systems is that they assess a predefined set of indicators on a routine-basis. Because of the nature of performance measurement systems, in this paper I argue one of the main limitations is that they limit what stakeholders can learn about social and educational programming. Based on past literature on performance measurement, I illustrate limitations on learning, such as through what outcomes are assessed or measured, how the outcomes are assessed or measured, and how results are interpreted and used. Drawing from the work of Lee Cronbach, I then argue that performance measurement systems ought to help 'society learn about itself.' Finally, I use a case example to illustrate how performance measurement systems can be used to facilitate learning.
|
| | | |
|
Session Title: Informing Government Policy Through Evaluation
|
|
Panel Session 759 to be held in Calvert Ballroom Salon E on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Government Evaluation TIG
|
| Chair(s): |
| Teserach Ketema,
United States Department of Labor,
ketema-teserach@dol.gov
|
| Abstract:
In response to the Government Performance Results Act (GPRA) and the Performance Assessment Rating Tool (PART), Federal agencies are now mandated to use rigorous types of independent evaluations to demonstrate the efficiency/effectiveness of their programs. Accordingly, the Office of Disability Employment Policy (ODEP), within the US Department of Labor, contracted with Westat, a private research firm, to conduct an independent evaluation of selected ODEP demonstration programs designed to identify successful models for serving people with disabilities. The evaluation is determining the extent to which ODEP's demonstration programs contribute to the workforce development system by examining systems change and employment and educational outcomes of participants. One program - Customized Employment (CE) -- will be the focus of our panel presenters. We will describe ODEP's mission, programs, and need for evaluation; the evaluation design, methodology, and key findings for Customized Employment; and some lessons learned on the usefulness of evaluation for government agencies.
|
|
Evaluation of Research Demonstration Programs: Customized Employment
|
| Richard Horne,
United States Department of Labor,
horne-richard@dol.gov
|
|
The Office of Disability Employment Policy was created by the Federal Government to address the low employment rate of people with disabilities. The Office provides national leadership on employment policy to the Department of Labor and other Federal agencies and policy direction for serving those with disabilities. As part of its mission, ODEP awards grants and cooperative agreements to implement research demonstration programs. ODEP demonstration programs consist of a variety of initiatives targeted at both adults and youth with disabilities. Customized employment is an-innovative evidence-based strategy that benefits both the employer and the job seeker in recruitment, hiring and retention.
Richard Horne, the senior Federal official has oversight responsibility for the evaluation of ODEP demonstration programs. His presentation will provide background information on the mission of the Office of Disability Employment Policy (ODEP) and research programs, with special emphasis on Customized Employment and the purposes of the evaluation.
|
|
|
Measuring Changes to the Workforce Development System: A Qualitative Approach
|
| Martha Palan,
Westat,
marthapalan@westat.com
|
|
As part of the ODEP independent evaluation, Westat addressed the following questions:
(1) Did the ODEP customized employment demonstration programs have an impact on local workforce development systems?
(2) To what extent were customized employment approaches effective in building the capacity of the workforce development system;
(3) to what extent were customized employment approaches integrated into the workforce development system complement of services? and
(4) To what extent did customized employment approaches facilitate positive employment outcomes?
Westat's evaluation design and methodology incorporated both qualitative and quantitative approaches to answer these questions.
Martha Palan, Research Analyst at Westat, has major responsibility for the qualitative aspects of this evaluation (e.g., design of interview guides, conduct and description of site visits findings). She will describe the methods used to answer the first three questions. She will also describe some of the key findings based on qualitative data and the strengths and weaknesses of the qualitative approach.
| |
|
Measuring Employment Outcomes: A Qualitative Approach
|
| Lynn Elinson,
Westat,
lynnelinson@westat.com
|
| William Frey,
Westat,
williamfrey@westat.com
|
|
In addition to systems change as a key outcome variable of interest, ODEP also was interested in focusing on the extent to which customized employment approaches facilitate positive employment outcomes (e.g., employment in a competitive job, salary above the minimum wage, 6- and 12-month job retention). Consequently, Westat designed and conducted the Outcomes Analysis Project as part of the ODEP independent evaluation. Consisting of the collection of three rounds of data on sampled program participants, Westat assembled a large database that is able to answer a variety of evaluation questions.
Lynn Elinson, the project director of the ODEP independent evaluation, will describe the Outcomes Analysis Project, paying particular attention to methodology; key quantitative findings; methods for reducing program staff burden; the strengths and weaknesses of the database; and ways in which Westat has been able to combine both qualitative and quantitative data into meaningful conclusions and recommendations to ODEP.
| |
|
Lessons Learned
|
| Richard Horne,
United States Department of Labor,
horne-richard@dol.gov
|
|
The current evaluation being conducted by Westat, ODEP's independent contractor, continues to provide ODEP with a wealth of information. This information is critical in determining the effectiveness and impact of ODEP demonstration programs and documenting effective strategies for achieving systems change. ODEP uses the conclusions and recommendations to inform future policy and employment practice initiatives on a regular basis.
Dr. Richard Horne, head of ODEP's research and evaluation unit, will describe what ODEP has learned from the evaluation thus far, identifying the key findings and their implication for policy. Dr. Horne will also describe how ODEP uses evaluation results to identify government program outcomes for the PART assessment and offers some 'do's' and 'don'ts' with such efforts. Lessons learned will also be shared for those government agencies considering building independent evaluations as part of their programs.
References:
Bond, G.R. (2004). Supported employment: Evidence for an evidence-based practice. Psychiatric Rehabilitation Journal, 22(4), 345-359.
National Center on Workforce and Disability/Adult (NCWD/A) (2004). Fact Sheet. Washington, DC: Office of Disability Employment Policy.
| |
|
Session Title: Documenting Math Science Partnership Projects in New York State
|
|
Multipaper Session 760 to be held in Fairmont Suite on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Dianna Newman,
University at Albany,
dnewman@uamail.albany.edu
|
| Abstract:
The US Education Math Science Partnership (MSP) program is seeking to bridge the gap between current and expected content, pedagogy, and student outcomes in math and science education. As federal and state priorities shift and funding has increased, the evaluation component of this initiative has become increasingly important. There is need not only for local project evaluation but also of documented success of evaluation methods and findings related to cross-project and statewide efforts. This multi-paper session will present evaluation methodologies proven to be successful in documenting local, cross project and statewide MSP programs and their initial findings. Evaluators of three separate programs will present common cross-site methods and findings related to drill down variables and comparative studies related to the impact of professional development and subsequent implementation. In addition, the state evaluator of these three projects will address how these data provide lessons learned and support the broader MSP initiative.
|
|
The Role of Professional Development: How to Document it and What Works
|
| Mary J Pattison,
University at Albany,
mpattison@uamail.albany.edu
|
| Dianna Newman,
University at Albany,
dnewman@uamail.albany.edu
|
|
The purpose of this paper is to aggregate data and findings pertaining to the impact of professional development on classroom mathematics instruction within and across two large MSP projects. Utilizing a logic model approach to evaluation, if the goal of MSP projects is to change student outcomes, they must first change teacher knowledge of math and math pedagogy, their affect toward mathematics and math instruction, and their subsequent mathematics instructional practices. Utilizing a series of drill-down, these evaluations have initiated a series of quasi-experimental studies that address the role of type and frequency of professional development on subsequent teacher and classroom activities. The paper will over-view variables under study and designs used to study the role of variations in professional development. In addition, barriers and facilitators to implementation of large-scale longitudinal professional development studies, and lessons learned in the process that can transfer to similar projects will be addressed.
|
|
Multiple Avenues to Documenting Student Achievement: Results From Two Large Scale Math Science Partnership Grants
|
| Leigh Mountain,
University at Albany,
lmountain@uamail.albany.edu
|
|
The purpose of this presentation will be to present strategies and initial findings related to the impact of two large-scale MSP projects on students' math achievement and affect. Using a drill down strategy, both projects assessed student-related outcomes via a series of data sources that ranged from statewide tests to individual student classroom performance. These data were used to support a series of longitudinal cohort-based analyses that traced students across grades, teachers, and instructional modes. The projects also tracked the types of professional development offered that might have impacted these changes as well as important teacher and building characteristics. The paper will present the methods and initial outcomes associated with the projects and will address barriers and facilitators to implementing the studies. Common findings across the two projects, as well as lessons learned that could be transferred to similar large-scale programs, also will be presented.
|
|
High Quality Local Evaluation of Federal Projects: At Long Last, Necessary!
|
| Kathleen Toms,
Research Works Inc,
ktoms@researchworks.org
|
|
The purpose of this paper is to present information related to Mathematics and Science Partnership (MSP) project for NYC Region 4. The project includes stakeholders from 106 buildings (K-12) and utilizes an interrupted time-series single group quasi-experimental design to measure the impact of activities carried out each project year using professional development as a mediating factor. Research Works, Inc. serves as the local evaluator measuring effects with high a level of rigor meeting local formative needs within the context of the required state and federal MSP evaluation. The paper poses the question: If a rigorous local evaluation identifies effective practice but the federal funder never sees this evidence, does the evaluation still have a funder-related purpose?
|
|
Promoting High Quality Evaluation of Math Science Partnership (MSP) Projects
|
| Amy Germuth,
Compass Consulting Group,
agermuth@mindspring.com
|
|
The MSP competitive grant program encourages partnerships between institutions of higher education and local schools to collaboratively engage in professional development activities that increase the subject matter knowledge and teaching skills of mathematics and science teachers. Since 2005 Compass Consulting Group, LLC has provided technical assistance to partnerships in NY and conducted a statewide evaluation of these partnerships. As part of this panel, Compass will report on lessons learned from providing evaluation technical assistance, reviewing local evaluations, and conducting a statewide evaluation. Much of the focus of this presentation will be on implications for promoting rigorous evaluations of future MSP partnerships, both for program officers during the RFP process and then for potential grantees during the proposal and development stages. Although this study is specific to the MSP program, we suggest that the findings are relevant for funders and developers of all projects that are competitively awarded and require evaluations.
|
|
Session Title: Technology Programs in Higher Education: Considering Digital Horizons
|
|
Multipaper Session 761 to be held in Federal Hill Suite on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Assessment in Higher Education TIG
|
| Chair(s): |
| Shaila Khan,
Tougaloo College,
skhan@tougaloo.edu
|
| Discussant(s): |
| Linda Thurston,
Kansas State University,
lpt@ksu.edu
|
|
Online Programme of Excellence Model
|
| Presenter(s):
|
| Luis Villar,
University of Seville,
mvillar@us.es
|
| Olga Alegre,
University of La Laguna,
oalegre@telefonica.net
|
| Abstract:
This paper illustrates an Online Programme Excellent Model, based on the European quality assurance model, for improving faculty and agency staff members' programme standards. The results of applying this Model indicate the necessity of quality policies that support the evaluators' competencies to improve programmes. This study concludes by outlining how faculty and agency staff members as evaluators can use this Model for the internal and external quality assurance of programmes.
|
|
Online and Flexible: The Changing Nature of Course Evaluation in Australian Universities
|
| Presenter(s):
|
| Christina Ballantyne,
Murdoch University,
c.ballantyne@murdoch.edu.au
|
| Abstract:
Murdoch University in Western Australia recently moved to a totally online system for end of semester evaluation of units (courses). Initial faculty concerns regarding diminishing response rates generally associated with online surveys have been allayed as an overall response rate of around 50 percent has been achieved.
This paper examines the changing nature of unit/course evaluation with the move to online and the fact that student feedback has also now been put on the national agenda in Australia, with a requirement that institutions publish aggregated results. Strategies that have been taken to maximize response rates, the benefits achieved by using online rather than paper systems and issues related to flexible learning environments, which are becoming the norm given that in Australian universities full-time students are in outside paid employment an average of fifteen hours per week, are also examined.
|
|
Aligning Evaluation of Instructional Technology Programs to the Innovation Cycle: The Case of the Duke Digital Initiative
|
| Presenter(s):
|
| Yvonne Belanger,
Duke University,
yvonne.belanger@duke.edu
|
| Abstract:
Evaluation strategies for instructional technology pilot projects should be different from the strategies used for more mature uses of instructional technologies. In developing a program evaluation approach for the Duke Digital Initiative, the Center for Instructional Technology at Duke University wanted a process that would recognize this difference and emphasize use of evaluation findings. Ultimately, a stakeholder-drive evaluation process was developed which aligned with a three-phase innovation cycle to ensure that evaluation focus was both participatory as well as tuned to the needs of decision-makers.
|
| | |
|
Session Title: A Roadmap for Developing a Public Health Research Portfolio Evaluation Program
|
|
Panel Session 762 to be held in Royale Board Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Abstract:
This panel presents an ongoing process for developing a program to evaluate the impact and performance of expenditures in public health research. The stages of development employed included (1) conducting a literature-based process and consulting evaluation practitioners to develop a framework for public health research evaluation; (2) identifying relevant knowledge; (3) summarizing the findings from the literature review; (4) evaluating models for gauging net payoffs from the research portfolio and (5) describing a framework to logically and empirically link outputs and outcomes of an agency's research portfolio. The process described could help provide an objective basis for decision makers to allocate limited public funds to support research when it is yielding net payoffs. Also, demonstrating the impact of a public health research portfolio may create additional public support for maintaining or increasing investments in public health research.
|
|
Overview of Methodology Used to Develop a Research Evaluation Program
|
| Jerald O'Hara,
Centers for Disease Control and Prevention,
johara@cdc.gov
|
| John Araujo,
Centers for Disease Control and Prevention,
jaraujo@cdc.gov
|
| Mona Choi,
Centers for Disease Control and Prevention,
mchoi@cdc.gov
|
| Catherine Pepper,
Centers for Disease Control and Prevention,
cpepper@cdc.gov
|
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Guijing Wang,
Centers for Disease Control and Prevention,
gwang@cdc.gov
|
| Trevor Woollery,
Centers for Disease Control and Prevention,
twoollery@cdc.gov
|
|
Increasing demands for accountability have heightened the need to rigorously assess the value of expenditures on public health research for improving public health. A process for developing a program to evaluate the impact and performance of public health research expenditures is presented. The development process revealed that (1) a rigorous, reliable, and replicable methodology to search and retrieve relevant research articles has great utility; (2) a standardized methodology to review, categorize, and summarize the relevant literature is essential to enhancing knowledge; (3) the extension of existing models and development of new models is essential to promoting the measurement of health impact in an evolving field; and (4) there is a need to identify critical data attributes and build datasets to facilitate rigorous evaluations over time. Future directions and next steps for further developing this evaluation program will be discussed.
|
|
|
MEDLINE Search Strategies vs. Relevant Retrieval: How Closely do They Match for a Research Evaluation Topic?
|
| Catherine Pepper,
Centers for Disease Control and Prevention,
cpepper@cdc.gov
|
| John Araujo,
Centers for Disease Control and Prevention,
jaraujo@cdc.gov
|
| Mona Choi,
Centers for Disease Control and Prevention,
mchoi@cdc.gov
|
| Jerald O'Hara,
Centers for Disease Control and Prevention,
johara@cdc.gov
|
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Guijing Wang,
Centers for Disease Control and Prevention,
gwang@cdc.gov
|
| Trevor Woollery,
Centers for Disease Control and Prevention,
twoollery@cdc.gov
|
|
Review of relevant literature is an integral component of research evaluation. Retrieval of relevant journal literature in MEDLINE depends on availability of appropriate Medical Subject Headings (MeSH) and/or textwords in citations' titles and abstracts. We identified 'relevant' articles through a team-based literature review that informed the development of a framework for evaluating a public health research portfolio. Team members all read 71 articles retrieved from MeSH/textword search strategies and determined article disposition (Keep, Future Reference, or Discard) based on relevance. MeSH headings within disposition categories were compared to determine an optimal search strategy for capturing relevant articles in future literature searches. Full confidence in literature searching for these topics in MEDLINE is limited by some conceptual gaps in MeSH as well as by the unavoidable lower precision of text word searching. An optimized workable search strategy for capturing relevant research evaluation articles will be discussed.
| |
|
Evaluating Public Research Investment: A Literature Review
|
| Guijing Wang,
Centers for Disease Control and Prevention,
gwang@cdc.gov
|
| John Araujo,
Centers for Disease Control and Prevention,
jaraujo@cdc.gov
|
| Mona Choi,
Centers for Disease Control and Prevention,
mchoi@cdc.gov
|
| Jerald O'Hara,
Centers for Disease Control and Prevention,
johara@cdc.gov
|
| Catherine Pepper,
Centers for Disease Control and Prevention,
cpepper@cdc.gov
|
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Trevor Woollery,
Centers for Disease Control and Prevention,
twoollery@cdc.gov
|
|
To identify methods for evaluating public health research investments and their impact, a six-member team reviewed 71 articles retrieved from MEDLINE and the grey literature. Through a structured process of assessment of defined attributes (discipline, method, model, relevance, and impact) 38 articles were identified as a core literature of public research evaluation. Themes that emerged from this analysis included
(1) a broad recognition among governments and non-governmental organizations that expenditures on research should impact health outcomes;
(2) evaluating public research investments is a complex process;
(3) several broadly defined evaluation methods consistently appeared, such as qualitative methods (e.g., peer-review) and quantitative methods (e.g., bibliometrics and cost-benefit analysis); and
(4) researchers may apply evaluation terminologies inconsistently, especially in defining and measuring research impact.
This review demonstrated a critical need to develop a standardized approach and terminology to support research portfolio investment and impact evaluation studies.
| |
|
Extending the Pay-Back Model to Incorporate Costs as Well as Benefits to Measure the Net Impacts of Organizational Expenditures on Public Health Research
|
| Trevor Woollery,
Centers for Disease Control and Prevention,
twoollery@cdc.gov
|
| John Araujo,
Centers for Disease Control and Prevention,
jaraujo@cdc.gov
|
| Mona Choi,
Centers for Disease Control and Prevention,
mchoi@cdc.gov
|
| Jerald O'Hara,
Centers for Disease Control and Prevention,
johara@cdc.gov
|
| Catherine Pepper,
Centers for Disease Control and Prevention,
cpepper@cdc.gov
|
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Guijing Wang,
Centers for Disease Control and Prevention,
gwang@cdc.gov
|
|
The Buxton and Hanney framework for the 'Pay-back Model' includes only benefits from expenditures on research (Buxton & Hanney, 1996).** To fully assess the impact or performance of a single project or a portfolio of research projects, consideration of expenditures is required. Without accounting for both costs and benefits of enabling research projects, erroneous conclusions may be drawn regarding the impact or performance of a single project or a portfolio of research projects. This work extends the Buxton-Haney framework (Pay-Back Model) to include expenditures on research and defines the concept of net impact. The enhanced framework explores the conditions under which expenditures on research derive a net impact in the single project and portfolio of projects cases.
**Buxton MJ, Hanney S (1996) How can payback from health services research be assessed?, Journal of Health Services Research & Policy, vol 1, no 1, pp 35-43.
| |
|
A Bibliometric Methodology to Inform a Logic Model for Evaluating a Public Health Research Portfolio
|
| John Araujo,
Centers for Disease Control and Prevention,
jaraujo@cdc.gov
|
| Mona Choi,
Centers for Disease Control and Prevention,
mchoi@cdc.gov
|
| Catherine Pepper,
Centers for Disease Control and Prevention,
cpepper@cdc.gov
|
| Jerald O'Hara,
Centers for Disease Control and Prevention,
johara@cdc.gov
|
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Guijing Wang,
Centers for Disease Control and Prevention,
gwang@cdc.gov
|
| Trevor Woollery,
Centers for Disease Control and Prevention,
twoollery@cdc.gov
|
|
The impact of public health research may be assessed by linking metrics of inputs, outputs, outcomes, and impact, which can form a logic model for evaluating a research portfolio. Data to study this logic model were obtained by applying a bibliometric methodology to the 2006 Recommendations of the Advisory Committee on Immunization Practices on Prevention and Control of Influenza. Approximately 90% of references in the Recommendations (n=375) were peer reviewed literature, appeared in 87 journals, and were published between 1958 and 2006. Through 1994, the average number of peer-reviewed publications by year cited by the Recommendations was 3.4¦2.6, compared to 20.1¦8.6 after 1994, and peaked at 40 publications for 2000. Approximately 75% of papers cited appeared after 1994, suggesting a 'knowledge cycle time' of 11 years. This approach illustrates a way of evaluating how peer-reviewed research can impact development of a public health guideline. Study limitations are also presented.
| |
|
Session Title: Theory to Practice: Strategic Considerations
|
|
Multipaper Session 763 to be held in Royale Conference Foyer on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG
|
| Chair(s): |
| Robert Hanson,
Health Canada,
robert_hanson@hc-sc.gc.ca
|
|
Using Evaluation Findings to Enhance Learning: From Theory to Practice
|
| Presenter(s):
|
| Paul Stiles,
University of South Florida,
stiles@fmhi.usf.edu
|
| Catherine Batsche,
University of South Florida,
cbatsche@fmhi.usf.edu
|
| Amber Gum,
University of South Florida,
agum@fmhi.usf.edu
|
| Roger Boothroyd,
University of South Florida,
boothroy@fmhi.usf.edu
|
| Abstract:
For the past three years, faculty at the Louis de la Parte Florida Mental Health Institute have conducted a 9-week intensive Research Experience for Undergraduates (REU), funded by the National Science Foundation. The REU is focused on mental health services research and evaluation and includes a series of seminars on research methods as well as an experiential component in which students design and conduct a research project. A program supplement provides an enhanced emphasis on research ethics. As part of the program, an extensive course evaluation has been developed and implemented. Modeled after Kirkpatrick's (1959) longstanding approach to training evaluation, the results have been used extensively each year to modify and improve the REU. The presentation will provide an overview of the REU, provide a detailed description of the evaluation approach, and detail how the evaluation findings has directly resulted in change in the scope and format of the program.
|
|
Challenges for Evaluating Model Programs to Reduce Alcohol Related Problems Among Youth: Communities Mobilizing for Change on Alcohol
|
| Presenter(s):
|
| Linda Bosma,
Bosma Consulting LLC,
linda@bosmaconsulting.com
|
| Abstract:
Communities Mobilizing for Change on Alcohol (CMCA) is a SAMHSA Model Program that uses community organizing as a method of reducing problems related to underage drinking by reducing youth access to alcohol. Community organizing is a new paradigm for many public health agencies more used to school based educational models. Traditional evaluation approaches such as student surveys do not adequately assess this program, which aims to make changes at the community level which will then reduce youth drinking rates. This research examined three groups that were implementing CMCA. This information was used to create an evaluation framework for the evaluation of CMCA. This paper gives an overview of the research, describes the evaluation framework that emerged from this research, and discusses the use of this model in other settings where CMCA is being implemented.
|
|
The Relationship Between Large Scale Student Surveys and Program Evaluation: Risks and Rewards
|
| Presenter(s):
|
| Gregory Rainone,
New York State Office of Alcoholism and Substance Abuse Services,
gregoryrainone@oasas.state.ny.us
|
| Michael Maranda,
Independent Consultant,
mic_maranda@yahoo.com
|
| Abstract:
Legal challenges at the local level and policy changes at the state and federal level (e.g. amendment of the Protection of Pupil Rights Act by No Child Left Behind) have made it increasingly difficult to conduct student surveys of illegal and socially undesirable behaviors such as substance abuse. Consequently, school administrators are increasingly reluctant to participate in these surveys; and, thus, researchers are forced to discover new strategies to encourage participation. One such strategy is to implement school-based surveys that could serve the purposes of both evaluation and epidemiology. This strategy can include collaboration with evaluators through the provision of baseline data and general population norms for outcome measures. This strategy may also include producing reports tailored to the interests of school administrators. The advantages and limitations of variations of this strategy are discussed drawing upon the experience of a survey unit in a single state agency.
|
|
The Non-use of a Longer-term Follow-up Evaluation in Drug and Alcohol Treatment Programs
|
| Presenter(s):
|
| Sharon Ross,
Founder's Trust,
sross@founderstrust.org
|
| Abstract:
While discussions of the use or misuse of evaluation and evaluation results is common in the literature, less common is the discussion of the non-use of evaluation. This paper examines the case of a group of drug and alcohol treatment programs who requested the development of a survey to measure client outcomes at six-months and one-year post-discharge. Despite the strong expression of need by the programs to track and report this data, none of the programs had begun to collect data after two months of having the instrument. Interviews with key program staff reveal how the complexities of implementing evaluation plans and staff beliefs toward evaluation impact whether and how evaluation tools are used. Specific attention is given to the issues that arise from choosing longer-term follow-up surveys to evaluate drug and alcohol programs.
|
| | | |
|
Session Title: Using Appreciative Inquiry in Evaluation: Exploring Impact and Lessons From the Field
|
|
Panel Session 764 to be held in Hanover Suite B on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Chair(s): |
| Tessie Catsambas,
EnCompass LLC,
tcatsambas@encompassworld.com
|
| Discussant(s):
|
| Tessie Catsambas,
EnCompass LLC,
tcatsambas@encompassworld.com
|
| Abstract:
This session presents the experience of applying Appreciative Inquiry to 3 different projects: to a post-conflict country for running focus groups (Kosovo); to evaluate the services provided at a college counseling center (U.S.A.); and to a human anti-trafficking project for developing an evaluation system (Albania). This panel explores examples of AI applications in these different contexts, and shows the variation and options in AI application. The session will also discuss the value of applying Appreciative Inquiry to evaluation and the exciting ways in which it changes the evaluation experience. Benefits include engaging stakeholders in exciting new ways, broadening participation, deepening the culture competence of evaluation, bringing a whole systems view to evaluation, and, ultimately, building evaluation and organizational capacity.
|
|
A Comparison of Appreciative Inquiry and Nominal Group Techniques in the Evaluation of a College Counseling Center
|
| Liz McGough,
University at Buffalo,
emcgough@buffalo.edu
|
|
This study used the second phase of an on-going, multi-year, outcomes based, evaluation initiative at the Buffalo State College Counseling Center as an opportunity to compare Appreciative Inquiry (AI) with Nominal Group (NG) techniques. Five focus groups were conducted two using AI and three using NG. Participants were also asked to complete brief pre, post, and follow-up questionnaires. Content and process comparisons produced results that revealed similarities and differences in the content and process data. The same topics in were raised in each method, however the framing was different. Participants responded well to both methods. Survey responses indicated that NGT participants enjoyed their experience a little more than in the AI participants, however, AI participants indicated slightly higher acceptance than NGT participants of their group's outcomes. Implications for practice, suggestions for modification of the methods, and implications for research are discussed.
|
|
|
Applying Appreciative Inquiry to Focus Groups in a Post-conflict Context
|
| Patty Hill,
EnCompass LLC,
phill@encompassworld.com
|
|
In April 2006, as part of a baseline assessment on women and on ethnicity in the media, EnCompass blended appreciative inquiry (AI) methods within a focus group interview setting, to conduct a unique series of focus group interviews with Kosovar media professionals of different ethnicities. Specifically, the interviews addressed opportunities for:
- The recognition and professional development of female journalists in Kosovo
- Overcoming the obstacles female journalists face in taking on leadership roles in journalism
- Better coverage of ethnic and interethnic issues
- Enhancing a more humane understanding between various ethnic groups
- Integrating various ethnic groups into their staff
This presentation shares the adapted AI focus-group interview protocol, and discusses why AI methods were used in this post-conflict context, what happened during and after the interviews, and implications for the use of AI in addressing such sensitive issues.
| |
|
Using Appreciative Inquiry to Develop an Evaluation System for an Anti-trafficking Project in Albania
|
| Mary Gutmann,
EnCompass LLC,
mgutmann@encompassworld.com
|
|
The Coordinated Action Against Human Trafficking (CAAHT) program of Creative Associates International, Inc., was funded by USAID in 2003 to enhance collaboration and coordination among local government and civil society stakeholders in strengthening anti-trafficking efforts in Albania for the prevention, assistance and reintegration of victims or those at-risk for human trafficking. In 2006, CAAHT decided to strengthen its monitoring and evaluation system in an effort to solidify the gains achieved, and work towards a measure of program sustainability that builds on the relationships established and increases the local capacity of stakeholders to address trafficking in their communities. In strengthening the evaluation system, appreciative inquiry was used as a lead methodology. It resulted in strengthening the use of evaluation data in the key project strategies, and strengthening in-country networks. The project elected to use appreciative inquiry after using it to evaluate one core project strategy related to collaboration and coordination.
| |
|
Session Title: Issues in Measuring Reliability and Retention
|
|
Multipaper Session 766 to be held in International Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Brian Dates,
Southwest Counseling Solutions,
bdates@swsol.org
|
|
Factors Affecting the Behavior of Interrater Reliability Statistics
|
| Presenter(s):
|
| Brian Dates,
Southwest Counseling Solutions,
bdates@swsol.org
|
| Jason King,
Baylor College of Medicine,
jasonk@bcm.tmc.edu
|
| Abstract:
Many evaluators are unaware of recent developments regarding measures of inter-rater agreement, being only familiar with the earlier kappa measure. Interpretation of kappa is limited by its dependence on marginal probabilities and trait prevalence. Although alternatives have been developed, these have not been widely implemented in the statistical computing packages nor systematically studied. We recently (Authors, 2007) developed user-friendly SPSS syntax for this purpose, which includes capabilities for calculating inter-rater reliability estimates for any number of raters and response categories. In the present paper, we present results from a series of Monte Carlo simulations comparing the performance of Scott's pi, Cohen's kappa, Conger's kappa, and Gwet's AC1 statistic across a number of conditions. Application to practice is emphasized.
|
|
A General Method for Estimating the Reliability of High-stakes Educational Decisions
|
| Presenter(s):
|
| Karen Douglas,
International Reading Association,
douglasdouglas@verizon.net
|
| Abstract:
Important educational decisions use complex rules to combine information from a number of tests and assessments. It is widely recognized that the reliability and validity of such scores is of central importance in supporting the fairness of such decisions. This paper presents a simulation method for estimating the reliability of conjunctive, complementary, and compensatory decision rules for a target group of tests and students. Results show that the reliability of a decision depends on the type of decision rule utilized, as well as the number of tests, test difficulty, and the allowable number of attempts to pass. Through application of the suggested simulation method, policy makers can strive to improve the reliability and equity of high-stakes decisions for all students.
|
|
Surveying Nonresponders: Implications for Surveying Methods
|
| Presenter(s):
|
| Jacey Payne,
Howard Research & Management Consulting Inc,
jacey@howardresearch.com
|
| Teresa Roeske,
Howard Research & Management Consulting Inc,
teresa@howardresearch.com
|
| Abstract:
When evaluators solicit information from the public, clients, or stakeholders, they may be pleased with a 25% response rate or ecstatic with a 50% response rate. While the results may inspire confidence, the question remains, what did the other half think? This paper focuses on the perspectives of responders versus nonresponders through the example of a government-sponsored evaluation of a responsible gaming program aimed at providing education and awareness among Video Lottery Terminal (VLT) retailers. A multi-mode survey of retailers yielded a response rate of approximately 50% with responses heavily weighted towards program participants. The sponsor's concern for the notably absent perspectives of nonparticipants became the impetus to retailor the methodology and subsequently survey known nonresponders. Differences between the two phases of the survey can provide insight into the implications of surveying in nonresponse, particularly when survey results are assumed to represent a given population.
|
|
Calculating Retention With Caution: A Look at How Much Measurement Matters
|
| Presenter(s):
|
| Mary Kay Falconer,
Ounce of Prevention Fund of Florida,
mfalconer@ounce.org
|
| Abstract:
Retention of participants in voluntary long-term interventions is a challenge that confronts many practitioners interested in providing support services to high-risk families. Participant retention is often an important moderator of outcome performance in evaluations of these programs. This analysis tests the statistical relationships between multiple predictors and different measures of retention. The participant sample used for this analysis is families who enrolled in a large home visiting program in a southern state between January 1 and July 1 of 2004. The predictors in the models tested include a selection of participant characteristics and programmatic experiences that have appeared consistently in the relevant research literature. The retention measures include retention rates at different time periods and number of days based on date parameters. The analytical techniques applied include linear regression and binary logistic regression. The results provide systematic documentation of the importance of retention measurement in program evaluation and improvement.
|
| | | |
|
Session Title: Evaluating Technology Adaptations in Higher Education
|
|
Multipaper Session 767 to be held in Chesapeake Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Distance Ed. & Other Educational Technologies TIG
|
| Chair(s): |
| Marcie Bober,
San Diego State University,
bober@mail.sdsu.edu
|
|
Formative Computer-based Assessment in Higher Education: Evaluation of an E-learning Instrument
|
| Presenter(s):
|
| Tess Miller,
Queen's University,
millert@educ.queensu.ca
|
| Abstract:
Formative computer-based assessment (CBA) instruments are being developed primarily for use in higher education. The purpose of these instruments is to promote independent learning while focusing on feedback to improve student learning. This paper describes the evaluation of a formative CBA instrument designed for use in a Bachelor of Education course at Queen's University, Canada. In particular, the feedback that was integrated into a CBA instrument and the extent in which the feedback mechanisms stimulated further learning were explored. Although our formative CBA instrument was deemed helpful by students, some feedback mechanisms were found to be more useful than others.
|
|
Course Transformation: A Descriptive Evaluation of the Experiences of 1000 Freshman Enrolled in a Blended Introductory Psychology Course
|
| Presenter(s):
|
| Marcie Bober,
San Diego State University,
bober@mail.sdsu.edu
|
| Abstract:
In Fall 2006, the Dept. of Psychology at San Diego State University, with support from Academic Affairs, conducted an exploratory study meant to determine students' receptivity to a blended learning environment and its impact on academic performance. Unique to this effort was its focus on two sections of a large Introductory course, whose enrollees (n=981) were mostly freshman new to the university environment. Large classes are the norm for freshmen, and there is considerable interest in ways to eliminate their impersonality by building a sense of community while (at the same time) exposing students to innovative uses of technology. This session describes how the project was formatively evaluated, the challenges associated with the evaluative effort (and how they were resolved), and next-steps decisions that key stakeholders made based on the data.
|
|
Surfing the Tidal Wave of Information on the Web: A Collaborative Evaluation of the Use of the Internet in Graduate Level Classrooms
|
| Presenter(s):
|
| Michelle Bakerson,
Western Michigan University,
michelle.a.bakerson@wmich.edu
|
| Abstract:
Although previous evaluations have investigated computer technology (e.g., student use of computers and software, use of the Internet as a tool for distance learning), Internet use as a tool for learning within the classroom and the effects this added resource has on student learning have not been examined. The purpose of this evaluation was to formatively evaluate the effectiveness of using the Internet as a learning tool by measuring graduate students' perception of the importance of using the Internet within the graduate level classroom setting and its effects on learning. A collaborative approach along with a mixed methods approach was selected, because this design gave the opportunity to address the evaluation questions using a combination of quantitative and qualitative methods. As a consequence, the evaluation was able to assess the effectiveness of using the Internet as a learning tool in a comprehensive way and provide appropriate direction for its improvement.
|
| | |
|
Session Title: Learning Within the Arts and Through the Process of Arts Evaluation
|
|
Panel Session 768 to be held in Versailles Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Evaluating the Arts and Culture TIG
|
| Chair(s): |
| Kathlyn Steedly,
The Academy for Educational Development,
ksteedly@smtp.aed.org
|
| Abstract:
In Champions of Change, researchers described a model of cognitive skills, social competencies, and personal dispositions and self-perceptions linked to learning in and through the arts (Burton, Horowitz and Abeles, 1999). Further studies explored the application of the model to evaluations of arts partnerships and programs (Horowitz, 2002). More recently, seven years of qualitative data was coded according to the model. Rating scale items were developed and piloted (Horowitz, 2005). This panel will explore further development of the model and related instrumentation. Presentations will include applications of the model and instrumentation to evaluating arts programs and partnerships.
|
|
A Model and Instrumentation for Evaluating Arts Programs
|
| Rob Horowitz,
Teachers College, Columbia University,
artsresearch@aol.com
|
|
Evaluators of arts programs have been limited by the lack of valid and reliable instrumentation to measure dimensions of cognitive, social and personal development related to arts learning. A study (Horowitz, 2005), published by the Dana Foundation and based upon a model published in Champions of Change (1999), described development and piloting of rating scale items derived from seven years of qualitative research. In this presentation, the development of the database of potential scale items will be described. Additional rating scale items, observational protocols and interview schedules, based upon the database of qualitative data, will be presented.
|
|
|
Applying the Horowitz Model to Evaluating Arts Integration in Public Elementary Schools
|
| Melanie Hwalek,
Social Program Evaluators and Consultants Inc,
mhwalek@specassociates.org
|
|
This presentation reports the results obtained when Horowitz's constructs are applied to the evaluation of Marygrove College's Institute for Arts Infused Education's program that infuses arts into the core literacy curriculum of fourth grade public school students in Detroit. Thirteen of the constructs were used to address three primary evaluation questions:
(1) Do teachers come to 'buy-into' the infusion of arts into classroom teaching (teacher buy-in, comfort level and collaboration with artists)?
(2) What impact has the program had on overall school climate (better teacher-student relationships, seeing students in new light, leadership support for arts-infused education)?
(3) What impact has the program had on student engagement and academic growth (elaboration, expression of ideas, cooperative learning, self-confidence, motivation, ownership of learning, writing)?
Although many data collection tools were developed, this paper will focus on how the Horowitz constructs were used to analyze focus group data from children.
| |
|
Application of Cognitive, Social and Personal Dimensions of Learning to Mixed-Method and Quasi-Experimental Designs in Arts Evaluations.
|
| Dan Serig,
Massachusetts College of Art,
danserig@gmail.com
|
| Rob Horowitz,
Teachers College, Columbia University,
artsresearch@aol.com
|
|
The model of teaching and learning dimensions identified in the Learning In and Through the Arts study (Burton, Horowitz, and Abeles, 1999), and more recently defined in Connections: The Arts and Cognitive, Social, and Personal Development (Horowitz, 2005), was applied to several evaluations of arts programs and partnerships. Evaluation studies (of U.S. Department of Education model development and dissemination and professional development grants, and for state and local arts agencies and organizations) used instrumentation based on cognitive, social and personal dimensions of learning in mixed-method and quasi-experimental designs. The benefits and limitations of the model will be presented within the context of findings from several evaluations. Implications for defining and evaluating arts integrated programs will be discussed, as well as distinctions between evaluating arts education and arts-infused programs.
| |
