|
Session Title: Fewer Errors and Faster Results: How to Automate Production of Tables and Reports With Software You Already Own
|
|
Demonstration Session 103 to be held in Pacific C on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Integrating Technology Into Evaluation
|
| Presenter(s): |
| Reese Jenniges, University of Wyoming, jenniges@uwyo.edu
|
| Humphrey Costello, University of Wyoming, hcostell@uwyo.edu
|
| Eric Canen, University of Wyoming, ecanen@uwyo.edu
|
| Abstract:
As evaluators, we frequently face the challenge of accurately and efficiently producing large numbers of data tables. Similarly, we often produce multiple near-identical versions of reports (for example, reports on county-level data for each county in a state). The Microsoft Office Suite contains underutilized tools that allow automated production of tables, graphs, and even whole reports. In the process, these tools can reduce errors and decrease the time and effort involved. This demonstration will cover three steps in the automation process: 1) exporting results from statistical packages so that they are friendly to automation within Microsoft Office, 2) using Excel as an automation tool to present the correct data in each table or graph automatically, and 3) linking the automated elements in Excel with a Word document to produce an automated report. Presenters will provide session participants with sample templates and automation scripts.
|
|
Session Title: Adaptive Evaluation in Philanthropy: Creating Conditions for Learning Amidst Complexity
|
|
Think Tank Session 104 to be held in Pacific D on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Presenter(s):
|
| Hallie Preskill, FSG Social Impact Consultants, hallie.preskill@fsg.org
|
| Discussant(s):
|
| Tanya Beer, Center for Evaluation Innovation, tbeer@evaluationinnovation.org
|
| Abstract:
Many thought leaders have been pushing the philanthropic field to engage in more catalytic and systems-focused grantmaking. While this transformation in funding is taking hold, as evidenced by a growth in funding of advocacy work, cross-sector collaboration, and systems change, the sector's approach to evaluation is not keeping pace. Many are still struggling to find an evaluation approach that is well-suited for social innovation in complex environments. Building on the findings of a study of the evaluation practices of ten major foundations conducted by FSG and the Center for Evaluation Innovation, this think tank will tackle how the practice of evaluation can be transformed in order to support strategic learning in an age of complexity and uncertainty. After hearing study results, participants will have a facilitated discussion exploring when, why and how to engage in developmental or adaptive evaluations and the internal conditions, structures, processes needed to support adaptive evaluations.
|
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Addressing Unvalued Factors in Evaluation Reports and Recommendations |
|
Roundtable Presentation 105 to be held in Conference Room 1 on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Data Visualization and Reporting TIG
|
| Presenter(s):
|
| Michael Burke, RTI International, burkemic99@hotmail.com
|
| Abstract:
Evaluators are often torn between providing only the service we are contracted or assigned to do and providing additional insights that were not asked for nor desired by our primary clients. Substantial resources will not be used on unvalued aspects of an evaluation, but there are several ways, formally and informally, that evaluators can share 'big picture insights' and detailed, procedural recommendations that respect the values of both the evaluator and his or her client. For example, evaluators often provide careful caveats related to the methods employed and often suggest additional research related to priority populations. In this session we will discuss several ways evaluators can both identify their own and their client's unvalued issues by looking at the scope of an evaluation and evaluation resources, questions, and methodologies. Then we will discuss how these issues, once identified, can be included in evaluation reports and recommendations.
|
| Roundtable Rotation II:
Weaving Words, Making Meanings and Talking Values |
|
Roundtable Presentation 105 to be held in Conference Room 1 on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Data Visualization and Reporting TIG
|
| Presenter(s):
|
| Jeehae Ahn, University of Illinois at Urbana-Champaign, jahn1@illinois.edu
|
| Abstract:
Alternative forms of expression and representation, when used for the right purpose in the right context, can enhance meaningful engagement with values. This presentation tells a story of a small evaluation study, where the evaluator used selected alternative representations to explore key values dimensions of her work, specifically to foreground the plurality of stakeholder values that exist in the given context, and to facilitate dialogue about and across the varied voices and views thus presented. Told from multiple viewpoints and in multiple forms, the story features a collage of poems, vignettes, stories and diagrams that interweave diverse stakeholder experiences and perspectives, and are interspersed with the evaluator's own ideals and aspirations. Followed with questions and discussions with the audience, the session is intended as a dialogic and playful space for exploring and fostering creative use of alternative media to engage more fully with values in our work.
|
|
Session Title: Using Matrix Ranking to Mitigate Facilitator Bias in Focus Groups
|
|
Skill-Building Workshop 107 to be held in Conference Room 13 on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Presenter(s):
|
| Linda Stern, National Democratic Institute for International Affairs, lstern@ndi.org
|
| Alison Miranda, National Democratic Institute for International Affairs, amiranda@ndi.org
|
| Abstract:
Focus groups are a staple of evaluation methodology, providing rich qualitative data for understanding attitudes, opinions and perceptions. However, research has documented the potential for facilitator bias when using focus groups and other participatory research methods. This skills-building session demonstrates the use of a participatory learning and action (PLA) tool - Matrix Ranking - and its potential for mitigating facilitator bias. The Matrix Ranking tool helps a focus group to largely self-facilitate their discussion, come to consensus and quantify their priorities. This hands-on skills session will highlight the use of the Matrix Ranking tool in assessing organizational capacity in the US, local NGO priorities in Uganda, and citizen priorities in Cambodia. The session will then give participants the opportunity to use the Matrix Ranking tool during a simulation exercise and then reflect on its rigor and utility.
|
|
Session Title: Collaborative Evaluation Meets YMCA Education Programs
|
|
Panel Session 108 to be held in Conference Room 14 on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Chair(s): |
| Rita O'Sullivan, University of North Carolina, rjosull@mindspring.com
|
| Discussant(s):
|
| Rita O'Sullivan, University of North Carolina, rjosull@mindspring.com
|
| Abstract:
This panel will present various aspects of collaborative evaluation from external evaluator, program staff, and meta-evaluation perspectives. First presenters will share the collaborative evaluation design of two evaluations that were conducted for YMCA USA. One of the programs worked with 40 sites across the country to help them evaluate their "Higher Education Services Projects." The other program worked with three sites in Springfield, MA; Pittsburgh, PA; and Minneapolis, MN that were developing community-based programs aimed at promoting educational achievment as the basis for increasing neighborhood stability and well-being. Next YMCA program staff will share reflections about how these two collaborative evaluation designs were introduced into their organization. Finally, the specific evalaution context will be compared with broader collaborative evaluation efforts.
|
|
Collaborative Evaluation Design of the YMCA USA Educational Achievement Initiative
|
| Alison Mendoza, University of North Carolina, a.mendoza.215@gmail.com
|
| Johnavae Campbell, University of North Carolina, johnavae@email.unc.edu
|
|
The Educational Achievement Initiative began in Fall 2010 to help promote educational achievement as the basis for increasing neighborhood stability and well-being. Three sites were selected for this work in YMCA affiliated programs in Springfield, MA; Pittburgh, PA; and Minneapolis, MN. Sites were expected to build 10-12 member volunteer Neighborhood Action Teams to assess community assets and plan programs. These programs will then be implemented and their results evaluated. The YMCA decided to work with Evaluation, Assessment, and Policy Connections (EvAP) at the University of North Carolina in concert with the AEA Graduate Education Diversity Internship Program as the external evalautor for this project. Together with the YMCA USA program staff EvAP designed a collaborative evaluation plan to support the program. This presentation will share the collaborative evaluation efforts of the first year of the project.
|
|
|
Collaborative Evaluation Design of the Higher Education Service Project
|
| Johnavae Campbell, University of North Carolina, johnavae@email.unc.edu
|
| Dawn Henderson, North Carolina State University, dawn_henderson@ncsu.edu
|
|
The Higher Education Service Project began in Fall 2010 to help YMCAs expand their higher education programming. Sites could work in the areas of college preparation with grades K-8; college access and transition programs (grades 9-12); non-traditional students, or some combination of participants. Forty sites received funding and were required to develop evaluation plans for their program. The YMCA decided to work with Evaluation, Assessment, and Policy Connections (EvAP) at the University of North Carolina in concert with the AEA Graduate Education Diversity Internship Program as the external evalautor for this project. Together with the YMCA USA program staff EvAP designed a collaborative evaluation plan to support the program. This presentation will share the collaborative evaluation efforts of the first year of the project.
| |
|
Collaborative Evaluation From the YMCA USA Perspective
|
| Marcia Weston, YMCA USA, marcia.weston@ymca.net
|
| Jarrett Royster, YMCA USA, arrett.Royster@ymca.net
|
| Beth Salazar, YMCA USA, beth.salazar@ymca.net
|
|
Faced with the need of implementing two new programs, YMCA USA decided to work with Evaluation, Assessment, and Policy Connections (EvAP) at the University of North Carolina in concert with the AEA Graduate Education Diversity Internship Program as the external evalautor for this project. As the program staff members were very interested in enhancing the evaluation capacity of their grantees as well as gathering process and outcome evidence for the funders of these programs, the program staff requested a collaborative approach to the evaluations. YMCA program staff were integrally engaged in the design and implementation of the evaluation, which included development of logic models, identification of common evaluation instruments, evalaution webinars, evaluation fairs, and cross site accomplishment summaries. Beyond that the two program evaluations introduced collaborative evaluation to the broader YMCA USA organization. This presentation will share the evaluation implementation from the program staff perspective.
| |
|
Session Title: The Business of Evaluation: Achieving Growth in Small and Medium Sized Companies
|
|
Panel Session 109 to be held in Avila A on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Independent Consulting TIG
|
| Chair(s): |
| Richard Hezel, Hezel Associates LLC, richard@hezel..com
|
| Abstract:
The leaders of small- to medium-sized evaluation businesses, especially for-profit, growth oriented firms, face significant challenges maintaining a balance between growth and quality. The panelists, all founders and heads of such companies, will share candidly with attendees the strategies they have employed to grow their businesses and adhere to quality standards in evaluation projects.
|
|
Transitions in Evaluation Companies
|
| Richard Hezel, Hezel Associates LLC, richard@hezel.com
|
|
Richard Hezel will address critical decision making in the evaluation business toward lifestyle business or growth business. He will also lead the discussion regarding an essential decision about growth: whether to staff up or outsource project activities and how to grow through subcontracts with larger organizations. He will also talk about creating the exit plan for the business owner.
|
|
|
Balancing Evaluation Quality and Business Growth
|
| Irene Goodman, Goodman Research Group Inc, goodman@grginc.com
|
|
Irene Goodman will lead discussion about balancing the ever-present imperative to find new clients, projects, and revenue for growth vs. the challenge of maintaining quality around the evaluation projects already in hand. She will also present ideas for finding the right staff and growing from an independent one- or two-person consultancy to the next level.
| |
|
Tactics in Building Strong Evaluation Businesses
|
| Corey Newhouse, Public Profit, corey@publicprofit.net
|
|
Corey Newhouse will discuss specific start-up issues around evaluation business formation, including personal financial liability, partnerships, and tax implications; contracting and billing, including client understanding; and practical tactics for time tracking and billing and other operational strategies.
| |
|
Session Title: South Carolina Access Health, Access Health in the Low Country, and Trident United Way: A Multi-level Health Access Systems Evaluation
|
|
Panel Session 110 to be held in Avila B on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Systems in Evaluation TIG
, the Health Evaluation TIG, and the Human Services Evaluation TIG
|
| Chair(s): |
| Aimee Sickels, Trident United Way,
|
| Abstract:
Panel features the principal evaluator for South Carolina Access Health Network, Executive Director and Data Specialist of a local Access Health project, and Director of Measurement and Policy for Trident United Way discussing the processes needed to establish connections that create a thorough, dynamic, and meaningful multi-system evaluation. It will address the challenges and strengths associated with using systems theory to design the evaluations in conjunction with the practical challenges facing the integration of evaluation with these varying levels of systems. The audience will learn how this group used systems theory while designing program level effectiveness evaluation so that the statewide evaluation becomes a more effective mechanism for informing and improving the system as a whole. The interconnections at the local and state level create fidelity and process challenges experienced across all types of systems level service delivery models, this will offer practical information for evaluators facing these challenges.
|
|
Designing a Tri-County Access Health Evaluation From a Systems Perspective
|
| Rosalia Valesquez, Tri-County Access Health, rosalia@ahtricounty.org
|
| Renee Linyard, Tri-County Access Health, renee@ahtricounty.org
|
|
The presentation is a brief review of the systems theory underlying the local project design and evaluation and then lengthier discussion of how that theory informed various aspects of the project. It describes how the Tri-County Access Health project is using systems theory at each design phase and how systems thinking has informed the selection of particular program level software systems and data collection processes. It will also address the "higher" level systems challenges associated with networked service delivery systems; challenges ranging from relationship development to formalizing processes and network flow. Practical learning will be shared on how to establish network structures that create seamless system functioning and in turn allow for continuous and dynamic evaluation processes to emerge and inform both program and system functioning.
|
|
|
Designing the State-wide Access Health System Evaluation for South Carolina Access Health
|
| Amy Brock-Martin, South Carolina Rural Health Research Center, brocka@mailbox.sc.edu
|
|
Using systems theory to design both formative and summative evaluation strategies for the South Carolina Access Health Network will be presented. The structures for how each of the local Access Health projects have been, and continue to be, informed by a highly functioning systems evaluation will be shared so that others in the field may design these around their own work. The use of data to inform the local project design processes will be shown so that the audience may see how using systems theory at each phase of design, implementation, and evaluation can create holistic strategies that establish foundational structures for sub-systems within the larger system structure. The interplay between the local project implementation and evaluation will be presented from the statewide system's perspective so that the audience can see how the sub-systems work together with the larger system for maximum impact and effectiveness.
| |
|
The South Carolina Access Health Project From a United Way Perspective: How do we fit in the System?
|
| Aimee White, Trident United Way, awhite@tuw.org
|
|
Trident United Way partnered with several local area hospitals applying for a Duke grant in order to establish the Tri-County Access Health project. By not only contributing to the grant writing process the United Way also contributed the infrastructure and support for the project staff the UW became a significant contributor to the Access Health system. The presentation will discuss how this type of systems level support for projects such as these are unique for United Ways and can contribute significant impetus for systems project with very little out of pocket investment. The discussion will be around how knowledge of systems theory and functioning can assist funders, such as United Way, in identifying leverage points and targeting investments to maximize system design and function. The Trident United Way is also contributing evaluation consulting and is assisting the local team in designing their system evaluation in conjunction with the state-wide model.
| |
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Evaluation to the Rescue: Combining Internal and External Evaluation Policies and Practices to Sustain Peripheral Educational Programs |
|
Roundtable Presentation 111 to be held in Balboa A on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Presenter(s):
|
| Ian Martin, University of San Diego, imartin@sandiego.edu
|
| Sharon Rallis, University of Massachusetts, Amherst, sharonr@educ.umass.edu
|
| Abstract:
In times of economic stress, school systems cut their budgets while aiming to maintain core academic services. Looking to eliminate services not deemed basic, districts put peripheral programs on budgetary chopping blocks. Thus, school social workers, counselors, nurses and librarians often find their programs and positions in jeopardy. Drawing on our work in school districts, we argue that those programs engaging in internal and external evaluation practices fare much better than programs or services without evaluation. They survive because they are proactive; that is, they use evaluation information to advocate their value to the district and to make on-going improvements, thus, increasing perceived effectiveness. This roundtable focuses on the role of evaluation within school counseling programs, presenting real cases that use combined internal and external evaluation. We highlight conceptual tools and evaluation capacity building activities that better position peripheral programs within state departments of education and local school districts.
|
| Roundtable Rotation II:
I'll Show You Mine If You Show Me Yours: How Are We Using Efforts To Outcomes (ETO) Systems to Evaluate Programs? |
|
Roundtable Presentation 111 to be held in Balboa A on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Presenter(s):
|
| Elizabeth Kelly, The Urban Assembly Schools, ekelly@urbanassembly.org
|
| Abstract:
The Urban Assembly is a family of small NYC public schools supported by a public/private partnership. Our central office staff provide school support in leadership development, instruction, student enrichment, school operations, youth development, and college readiness. We have a complex value proposition: Our work is with the adults in our schools yet we hold ourselves accountable for our students' success. In order to best measure our impact we had to be sure of the integrity of our implementation - of how we provide the support to schools as needed? We monitor that through a performance management system called ETO (Efforts to Outcomes.) None of the tools in the market is specifically designed for our needs, so we customized this as much as we could. We know we are not alone, and we need to be among our kind. Please join to share lessons from our journeys and improve our evaluative practices.
|
|
Session Title: Methodological Issues in Planning and Conducting Needs Assessments
|
|
Multipaper Session 112 to be held in Balboa C on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Needs Assessment TIG
|
| Chair(s): |
| Sue Hamann,
National Institutes of Health, sue.hamann@nih.gov
|
| Discussant(s): |
| Maurya West Meiers,
World Bank, mwestmeiers@worldbank.org
|
|
Scale Influence on the Identification of Needs: Are Needs From a Fuzzy Approach Similar to Those from a Likert Scale?
|
| Presenter(s):
|
| Yi-Fang Lee, National Chi Nan University, ivanalee@ncnu.edu.tw
|
| James W Altschuld, The Ohio State University, altschuld.1@osu.edu
|
| Abstract:
Traditional needs assessment is based on the measurement of a gap between desired and current states. Discrepancy-based needs are often measured via Likert-scaled surveys which may not reflect uncertainty/ambiguity in judgments as is often the case in social science. A fuzzy approach using a scale range instead of a single score has the potential to deal with the problem. A literature review indicated little empirical study exploring the usage of such a fuzzy approach in identifying needs. The intent of this presentation is to examine whether Likert or fuzzy scaled forms influence how respondents rate importance and current status via two different surveys for a study of essential competencies for program evaluators. The respondents were about 200 evaluators in university evaluation programs organized by the Higher Education Evaluation and Accreditation Council in Taiwan. Recommendations are drawn from the findings.
|
|
Methodological Issues in Needs Assessment for Quality Assurance in a National Context: The Case of Head Start Needs Assessment
|
| Presenter(s):
|
| Hsin-Ling Hung, University of North Dakota, sonya.hung@und.edu
|
| James W Altschuld, The Ohio State University, maltschuld1@columbus.rr.com
|
| Abstract:
Needs Assessment (NA) is commonly conducted by agencies or institutions for assessing needs related to program improvement or the provision of services. When this type of NA is conducted at the individual agency or organizational level, the results are critical input for program/ strategic planning or for looking at the implementation of technical assistance by agencies. The head start needs assessment national analysis project is a case of the second situation. This presentation begins with an overview of the context and then proceeds into methodological issues and concerns encountered by conducting needs assessment in a national context. Lessons learned through the example case will be discussed. Participants' views on NA methodological challenges and possible resolutions will be incorporated into the presentation.
|
|
Statewide Needs Assessment on a Shoestring: Obstacles to Effective Practice
|
| Presenter(s):
|
| Jacque Ewing-Taylor, University of Nevada, jacque@unr.edu
|
| Bill Thornton, University of Nevada Reno, thorbill@unr.edu
|
| Abstract:
In 2010, the Raggio Research Center for STEM Education at the University of Nevada (UNR) was asked to conduct a statewide technology needs assessment by the Nevada Department of Education. This assessment was guided by legislative requirements, as well as by a previous technology needs assessment that had been conducted in 2008. Data regarding state technology standards, district technology plans, technology availability, classroom technology use, and professional development opportunities regarding technology were collected from teachers, technology coordinators, and parents from 17 districts in the state of Nevada using online surveys and some telephonic interviews. Results of this needs assessment will be used by the 2011 Nevada Legislature to determine funding for educational technology, and to inform the teacher education programs and their information technology content needs at UNR as they undergo a significant restructuring in 2011.
|
| | |
|
Session Title: Evaluation Capacity Building Tools for the Practitioner
|
|
Multipaper Session 113 to be held in Capistrano A on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Organizational Learning and Evaluation Capacity Building
|
| Chair(s): |
| Kimberly Leeks,
Centers for Disease Control and Prevention, kleeks@cdc.gov
|
|
Benchmarking Capacity Building through Levels of Use
|
| Presenter(s):
|
| Kimberly Cowley, Edvantia, kim.cowley@edvantia.org
|
| Kimberly Good, Edvantia, kimberly.good@edvantia.org
|
| Sharon Harsh, Edvantia, sharon.harsh@edvantia.org
|
| Abstract:
Benchmarking began more than 20 years ago as a business and industry management tool (Stauffer, 2003). Unlike industry benchmarking, which compares performance and enablers across departments or organizations, the Appalachia Regional Comprehensive Center (ARCC) at Edvantia benchmarking process compares the performance of an individual with a research-based theory that describes the stages or levels of capacity development. The Concerns Based Adoption Model (CBAM) Levels of Use (LoU) (Hall & Hord, 2005, 2010) is used in this benchmarking process to determine the changes in capacity from the beginning to the end of an initiative, and the gap analysis, conducted through a calibration process, determines how close the capacity comes to meeting the organization's vision of the desired performance level. The technical assistance initiative, infused with customized ARCC support, is viewed as the enabler and catalyst for capacity development.
|
|
The Value of using Self-Assessment tools to Increase Capacity for Evaluation
|
| Presenter(s):
|
| Rashell Bowerman, Western Michigan University, rashell.l.bowerman@wmich.edu
|
| June Gothberg, Western Michigan University, june.gothberg@wmich.edu
|
| Paula Kohler, Western Michigan University, paula.kohler@wmich.edu
|
| Jennifer Coyle, Western Michigan University, jennifer.coyle@wmich.edu
|
| Abstract:
Building capacity for evaluation is often difficult when working with people who have numerous organizational roles. In many situations, the ones called to conduct evaluation have little formal training. Upon examination of yearly plans (n = 220) created at strategic planning institutes using a logic model process, evaluators found little correlation exists between participant created goals, activities, outputs, outcomes, and indicators. What results from this lack of correlation is shallow fidelity of evaluation, hence evaluation results don't answer whether teams met their goals and intended outcomes.
This session presents results from implementing a self-assessment tool to measure both the fidelity of their yearly plans and evaluations of those plans. Outlined is the use of this self-assessment tool as a two step project; evaluating plan integrity through inter-rater reliability and improvement in later plan development. Preliminary data supports the use of self-assessment process to increase capacity for the evaluation process.
|
|
Evaluating Capacity Development for Commissioners of Evaluations in South Africa
|
| Presenter(s):
|
| Rita Sonko-Najjemba, Pact Inc, rsonko@pactworld.org
|
| Ana Coghlan, Pact Inc, acoghlan@pactworld.org
|
| Abstract:
African Civil society organisations (CSOs) are inundated with numerous capacity and performance challenges related to insufficient skills in designing, planning and managing evaluations. The lack of access to training often results in CSOs being unable to improve their program designs as well as their ability to track key results. Pact South Africa is a not-for-profit implementing one of the largest USAID HIV/AIDS program in Africa. Pact provides grantees with short-term training and mentorship on M&E and most recently, a new focus on developing basic evaluation capacity has been initiated through a five days training intervention. The program aims to enhance evaluation design, planning and management skills for commissioners of evaluations among CSOs. Evaluation methods targeting the 15 participating organisations include an online survey, interviews as well as a review of program management documents to discern the extent to which the program is influencing change in practice within targeted CSOs. Lessons learnt include the fact that in such settings, interventions like these are essential in improving organisational performance and learning and may thus contribute to sustainability.
|
| | |
|
Session Title: Strategies for Quality Evaluations With the Poor and Disadvantaged: What Does It Take?
|
|
Multipaper Session 114 to be held in Capistrano B on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Social Work TIG
|
| Chair(s): |
| Mansoor Kazi,
State University of New York, Buffalo, mkazi@buffalo.edu
|
|
From Being Formally Non-Existent to Becoming Officially Registered: What is the Impact on the Poor?
|
| Presenter(s):
|
| Ahmed Dewidar, Validity for Research, Evaluation and Quality Assurance, adewidar@gmail.com
|
| Abstract:
130,000 of the poorest Egyptians, who were not officially registered, were supported by a poverty alleviation project to have access to official identification documents. Accordingly, they had access to basic economic and social services. An impact beneficiary assessment was conducted in 2010 to assess the effect of 'getting registered' on their social and economic statuses. An evaluation framework was established with evaluation questions, indicators, parties involved, and data collection procedures. Five tools were created to collect data from a sample of 1,200 beneficiaries and 70 Non-beneficiaries. The findings describe the impact on the beneficiaries' sense of security, sense of citizenship, awareness of their rights as citizens, socio-economic statuses and their attitudes toward their country. In addition, an analysis was conducted for the reasons for the 'non registration phenomenon' and the needs of the poor for better quality of life. Lessons learned for similar evaluation studies are shared.
|
|
The Role of Evaluation and Stakeholder Values in California's Title IV-E Child Welfare Waiver Demonstration Project Evaluations
|
| Presenter(s):
|
| Charlie Ferguson, San Jose State University, charlie.ferguson@sjsu.edu
|
| Abstract:
Child Welfare Waiver Demonstration Projects are being used by the U S Department of Health and Human Services as a means of 'generating new knowledge about innovative and effective child welfare practices' (Children's Bureau, 2010) through the evaluation of those projects. This mission has sometimes conflicted with the goals of state and local stakeholders to use federal fiscal flexibility to develop and implement services for children. This paper discusses the sociopolitical contexts of two Demonstration Projects in California and the role of values in the evaluation planning and implementation dynamic. The first evaluation was a randomized trial of discrete interventions at multiple county sites while the second uses a time-series design to assess the impact of a capped allocation fiscal strategy in two counties. The paper will provide recommendations based on the lessons learned. [Children's Bureau. (2010). Summary of the Title IV-E Child Welfare Waiver Demonstrations. Retrieved from http://www.acf.hhs.gov/programs/cb/programs_fund/cwwaiver/2010/summary_demo2010.htm]
|
|
Complications when Evaluating Disadvantaged Populations: A Needs Assessment of the Knoxville Family Justice Center
|
| Presenter(s):
|
| Ann Cisney-Booth, University of Tennessee, acisneybooth@utk.edu
|
| Amy Dilworth, Knoxville Family Justice Center, adilworth@fjcknoxville.org
|
| Abstract:
In this paper the evaluator discusses the complexity of evaluating programs that serve disadvantaged populations, specifically victims of domestic violence. According to Riger and colleagues (2002), there have been significant improvements in the social, legislative, and political arenas to help those who have experienced domestic violence. To continue this progression it is important that evaluators poignantly examine how evaluations of domestic violence programs are conducted. This paper explores the planning phase of such an evaluation, barriers with obtaining Institutional Review Board approval, and data collection methods. Emphasis will be placed on keeping the victims who participate in the evaluation safe. The purpose of the presentation is to provide other evaluators with strategies to combat the complexities involved with evaluating such programs. Evidence from the needs assessment of the Knoxville Family Justice Center will be provided to help illustrate the challenges with working with an at risk population.
|
| | |
|
Session Title: Crime and Justice TIG Business Meeting and Presentations
|
|
Business Meeting and Multipaper Session 115 to be held in Carmel on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Crime and Justice TIG
|
| TIG Leader(s): |
|
Roger Przybylski, RKC Group, rogerkp@comcast.net
|
|
Evaluating Sensitive Topics: Access, Institutional Review Board (IRB), Design & Methodological Considerations
|
| Presenter(s):
|
| Billie-Jo Grant, Magnolia Consulting LLC, bgrant@magnoliaconsulting.org
|
| Abstract:
Researchers and evaluators of sensitive topics investigate some of society's most pressing social issues and policy questions, but because of the intimate, discreditable, or incriminating nature of these topics (e.g. death, sexual abuse, violence, drug use, or homelessness), many methodological challenges can limit the design and execution of these studies (Lee & Renzetti, 1993). Drawing on the literature and experiences from a completed, institutional review board (IRB) approved, sensitive research study on sexual abuse, this paper presents recommendations for adapting research and evaluation designs for sensitive topics. Specifically, this paper (1) describes the legal, ethical, and confidentiality challenges associated with sensitive research designs, and (2) provides recommendations for subject recruitment, IRB approval, and study design.
|
|
Sex Offender Recidivism and Treatment Effectiveness: What Do We Know, What Can We Know, and What Are the Implications for Policy and Practice
|
| Presenter(s):
|
| Roger Przybylski, RKC Group, rogerkp@comcast.net
|
| Abstract:
According to the Association for the Treatment of Sexual Abusers (ATSA), most states require convicted sex offenders to participate in treatment while in prison or on probation. One of the basic assumptions underlying this requirement is that sex offender treatment works, yet an important debate exists about how to interpret findings from sex offender treatment outcome studies. From an evaluation standpoint, much of the controversy about treatment effectiveness stems from the difficulties associated with measuring the reoffense rates of sex offenders, and the paucity of randomized controlled trials in sex offender treatment research. This discussion will begin with an overview of 1) key measurement issues in sex offender recidivism and treatment effectiveness research, and 2) findings from recent evaluations and meta-analyses of treatment effectiveness. This will be followed by an interactive audience discussion about sex offender recidivism and treatment effectiveness research methodologies and findings, and the implications they have for policy and practice.
|
| |
|
Session Title: A Method to Our Madness: Program Evaluation Teaching Techniques
|
|
Panel Session 116 to be held in Coronado on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Teaching of Evaluation TIG
|
| Chair(s): |
| Bonnie Stabile, George Mason University, bstabile@gmu.edu
|
| Discussant(s):
|
| Linda Schrader, Florida State University, lschrader@fsu.edu
|
| Abstract:
Those who teach evaluation are always on the lookout for teaching techniques to engagingly and effectively communicate ideas about the core concepts of evaluation. This panel is intended to serve as an idea exchange for those who teach evaluation to learn from their AEA colleagues about techniques that have worked particularly well in the classroom. Topics covered will range from novel means for the introduction of the study of evaluation, to tips for the teaching of particular tools, such as logic models. Teaching techniques that help students to effectively engage with evaluation clients will also be considered
|
|
Teaching Evaluation Using a Photography Analogy
|
| Nick Fuhrman, University of Georgia, fuhrman@uga.edu
|
|
Evaluation and photography have a lot in common. If the purpose of evaluation is to collect data (formative and summative) that informs decisions, more than one "camera" or data collection technique is often best. We have qualitative cameras (long lens to focus on a few people in depth) and quantitative cameras (short lens to focus on lots of people, but with less detail). Some pictures will be up close and some will be wide angle. Both are needed to make a decision. In evaluation, we call different aspects of what we're measuring "dimensions." I think about three major things that we can measure...knowledge change, attitude change, and behavior/behavioral intention change following a program/activity. Each of these has dimensions (or different levels of intensity) associated with them. It takes more than one picture to determine if our educational efforts influenced knowledge, attitude, or behavior and to make decisions about program value.
|
|
|
A Worksheet for Conducting Evaluability Assessment
|
| Helen Holmquist-Johnson, Colorado State University, helen.holmquist-johnson@colostate.edu
|
|
I have designed a worksheet to help students navigate / negotiate a program evaluation to be carried out in their internship agencies. The worksheet is on conducting the evaluability assessment. Though it was designed specifically for use by graduate students in the Masters of Social Work Evaluation Class, it will likely work well in other program settings.
| |
|
Online Discussions in Teaching Program Evaluation
|
| Jill Hendrickson Lohmeier, University of Massachusetts, Lowell, jill_lohmeier@uml.edu
|
|
Particularly for students from diverse geographic and professional areas, using targeted discussion questions allows/requires students to explain evaluation situations and challenges that may be unique in their areas. These discussions last at least a week and include multiple comments from all students. Students have commented on how eye opening this has been for them.
| |
|
Roleplay for Teaching Investigation and Negotiation of Client Evaluation Needs
|
| Mara Schoeny, George Mason University, mschoeny@gmu.edu
|
|
This presentation will highlight a roleplay option for teaching investigation and negotiation of client evaluation needs. The roleplay provides a brief program background and a variety of roles, including funders, project directors and service providers, each of whom have different interests in the evaluation scope and direction. The evaluators receive practice in investigation, facilitating participation and recognizing the different perspectives and interests in evaluation results.
| |
|
Hot or Cold? Introducing Evaluation
|
| Claire Tourmen, AgroSup, claire.tourmen@educagri.fr
|
|
When introducing the concept of evaluation, I begin by asking a simple question: "If I say that it's too cold in this room, what did I do?" Students must find the main operations involved in an evaluation: the final operation is to assert a judgment, such as "It is too cold". To be able to do it, I had to gather some data (by any means: I checked a thermometer, I shivered, I saw people shivering etc.). The point is that I had to interpret these data to make my judgment.
Then I ask people a second question: "For example, if I saw that the temperature was 59 F, what does it mean? Is it cold or not?" The answer they always give is: "It depends!" People understand that to be able to judge any object, you need to compare gathered data to other elements that give it value.
| |
Cookies4
| Susan Allen Nan, George Mason University, snan@gmu.edu
|
|
he classic evaluation exercise of evaluating cookies started us on a cookie themed course. Starting with evaluating cookies as a way to learn about evaluation and indicators, we found at each subsequent class a reason to engage cookies. We tried weeks later to "reconstruct the baseline" and recall how the cookies had been the first week after that data was lost. We considered ethics of cookies provision just before the course evaluations. An element of developmental evaluation led the class to consume carrots in lieu of cookies. We could have introduced adjusting a cookie recipe together after formative evaluation.
| |
|
Using Note Cards in Logic Model Development
|
| Shandiin Wood, University of Arizona, shandiin.wood@gmail.com
|
|
The process involves an interview between the evaluator and the stakeholder. During this time while the stakeholder is recalling any causal conditions to a stated problem the evaluator is writing them down on the note cards and laying them on the table or surface before them. During the process of root cause analysis the evaluator is assisting the stakeholder through an "if-then" logical flow where 'if' a causal condition where to occur 'then' this other causal condition would exist. This process is much like that described in the ATM logic modeling process as described in Renger and Titicomb 2002. A key difference is the use of the note cards in this case and as a benefit the stakeholder is able to move the cards themselves as a small but important means to create buy-in from the stakeholder.
|
|
Evaluation Start to Finish: A Final Project
|
| Liudmila Mikhailova, CRDF Global, mikhailova@msn.com
|
|
This project leads students through the whole semester in order to learn the entire process of evaluation, starting from how to design an evaluation plan, evaluation questions and evaluation instruments to evaluation methodologies, data collection and data analysis and report writing. A very important emphasis is made on cultural contexts that programs are operating in. Students work in teams to design an evaluation study as their final project. Each group of students conducts research on current or past international programs that are administered by U.S., DC-based non-profit organizations, such as World Learning, Eurasia Foundation, NED (National Endowment for Democracy), CIEE (Council for International Educational Exchange) and other development and exchange organizations. Students have to make appointments with respective organizations in order to collect information about their selected projects and interview project/program managers, which enhances their interviewing skills as well.
|
|
Session Title: Reaching the Gold Standard: Evaluating Tobacco Policies in K-12 Schools and Institutions of Higher Education Using a Practical Assessment Tool
|
|
Skill-Building Workshop 117 to be held in El Capitan A on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Health Evaluation TIG
|
| Presenter(s):
|
| Doneisha Snider, Washington University, St Louis, dsnider@brownschool.wustl.edu
|
| Sarah Moreland-Russell, Washington University, St Louis, srussell@gwbmail.wustl.edu
|
| Abstract:
Two practical evaluation tools were developed to evaluate the strength of tobacco-related policies in K-12 schools and institutions of higher education. During this session, both tools will be demonstrated, and attendees will use them to evaluate sample policies. Presenters will discuss the challenges and opportunities of the evaluation process (e.g., collecting policies, using the tools, rating policies). To illustrate the differences in policy language that evaluators may encounter, presenters will compare the language in policies that received high scores with those that did not. By the end of the session, attendees will have the skills and tools to evaluate the strength of tobacco free policies in order to inform future policy development and amendments. After the session, attendees will be able to evaluate tobacco-related policies in schools or institutions in their community using the tools provided.
|
|
Session Title: Contested Terrains, Multipartiality and Interpersonal Validity: Engaging Multiple Voices, Views and Vantage Points for Inclusive Excellence and Social Justice
|
|
Think Tank Session 119 to be held in Huntington A on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Multiethnic Issues in Evaluation TIG
|
| Presenter(s):
|
| Hazel Symonette, University of Wisconsin, Madison, hsymonette@studentlife.wisc.edu
|
| Abstract:
Contested terrains live large in the emergent complexities of increasingly diverse, globally-interconnected communities. To honor our ethical responsibilities as evidence-grounded truth-seekers and truth-speakers, evaluators need to authentically engage our world's complexities by mindfully calibrating and cultivating capacities to be clear channels as *legitimized judgment-makers* and, thus, as privileged authorities. How do you discern your Forcefield of Preparedness and Readiness to activate truth-seeking processes, practices and decisions? In what ways can you--with responsible transparency and interpersonal validity-facilitate and support deliberative judgment-making where the judgment criteria and evidentiary rules are public, explicit, fair and appropriate from multiple stakeholder vantage points?
Participants will start exploring these very complex issues in a multi-tiered deliberative forum. Private brainstorming is followed by dyad deliberations then quad-groupings (dyad-pairs). The session will close with individuals posting their top 2-3 insights for a Gallery Walk and culminate with full-community shout-outs as final harvest of our collective wisdom.
|
|
Session Title: Strengthening Federal Program Evaluation Policies and Practice
|
|
Panel Session 120 to be held in Huntington B on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Government Evaluation TIG
and the Evaluation Policy TIG
|
| Chair(s): |
| Stephanie Shipman, United States Government Accountability Office, shipmans@gao.gov
|
| Abstract:
After two decades of increasing pressures on federal agencies to provide credible information on the efficiency and effectiveness of their programs and policies, we are recently seeing increased interest and investment in strengthening agency evaluation. This panel provides concrete suggestions for strategies that federal (and other public) agencies could adopt or adapt to increase the usefulness of their evaluations for policy and program management and oversight. GAO staff will report their findings on the policies and criteria that four agencies with mature evaluation capacity use to develop effective evaluation agendas. A CDC official will describe CDC's recent efforts to instill a culture of continuous quality improvement, better integrate planning, performance measurement, and evaluation, and build evaluation capacity across the Centers. A State Department official will discuss the development and initial implementation of their new department-wide evaluation policy, one of the few such examples in the federal government.
|
|
Federal Agencies Balance Multiple Criteria to Develop Effective Evaluation Agendas
|
| Stephanie Shipman, United States Government Accountability Office, shipmans@gao.gov
|
| Valerie J Caracelli, United States Government Accountability Office, caracelliv@gao.gov
|
| Jeff Tessin, United States Government Accountability Office, tessinj@gao.gov
|
|
Recent congressional and executive initiatives seek to improve federal government performance and accountability through expanding the availability and use of program evaluation. GAO was asked to study the evaluation planning policies of federal agencies that have mature evaluation capacity in order to help other agencies focus their evaluation resources on supporting effective management and legislative oversight. The project team will discuss the remarkably similar planning processes and selection criteria used by four agencies in the Departments of Education, Health and Human Services, and Housing and Urban Development to develop effective evaluation agendas-a portfolio of studies that strive to provide timely, credible answers to important policy and program management questions. The presentation will also discuss some of the conditions that influenced variation in processes across the agencies and could influence others' adoption of the shared model. The report, GAO-11-176, was issued in January 2011.
|
|
|
A Rising Tide Raises All Boats: Supporting Program Evaluation at the Centers for Disease Control and Prevention
|
| Thomas Chapel, Centers for Disease Control and Prevention, tchapel@cdc.gov
|
|
CDC's many Centers and large grant programs have a long history of program evaluation. But approaches, resources, and uses of data for program improvement vary widely. Furthermore, program evaluation has tended to exist apart from strategic planning and performance measurement efforts in many programs. In 2010, CDC created the Office of the Associate Director for Program to instill a culture of continuous quality improvement, better integrate planning, performance measurement, and evaluation, and to help support and improve program evaluation and evaluation capacity. This presentation will discuss key efforts of the Office, focusing in particular on the creation and implementation of standards and recommendations for program evaluation, the role of a new agency-wide advisory group, a "meaningful measures" project to better link program evaluation and mandated performance measurement, and several key efforts to build evaluation capacity through training, resources, and technical assistance.
| |
|
Department of State Evaluation Policy: From Evolution to Implementation
|
| Stephanie Cabell, United States Department of State, cabellsd@state.gov
|
|
The U.S. Department of State introduced a new evaluation policy in late 2010 and is rolling out implementing procedures and guidance to support successful implementation of the policy. State's evaluation policy provides a framework for the implementation of evaluations for programs, projects, and activities carried out by the Department. The policy supports the Department's goal of connecting evaluation, an essential function of effective performance management, to investments in diplomacy and development to ensure they align with the agency's overarching strategic goals and objectives. The presentation will also touch upon how State and USAID are working together to build evaluation capacity within the two agencies and to bring about more effective collaboration and coordination of evaluations.
| |
|
Session Title: Rethinking Evaluation of Civic Engagement
|
|
Think Tank Session 121 to be held in Huntington C on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Presenter(s):
|
| Jane Reisman, Organizational Research Services, jreisman@organizationalresearch.com
|
| Discussant(s):
|
| Audrey Jordan, Annie E Casey Foundation, ajordan@lcworks.org
|
| Anne Gienapp, Organizational Research Services, agienapp@organizationalresearch.com
|
| Mary Achatz, Westat, maryachatz@westat.com
|
| Mayur Patel, John S and James L Knight Foundation, patel@knightfoundation.org
|
| Kimberly James, W K Kellogg Foundation, kimberly.james@wkkf.org
|
| Abstract:
Annie E. Casey Foundation has a deep and long-standing interest in civic engagement and how to thoughtfully evaluate this work. Casey has recently begun to develop a framework and tools/approaches that would be useful for both its own internal efforts and those of other foundations, while simultaneously, being useful to organizations and initiatives engaged in this work. Organizational Research Services (ORS) is supporting the development of this thinking. The emerging framework and tools build on Casey's place-based evaluation frameworks developed as part of the Foundation's comprehensive community change initiatives. To develop the framework, Casey seeks to engage other foundations as thought partners, similar to what were successful collaborative field-building experiences in the area of advocacy evaluation. Casey is also engaging in thought partnership with other foundations. This session will present the emerging framework and thinking to date and engage participants in a thoughtful discussion about key questions facing this field.
|
|
Session Title: Challenges in Responding to Stakeholder Values: An Example From a Large-scale, Multi-site/Multi-level Effort to Create a National Warehouse of State and Territorial Tobacco Cessation Quitline Data
|
|
Multipaper Session 122 to be held in La Jolla on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG
|
| Chair(s): |
| Martha Engstrom, Centers for Disease Control and Prevention, mengstrom@cdc.gov
|
| Abstract:
This session will provide a real-world example that highlights the challenges evaluators face in their efforts to respond to the values of multiple stakeholders while planning and implementing a multi-site, multi-level evaluation. The example discussed is the National Quitline Data Warehouse (NQDW), a large-scale initiative by the Centers for Disease Control and Prevention (CDC) to create the first national warehouse of data from state and territorial tobacco cessation quitlines. The primary goal of the NQDW is to serve as a continuing national resource for data on utilization, success, and services of quitlines to support evaluation for accountability and program improvement at both federal and state levels. Stakeholders for this evaluation include: federal, state and territorial programs, funders of tobacco cessation quitlines, service providers, and cessation researchers. The panelists will describe challenges and lessons learned in their efforts to respond to stakeholder values throughout the planning and implementation of this evaluation.
|
|
Overview of the Challenges in Engaging Stakeholders and Responding to Their Values
|
| Martha Engstrom, Centers for Disease Control and Prevention, mengstrom@cdc.gov
|
|
A multi-site, multi-level evaluation cannot be successful if it does not include a detailed and thoughtful plan to obtain and respond to stakeholder values. Challenges arise when it is time to consider whether or not (and how) to incorporate the values of multiple stakeholders into a complex evaluation effort. These challenges include: 1) engaging stakeholders when they have concerns about the evaluation and the use of the evaluation results; 2) responding to the values of multiple stakeholders when their values are in conflict; 3) responding to the values of multiple external stakeholders while staying true to the original goals and needs of the evaluation; and 4) responding to the values of multiple stakeholders while still obtaining reliable and valid data that answer the key evaluation questions in a credible way. These challenges will be presented with an eye to lessons learned for future efforts.
|
|
Project Management in Response to Stakeholder Values
|
| Henraya McGruder, Centers for Disease Control and Prevention, hmcgruder@cdc.gov
|
|
The importance of obtaining stakeholder input and responding to stakeholder values is clear in many phases of an evaluation project such as evaluation planning and use. However, what may not always be clear is the importance and utility of engaging stakeholders and responding to their values throughout the project management and implementation process. Challenges, lessons learned, and successful examples of this will be discussed. For example, the original data collection plan for the National Quitline Data Warehouse included a specific data format that all sites were asked to follow when submitting their data files to us; however, we soon learned that requiring this format would be burdensome to many sites in terms of both cost and effort. Therefore, we changed the data collection plan so that sites are allowed to submit their data in the format that is easiest for them.
|
|
Provision of Technical Assistance in Response to Stakeholder Values
|
| Adriane Niare, Centers for Disease Control and Prevention, aze2@cdc.gov
|
|
The creation of CDC's National Quitline Data Warehouse involves providing a high level of technical assistance to diverse stakeholders-all of whom may vary in their information needs and values. Successful technical assistance will facilitate "buy-in" and increase the likelihood that the data collection effort, and therefore, the evaluation, are successful. The panelist will discuss how an effective technical assistance (TA) team is created, and also provide examples of how to provide evidence and data-based technical assistance to stakeholders including: 1) the importance of training technical assistance providers on how to interact with stakeholders to respond appropriately to stakeholder values, 2) the utility of an effective, dynamic database for tracking interactions with stakeholders, 3) the importance of developing detailed guidance documents, sharing them with stakeholders and revising them based on stakeholder input, and 4) the importance of developing and maintaining positive rapport between the technical assistance team and the stakeholders.
|
|
Evaluation Planning, Data Analysis, and Dissemination in Response to Stakeholder Values
|
| Lei Zhang, Centers for Disease Control and Prevention, lzhang2@cdc.gov
|
|
It is standard evaluation practice to engage multiple stakeholders and obtain their input throughout the evaluation process, from the initial planning to the analysis and dissemination of the results. However, there are significant challenges to accomplishing this objective successfully, especially when attempting to do this in a multi-site, multi-level evaluation. For every given level or site, a stakeholder may have differing needs or agendas. This presentation will describe how the National Quitline Data Warehouse project team accomplished this objective by convening an external Evaluation Advisory Workgroup comprised of state and territorial tobacco control programs, funders of tobacco cessation quitlines, tobacco cessation quitline service providers, and tobacco cessation researchers. The panelist will discuss challenges, lessons learned, and successes related to our experiences in convening this group of stakeholders, obtaining their input, and responding to their often conflicting values in the evaluation planning, data analysis, and dissemination phases.
|
|
Session Title: Evaluation Within Cultures: A Research on Evaluation Perspective
|
|
Multipaper Session 123 to be held in Laguna A on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Research on Evaluation
|
| Chair(s): |
| Matthew Galen,
Claremont Graduate University, matthew.galen@cgu.edu
|
|
Why are Some Public Agencies More Active in Policy Evaluation Than Others?
|
| Presenter(s):
|
| Valerie Pattyn, University of Leuven, valerie.pattyn@soc.kuleuven.be
|
| Abstract:
Why do public sector organisations differ in policy evaluation activeness? While a long list of potential explanations can be collected in evaluation circles, systematic research evidence on this issue is only scarcely available. The only certainty thus far is that context highly matters. This observation should not discourage us, however, from searching for patterns behind this contingency.
The paper deals with part of this research agenda. Central focus is to explain the extent of evaluation activeness: differentiating between organisations that have not yet conducted evaluations; that plan to evaluate; and that indeed conducted evaluations. Explanatory variables are ordered along four dimensions: actor characteristics (capabilities; orientations), institution related characteristics, policy issue characteristics and legacies.
28 public sector organisations of the Flemish administration (Belgium) constitute the area of investigation. In line with the contingent nature of evaluation practices, the research relies on the'family' of configurational comparative methods (QCA).
|
|
Differences and Similarities in American and European Evaluation Research
|
| Presenter(s):
|
| Nanna Friche, Aalborg University, nanna@learning.aau.dk
|
| Leanne Kallemeyn, Loyola University, Chicago, lkallemeyn@luc.edu
|
| Jori Hall, University of Georgia, jorihall@uga.edu
|
| Abstract:
Current bibliometric research documents how evaluation theorists in North America tend to focus on methodological issues of (educational) evaluation compared to issues of use and valuing. The authors of this paper expect this tendency to be reflected in the literature cited by these theorists. In contrast European theorists tend to be far more influenced by Continental philosophy than their U.S. counterparts. It is these apparent differences between evaluation in Europe and North America that this paper is engaged with. Thus, the purpose of the study described in this paper is to explore empirically the similarities and differences between articles published in the American Journal of Evaluation and Evaluation, a journal supported by the European Evaluation Association in order to address the question; what do the literature cited in Europe and the United States say about the field of evaluation, as well as evaluation theory and practice on each continent?
|
|
Values in Evaluation: The What and How Values in Swedish School Inspections
|
| Presenter(s):
|
| Christina Segerholm, MidSweden University, christina.segerholm@miun.se
|
| Abstract:
This paper analyses values permeating the Swedish school Inspections. School inspection is becoming increasingly influential in education policy and practice globally. As all evaluative activities school inspection conveys particular values albeit sometimes disguised in a more supervisory or legal rhetoric. This analysis highlights the What (substantial) and How (procedural) values embedded in the Inspectorate as expressed in interviews with high-ranking officers. Results point to tensions and ambiguities in values. Values concerning what to promote stress each child's right to be awarded a pass grade in a safe environment. Values about how inspections are to be undertaken underline assessments based on equal requirements on schools, but also on a needs-oriented procedure. Reports are to sustain: Credibility, clarity, accessibility, comprehensiveness and impartiality. These values are not easily reconciled with the reporting style (deviation reports) where mainly flaws and irregularities are noticed, i.e. a style based on values like shaming and blaming.
|
| | |
|
Session Title: Impact Evaluation Design for the Peru New Private Sector Competitiveness and Poverty Reduction and Alleviation Activity Project (PRA II), 2009-2012
|
|
Panel Session 124 to be held in Laguna B on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Chair(s): |
| Mark Bardini, Chemonics International, mbardini@chemonics.com
|
| Abstract:
The main objective of this session is to present the nature of the USAID Peru Poverty Reduction and Alleviation (PRA) project approach and how a goal-based evaluation design can incorporate the future expectations of its main beneficiaries. Chemonics International (CI) is implementing the second stage of the PRA project from 2009 to 2014. The strategy of this project is market-driven. Therefore, its development approach is to generate dynamic economic growth in the poverty zones of Peru through the creation of business plans for SMEs. Each plan implies associating and linking the production of small urban and rural producers with value chain agents with greater capacity (project's clients). At the end of the fifth year of the intervention (2014), the project implementation should show that the household income of project beneficiaries (the micro-entrepreneur producers and suppliers of the project's clients) has increased, on average, 30%, in comparison to a control group.
|
|
Presentation of the Impact Evaluation Design for the Peru New Private Sector Competitiveness and Poverty Reduction and Alleviation Activity Project (PRA II)
|
| Pedro Mateu, Western Michigan University, pmateub@yahoo.com
|
|
In this part of the presentation, Pedro will explain the rigorous impact evaluation design (quasi-experimental) for PRA II. He will detail the intricacies of the design, including how propensity score matching will be used to simulate treatment and control groups. Pedro will also explain how this rigorous design will measure the project impact of increasing incomes for the project beneficiaries (micro-entrepreneurs and supply chain providers)in the intervention zones in Peru.
|
|
|
Implications and Relevance of the Impact Evaluation Design of Peru PRA II for the Evaluation Field
|
| Mark Bardini, Chemonics International, mbardini@chemonics.com
|
|
In this part of the presentation, Dr. Bardini will explain the implications and relevance of the impact evaluation design of Peru PRA II for the evaluation field. This impact evaluation design is unique in that its methodology includes an alternative (contingency) impact design which will be applied if the intervention of the project changes significantly over time or if the propensity scores lose their validity.
| |
|
Current Status of the Impact Evaluation Design for Peru PRA II Project
|
| Marco Aspilcueta, Chemonics International, maspilcueta@chemonicsalliances.com
|
|
In this part of the presentation, Marco will provide the most current updates to the data collection and impact evaluation design. He will also provide insights on how the imapct evaluation deisgn is changing, make recommendations on how it is being improved and solicit feedback from the audience on the design.
| |
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Value-resonant Evaluation Design Within Community-School Partnerships to Improve College Access |
|
Roundtable Presentation 125 to be held in Lido A on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the College Access Programs TIG
|
| Presenter(s):
|
| Silvia Swigert, University of California, Irvine, sswigert@uci.edu
|
| Amanda Valbuena, University of California, Irvine, avalbuen@uci.edu
|
| Abstract:
Building effective community-school partnerships with post-secondary partners to improve college-going rates in low-income schools is one of the goals of GEAR UP, a federal college access program implemented through local partnerships. The participants in these partnerships include school administrators, teachers, and parents, in addition to business, community, and post-secondary partners. Designing a useful evaluation for a parent college access outreach program can be a difficult enterprise for the evaluation team. Conflicting values guiding the decisions about what to study and how to study it can lead to a stalled evaluation of the program processes. This paper describes the context of a parent-to-parent outreach program co-created in 2001 by a post-secondary partner, along with parents in the community surrounding the target schools, entering a new phase of interdependence between partners and requiring a fresh look at whose values will guide the future evaluation of the partnership.
|
| Roundtable Rotation II:
The College of 'Best Fit': Evaluating the Whole Student to Increase College Graduation Rates |
|
Roundtable Presentation 125 to be held in Lido A on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the College Access Programs TIG
|
| Presenter(s):
|
| Ashley Allen, Horizons for Youth, ashley@horizons-for-youth.org
|
| Abstract:
College access programs assist in removing the barriers for students to attend college; especially for those in underrepresented populations. These programs must now focus to remove barriers that prevent students from actually graduating from 4-year institutions. A college's ability to meet the financial, academic, emotional and social needs of a student once they are enrolled is a large determining factor on whether they will graduate. For evaluation purposes the college that most closely meets the previously mentioned needs will be referred to as the college of 'Best Fit'. The proposed roundtable will explore best practices in evaluating financial, academic, emotional and social factors in assisting students in college access programs with choosing a college of 'Best Fit'.
|
|
Session Title: Improving Outcomes for Patients, Caregivers, and Physicians
|
|
Multipaper Session 126 to be held in Lido C on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Health Evaluation TIG
|
| Chair(s): |
| Jessica Rice,
University of Wisconsin, rice4@wisc.edu
|
|
Development of a New Service Delivery Model for Managing Shoulder Injuries in a Workers' Compensation Population Using an Evaluation Framework
|
| Presenter(s):
|
| Pamela Wasson, WorkSafeNB, pam.wasson@ws-ts.nb.ca
|
| Barbara Keir, WorkSafeNB, barb.keir@ws-ts.nb.ca
|
| Abstract:
WorkSafe New Brunswick identified that claim duration and outcomes for clients with shoulder injuries were poor relative to other diagnostic groups. A review identified there was opportunity to improve access to appropriate medical management and align rehabilitation with current best practice. A new model for case management and medical management was developed. An evaluation model was used to test the model and determine the impact. A logic model was used to identify all stakeholders that served as inputs to the model, to determine outputs used in formative evaluation, including evaluation of the implementation of the program, used to provide feedback to providers and to revise the model as required. Short and long term outcomes formed the summative data that supported the decision to adopt this model for all New Brunswick injured workers with shoulder injuries.
|
|
An Examination of Perceptual and Structural Barriers to Participation in a Stroke Quality Improvement Network
|
| Presenter(s):
|
| Neale Chumbler, Veterans Affairs Center for Implementing Evidence-based Practice, neale.chumbler@va.gov
|
| Elizabeth Sternke, Veterans Affairs Center for Implementing Evidence-based Practice, elizabeth.sternke@va.gov
|
| Abstract:
We evaluated an ongoing health information and communication strategies established by the Department of Veterans Affairs (VA) Stroke Quality Improvement Network (SQUINT). SQUINT includes clinicians and administrators interested in improving VA stroke care quality and exchange strategies and best practices via monthly conference calls. Semi-structured telephone interviews were performed on 30 clinicians and administrators across twenty US VA facilities and were asked about their perceptions of the value of participating and what barriers (if any) keep them from participating. An in-depth qualitative analysis of the transcribed interviews was conducted using a consensus coding scheme. A dual approach to developing a coding scheme was employed using deductive and inductive methods. Key themes generated were the applicability and format of SQUINT calls and administrative support. These themes differed across clinicians and size of facility. This evaluation underscores the importance of tailoring health information and communication strategies to the needs of clinician type.
|
|
Reducing the Risk of Opioid Poisoning: Evaluation of a Community-Based Approach to Changing Physicians' Management of Chronic Pain Patients
|
| Presenter(s):
|
| Doug Easterling, Wake Forest University, dveaster@wfubmc.edu
|
| Jessica Pockey, Wake Forest University, jpockey@wfubmc.edu
|
| Montez Lane, Wake Forest University, ymlane@wfubmc.edu
|
| Abstract:
A community coalition in Wilkes County, North Carolina developed the Chronic Pain Initiative (CPI), a multi-pronged strategy to reduce opioid poisoning deaths. This evaluation investigated the changes in physician practice resulting from one of the CPI strategies - dissemination of a 'physician toolkit' containing various guidelines and templates to apply in assessing and managing chronic pain. The toolkit was delivered by the county medical director, who also provided education on the use of the tools as well as background on the scope of substance use problem within the county. Interviews with health care providers indicated that the intervention increased the use of Pain Agreements and the Controlled Substances Reporting System (CSRS). A review of publicly avialble health statistics showed a decline in the average number of narcotic prescriptions filled by local residents, as well as a decline in poisoning deaths concurrent with the intervention.
|
|
Evaluation of the American Cancer Society Caregiver Kit: A Randomized Trial
|
| Presenter(s):
|
| Joseph Bauer, American Cancer Society, joseph.bauer@cancer.org
|
| Dawn Wiatrek, American Cancer Society, dawn.wiatrek@cancer.org
|
| Greta Greer, American Cancer Society, greta.greer@cancer.org
|
| Abstract:
The American Cancer Society recently created a set of written materials for caregivers. Referred to as The Caregiver Kit, this resource provides information specific to cancer caregiving including topics such as communication, self care, and resources. A self-directed guided meditation CD for caregivers designed to reduce stress levels has also been included.
A randomized trial is currently underway to test the efficacy of these materials with caregivers contacting the American Cancer Society. Callers are randomized to one of three conditions: Caregiver Kit alone, Kit plus CD, and standard of care. All subjects are completing a baseline survey which includes measures of knowledge, efficacy, and frequency of engagement in key behaviors addressed in the kit and CD, as well as thoughts about Kit design. Follow-up surveys are planned for 1 and 6 months post enrollment, to measure changes in these variables. The sample size target is 100 subjects per arm.
|
| | | |
|
Session Title: Exploring Values Alignment in Five Evaluations of Science of Learning Centers
|
|
Multipaper Session 127 to be held in Malibu on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Brenda Turnbull, Policy Studies Associates, bturnbull@policystudies.com
|
| Discussant(s): |
| Gretchen Jordan, Sandia National Laboratories, gbjorda@sandia.gov
|
| Abstract:
During the past six years, and continuing for another five, the National Science Foundation has funded six Science of Learning Centers (SLCs), which are research consortia each focused on an interdisciplinary set of topics in the learning sciences (e.g., temporal learning, spatial cognition). NSF required the SLCs to engage an external evaluator to provide formative and summative evaluation services. Evaluation has had an interesting history among the SLCs, as the four center evaluators in this session will attest. The role of evaluators, client valuing of the evaluation enterprise, funder requests of the evaluation, and even the evaluation teams have changed considerably over the course of the SLC program life span. This session delves into the experiences of the evaluators as they have addressed issues of what is valued in research center evaluation, how it is valued, and by whom it is valued in these dynamic, political, multiple stakeholder environments.
|
|
It's an Evolution: Changing Roles and Approaches in the Evaluation of the Pittsburgh Science of Learning Center
|
| Brian Zuckerman, Science and Technology Policy Institute, bzuckerm@ida.org
|
|
The Science and Technology Policy Institute (STPI) has served as the Pittsburgh Science of Learning Center (PSLC)'s external evaluator since the Center's first year of operations. This presentation will describe changes in the Center logic model and evaluation approach in tandem with the evolution of the PSLC's research strategy and organization over the lifecycle of the award. Points to highlight include formative assessments of "centerness" in the early years of the award; the use of (and pitfalls in) bibliometric assessment of research quality and interdisciplinarity; and the development of a set of key indicators for annual analysis and visualization. The complexity of the role of the evaluator, and the need to balance sometimes-competing values and stakeholder concerns, will also be discussed.
|
|
Evaluating the Temporal Dynamics of Learning Center: Addressing Multiple Stakeholders' Information Needs
|
| Brenda Turnbull, Policy Studies Associates, bturnbull@policystudies.com
|
|
The Temporal Dynamics of Learning Center (TDLC) comprises four networks totaling 39 labs across 12 institutions. The evaluation uses member surveys to assess center functioning (e.g., communication about science and communication about administrative matters such as budgets), cross-lab collaboration, and trainee experiences. Responding to the center's aim of creating a "network of networks" for scientific research, the evaluators use social network analysis of survey data to document evolving cross-lab collaboration within and across TDLC's networks. Responding to NSF's interest in the value added by a center mode of funding, the evaluators survey a comparison group of scientists and compare their collaboration practices with those of TDLC investigators and trainees. Responding to external Site Visit Teams who review centers for NSF, the evaluators conduct bibliometric analyses of center and comparison scientists' publications. A challenge for the evaluators has been stakeholders' unfamiliarity with survey methods for evaluation.
|
|
Organizational Consultant, Critical Friend, and Evaluator: The Value and Challenge of Flexible Roles
|
| Kristine Chadwick, Edvantia, kristine.chadwick@edvantia.org
|
| Jessica Gore, Edvantia, jessica.gore@edvantia.org
|
|
The evaluations of the Science of Learning Center on Visual Language and Visual Learning (VL2) and the Spatial Intelligence and Learning Center (SILC) have required a great deal of flexibility. In both evaluations, Edvantia became the evaluator halfway through the centers' 5-year funding due to center leaderships' dissatisfaction with prior evaluators. In both cases, the leadership was not clear on what evaluation was needed and wanted to try something (someone) else. The leadership teams were unfamiliar with evaluation prior to the centers, and their values toward evaluation were forming during the first two or three years of center operation. Since Edvantia took over, the centers' leadership teams have valued the ability of the evaluator to be an organizational consultant and critical friend. Less valued by leadership yet highly valued by the funder has been the evaluation function-assessing the relative merit, "centerness," or "value added" of center-based research.
|
|
Coming from Behind: Developing a Logic Model and an Evaluation Strategy Late in the Center's Life Cycle
|
| Judah Leblang, Lesley University, bleblang@lesley.edu
|
|
Judah Leblang served as the primary evaluator for the Center of Excellence for Learning in Education, Science and Technology (CELEST) from July 2009 to June 2010. Until PERG took over the evaluation, CELEST had no logic model or rigorous evaluation plan. His evaluation work was focused on assessing the "centerness" or value of CELEST as a center, and in collecting and analyzing data for the project during Year 6, as well as reviewing key trends during Years 1-5. Leblang, in conjunction with PERG staff, worked with the CELEST PI, co-PIs and advisory board in order to prepare for the project's critical site review, which occurred in March 2010. CELEST had undergone a major reorganization shortly before PERG became the evaluators, and much of the evaluation focused on the organizational changes made by CELEST's leadership team, and how those changes impacted key stakeholders, including researchers, graduate students, and the participating institutions.
|
|
Session Title: The American Evaluation Association International Listening Project Think Tank
|
|
Think Tank Session 128 to be held in Manhattan on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the AEA Conference Committee
|
| Chair(s): |
| Victor Kuo, FSG Social Impact Consultants, victor.kuo@fsg.org
|
| Tristi Nichols, Manitou Inc, tnichols@manitouinc.com
|
| Presenter(s):
|
| Jim Rugh, RealWorld Evaluation, jimrugh@mindspring.com
|
| Ross Conner, University of California Irvine, rfconner@uci.edu
|
| Discussant(s):
|
| Patricia Rogers, Royal Melbourne Institute of Technology, Australia, patricia.rogers@rmit.edu.au
|
| Tessie Catsambas, EnCompass LLC, tcatsambas@encompassworld.com
|
| Efrain Gutierrez, FSG Social Impact Consultants, efrain.gutierrez@fsg.org
|
| Thomaz Chianca, COMEA Evaluation Ltd, thomaz.chianca@gmail.com
|
| Mary T Crave, University of Wisconsin-Extension, mary.crave@uwex.edu
|
| Abstract:
In November of 2010, a task force was established by the AEA Board of Directors to develop strategies for learning what might be appropriate roles for AEA in the international community. The International Listening Project Task Force seeks to learn how AEA can engage in ways that are mutually beneficial to the association and entities internationally. The short-term goal of the task force is to identify a set of activities that AEA can undertake with partners in the next few years in pursuit of AEA's commitment to providing international services and programs. Longer-term goals of the task force are to develop AEA policies addressing international programs and services in accordance with AEA's annual Goals Policies. A key aspect of this project is AEA's intention to "listen" to others among international communities. AEA values its relationship with partners in a global community and believes that AEA will only benefit by listening well to the perspectives, experiences, and needs of its partners. This Think Tank will be one of a series of activities to gather member ideas and suggestions for the project as well as to report to the membership on current activities.
|
|
Session Title: Assessing Student Engagement
|
|
Multipaper Session 129 to be held in Monterey on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Assessment in Higher Education TIG
|
| Chair(s): |
| Courtney Brown,
Indiana University, courtbrown@yahoo.com
|
|
Assessment of Student Personality Factors to Influence Student Attitudes Toward Group Experiences
|
| Presenter(s):
|
| Christine Perakslis, Johnson & Wales University, cperakslis@jwu.edu
|
| Stacey Kite, Johnson & Wales University, skite@jwu.edu
|
| Abstract:
The purpose of this study was to investigate differences between and among personality factors relative to levels of urgency and student attitudes toward group experiences
Previous studies primarily used the Myers Brigg Type Indicator (MBTI) or the Big Five Personality Factor Inventory (Big Five) (Halfhill et al., 2005). In contrast, this study used a behavioral (personality) assessment known as the Predictive Index (P.I.), not previously used to investigate student attitudes. In this study, the intensity of attitude (agreement) was reported by personality factors relative to levels of urgency for the following dimensions: 1.) benefits of interpersonal interactions; 2.) value that other students bring to group experiences; 3.) trustworthy attributes identified in other students during group experiences and; 4.) benefits of group experiences. Two individual items were utilized to investigate the differences between and among personality factors and student attitudes about: group experiences, in general, and the value of group experiences.
|
|
Assessing Student Engagement at the Course Level
|
| Presenter(s):
|
| Rick Axelson, University of Iowa, rick-axelson@uiowa.edu
|
| Arend Flick, Norco College, arend.flick@norcocollege.edu
|
| Marc Pizzimenti, University of Iowa, marc-pizzimenti@uiowa.edu
|
| Tina Taylor-Ritzler, Dominican University, tritzler@dom.edu
|
| Abstract:
For many instructors, an understanding of how to engage their students more productively in learning activities is an ongoing and fundamental challenge. Although there are many survey instruments for studying the prevalence of engagement, they generally do not provide the detailed diagnostic information instructors need to make courses more engaging for their students. To fill this gap, we have been developing and testing a course-level engagement survey instrument that can be administered to students. The survey items are designed to assess student's cognitive, affective, and behavioral engagement as well as aspects of the course that are enhancing or inhibiting these components of engagement. In this session, we will discuss survey results from courses at three institutions (University of Iowa's Medical School, Norco College, and Dominican University) where the instrument has been tested.
|
|
Evaluating the Impact of Service Engagement at Higher Education Institutions: The Process of Creating, Measuring, and Sustaining
|
| Presenter(s):
|
| Courtney Brown, Indiana University, courtbrown@yahoo.com
|
| Abstract:
This presentation provides an evaluation created and implemented across a representative sample of higher education institutions in the Midwest developing service engagement initiatives. More specifically the evaluation developed was to measure the effects of community engagement, service activities, service-learning courses and volunteer opportunities on students enrolled in a given institution. This presentation will discuss the roadblocks to evaluating such initiatives, provide the process for developing a survey instrument across a wide variety of stakeholders (and opinions), provide the survey instrument created, and examine the challenges for sustaining the evaluation measures set in place. This tool and evaluation process will be valuable to evaluators as they set out to work with higher education institutions who are interested in measuring the impacts of service engagement initiatives.
|
| | |
|
Session Title: How Software can Enhance Value and Expose Values in Qualitative Inquiry: The Value(s) of NVivo 9
|
|
Demonstration Session 130 to be held in Oceanside on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Qualitative Methods TIG
|
| Presenter(s): |
| Kari Greene, Oregon Public Health Division, kari.greene@state.or.us
|
| Linda Drach, Oregon Public Health Division, linda.drach@state.or.us
|
| Abstract:
Using NVivo can increase an evaluator's ability to meet the "value anchors" of our profession, the Guiding Principles and Program Evaluation Standards. By bolstering our technical capacity, NVivo offers value to both process and product in qualitative inquiry, allowing evaluators the opportunity to be fully transparent in our work. This didactic session will offer useful information for maximizing the utility of NVivo to conduct even the most straightforward qualitative data analysis. The presenters will cover the software fundamentals for beginner and intermediate users, highlighting the new features of NVivo 9. This session will demonstrate how to organize and manage data, incorporate spreadsheets into a project, code and query data in multiple ways, and visualize patterns and connections in the data. The session is intended to build the working knowledge of attendees to use NVivo in their everyday work and to demonstrate the value of computer-assisted qualitative data analysis.
|
|
Session Title: Values-driven Approaches to Monitoring Outcomes and Assessing Fidelity in Programs for Children and Youth
|
|
Multipaper Session 131 to be held in Palisades on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Human Services Evaluation TIG
|
| Chair(s): |
| Tania Rempert,
University of Illinois at Urbana-Champaign, trempert@illinois.edu
|
| Discussant(s): |
| Margaret Polinsky,
Parents Anonymous Inc, ppolinsky@parentsanonymous.org
|
|
The Value of a Center of Excellence in Getting to Outcomes for Children and Youth
|
| Presenter(s):
|
| Evangeline Danseco, Ontario Centre of Excellence for Child and Youth Mental Health, edanseco@cheo.on.ca
|
| Sherry McGee, Ontario Centre of Excellence for Child and Youth Mental Health, smcgee@cheo.on.ca
|
| Abstract:
What is the value of a 'center of excellence' in achieving better outcomes for children and youth? What outcomes are feasible and relevant from the perspective of internal stakeholders, external government funders, and service providers who are the primary target audience? In 2010, the Ontario Centre of Excellence for Child and Youth Mental Health obtained a new 6-year contract from the provincial government to improve the effectiveness of mental health services among children and youth through the use of evidence-informed practices. A strategic plan for 2010-2016 was developed with input from various stakeholders across Ontario. A performance measurement was also developed, reflecting various stakeholders' values and expectations for a 'center of excellence.' The framework is designed to effectively assess the results and impact of the Centre's work. This presentation will discuss the framework, proposed core outcomes and performance indicators, reflecting views from internal leadership, external stakeholders and government representatives.
|
|
Using Web-Based Systems to Monitor Service Delivery in a School-Based Aftercare Program for Youth Reintegrating Into the Home and Community School Settings Following a Stay in Out-Of-Home Care
|
| Presenter(s):
|
| Alexandra Trout, University of Nebraska, Lincoln, atorkelson-trout2@unl.edu
|
| Jacqueline Huscroft D'Angelo, University of Nebraska, Lincoln, jhuscrof@yahoo.com
|
| Abstract:
Monitoring service delivery is often noted as one of the greatest challenges in evaluating human service programs (Staff & Fein, 1994). This information is important for many reasons including understanding what services are provided to and required by clients, how provider time is spent, necessary resources, caseload feasibility, and model implementation and adherence. This information also provides valuable information for supervision and is critical to understanding the relationship between client outcomes and service provision. Although observations and case-notes allow for documentation of services, these methods may also present challenges which limit their utility. This session will describe a web-based approach that we have adopted for monitoring service delivery in an academic-focused aftercare program for high-risk youth reintegrating into the home and community school settings following a stay in out-of-home care. Lessons learned during the development process as well as strengths, challenges, and ideas for further enhancement will be discussed.
|
| |
|
Session Title: Topical Interest Groups (TIGs) as Communities of Practice: Strategies for Building a Community of Practice
|
|
Think Tank Session 133 to be held in Palos Verdes B on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the AEA Conference Committee
|
| Presenter(s):
|
| Manolya Tanyu, American Institutes for Research, mtanyu@air.org
|
| Discussant(s):
|
| Tom McKlin, The Findings Group LLC, tom@thefindingsgroup.com
|
| Sheila Robinson Kohn, Greece Central School District, sbkohn@rochester.rr.com
|
| Chad Green, Loudoun County Public Schools, chad.green@loudoun.k12.va.us
|
| Tiffany Berry, Claremont Graduate University, tiffany.berry@cgu.edu
|
| James Sass, Rio Hondo College, jimsass@earthlink.net
|
| Javan Ridge, Colorado Springs School District 11, ridgejb@d11.org
|
| James P Van Haneghan, University of South Alabama, jvanhane@usouthal.edu
|
| Anane Olatunji, Alignment Ed LLC, aolatunji@alignmenteducation.com
|
| Abstract:
Over the past few years, the PreK-12 Educational Evaluation Topical Interest Group (TIG) has worked to solidify its own leadership and better communicate with its members. In the process of doing this, we have asked several other TIGs about their activities and what they do to form a community of practice. This Think Tank will summarize the range of activities AEA TIGs perform to build strong communities. It also aims to facilitate a conversation among leaders and members of AEA's 46 TIGS about the purpose of the TIGs and ways to encourage the development of online and distant communities of practice as a growing venue of professional development.
|
|
Session Title: Evaluation Policies That Matter: Naming Them and Explaining Their Value
|
|
Think Tank Session 134 to be held in Redondo on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Evaluation Policy TIG
|
| Presenter(s):
|
| Margaret Johnson, Cornell University, maj35@cornell.edu
|
| Discussant(s):
|
| Kristin Kaylor Richardson, Western Michigan University, kkayrich@comcast.net
|
| William Trochim, Cornell University, wmt1@cornell.edu
|
| Abstract:
What evaluation policies matter most? How might evaluation policies affect the quality of programs and services? What values might be relevant in establishing and implementing evaluation policies within different service contexts and under different political, economic and social conditions? The purpose of this think tank is to provide an opportunity for evaluators working in agency, community, state or federal levels of governance and service provision to reflect upon the nature of evaluation policies and their potential impact. Discussion will center on: 1) Identifying and classifying examples of evaluation policy; 2) Considering possible mechanisms through which evaluation policy may influence program quality; and 3) Envisioning how an inventory approach can inform evaluation policymaking and help us develop coherent, sensible evaluation policies to support high quality evaluation work at all levels of practice.
|
|
Session Title: Somewhere Beyond the Sea: International Evaluation
|
|
Multipaper Session 135 to be held in Salinas on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Assessment in Higher Education TIG
|
| Chair(s): |
| George Reinhart,
University of Maryland, greinhart@casl.umd.edu
|
|
'What do you think about'?: Formative Assessment Practices at University
|
| Presenter(s):
|
| Serafina Pastore, University of Bari, serafinapastore@vodafone.it
|
| Abstract:
In the light of the changes which have affected the Italian University system so far, a new evaluation practice is widely standing out. It mainly aims to embrace the students' opinions about the formative services provided/received within a University degree course.
Higher education assessment does not attract the scholars' current research interests. As a result, learning assessment seems to be practised more as a bureaucratic tool: this kind of evaluation is useful to collect data about student's characteristics (expectations, motivations, etc.), teacher's quality, but it is not really functional to student's learning.
I will present findings of a multiple case study conducted at the Faculty of Education, University of Bari. The paper will examine a common evaluation belief and its related problems, but also different conceptions and practices of assessment which reflect different values and purposes (both for teachers and for students).
|
|
Assessment of Degree Awarding Powers in Sweden - Methods and Examples
|
| Presenter(s):
|
| Tomas Egeltoft, The Swedish National Agency for Higher Education, tomas.egeltoft@hsv.se
|
| Robert Östberg, The Swedish National Agency for Higher Education, robert.ostberg@hsv.se
|
| Abstract:
In the Swedish higher education system all Higher Education Institutions must apply to the Swedish National Agency for Higher Education for entitlement to award professional degrees on all three academic levels, e.g. for engineering, medical and different teacher degrees.
In the assessment process the Agency is using an external expert group consisting of university teachers, students and other stakeholders. An application is reviewed towards the Swedish Higher Education Act and Ordinance, with seven given quality aspects.
In April 2010 the Swedish government decided to replace the existing teacher degree with four new degrees. The decision also included that all universities/university colleges who wanted to give one or more of these new study programs must apply for the entitlement to award the corresponding teacher degree.
The presentation contains a description of the general assessment process in Sweden, with examples from the above mentioned assessment project with a large number of applications.
|
|
Teachers' Reflections as a Means to Evaluate Teacher Performance: The Case of a Colombian University
|
| Presenter(s):
|
| Alexis Lopez, Universidad de los Andes, Colombia, allopez@uniandes.edu.co
|
| Gabriel Cote, Universidad de Pamplona, Colombia, gabrielcote@yahoo.com
|
| Ruth Torrenegra, Universidad de Los Andes, Colombia, re.torrenegra26@uniandes.edu.co
|
| Abstract:
In this paper we present the results of a qualitative study that examines the use of a new teacher evaluation system to improve teaching practices. This new evaluation system requires all the teachers in the Department to reflect on their teaching practices on an ongoing basis. The evaluation system uses for instruments: 1) a teaching portfolio, 2) self-assessment of a video-taped class, 3) peer evaluation, and 4) students perceptions (course evaluation form). We found that these reflections allowed teachers to set clear performance expectations, to monitor their teaching practices, and to improve their performance in the classroom. The assessment system also allowed the Department to evaluate teacher performance and to use this information to make important decisions to improve their program. The teachers felt that this evaluation system was useful and felt empowered by their active role in the evaluation process.
|
| | |
|
Session Title: Case Studies in Urban Marginality, Renewal and Gentrification on the Poor and Vulnerable in Chicago, Illinois, Pittsburgh, Pennsylvania, and Portsmouth, Virginia
|
|
Multipaper Session 136 to be held in San Clemente on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Advocacy and Policy Change TIG
|
| Chair(s): |
| Carol Camp Yeakey, Washington University, St. Louis, cyeakey@artsci.wustl.edu
|
| Abstract:
The purpose of this symposium is to examine the impact of urban renewal and gentrification on the poor, in three cities in the U.S.: Pittsburgh (Pennsylvania), Portsmouth (Virginia), Chicago (Illinois). The symposium draws upon the urban haves and have-nots to dramatically portray the intended and unintended consequences of urban renewal. All presentations will seek to answer the critical question, who benefits? These multi-city studies contribute further to our understanding of urban inequality and urban marginality based on original city wide data. Examination of federal, state and city public policies pertinent to urban renewal in each site will be analyzed, as well as relevant site based census data.
|
|
A Case Study of Urban Renewal and Gentrification in Portsmouth, VA, 2000-2010
|
| Judith Brooks Buck, Virginia State University, jbec2000@msn.com
|
|
This paper examines the unique plan to support the revitalization of the city of Portsmouth, Virginia, funded by a Hope VI grant under the auspices of the U.S. Office of Housing and Urban Development. The twenty four million dollar grant award was to be used to change conditions for public housing residents, as well as individuals and properties in other impoverished regions of the city. The theme of the grant included the notion of moving the residents from "GǪhopelessness to hope" by providing living wage employment, home ownership, education and training and safe environments. Although such plans were said to include resident input, the hopes and dreams of a substantial portion of the persons for whom the grant proposal was designed left the project without fulfilling their unique aspirations. Some hopes were thwarted by grant design and implementation flaws; while others were interrupted when the primary issue entrepreneur lost a city election. While the crumbling stock of substandard public housing was demolished and new homes were built, the former public housing residents were dispersed to a variety of different areas. Some moved to homes, and some to homelessness. Bankers, builders, and businesses left the project financially enriched. Preliminary findings suggest that the policy and the structures for service provision created incentives for the maintenance of urban inequality and marginality.
|
|
A Case Study of Urban Promise and Neglect in Post-Industrial Pittsburgh, 1990 - 2010
|
| Rodney Hopson, Duquesne University, hopson@duq.edu
|
|
In several independent surveys over the last decade, the city of Pittsburgh has ranked among the top 25-50 cities in the world in terms of stability, health care, culture, environment, education and infrastructure (Carpenter 2010). However, another revealing set of studies which provide indicators of quality of life by race and ethnicity in the Pittsburgh region, suggest that the city is one of the most racially segregated communities of its size in the country (Center on Race and Social Problems, 2007). How could this be, Pittsburgh as most livable but among the most racially segregated? To explain the extent to which Pittsburgh's promise and neglect are portrayed, this paper will analyze a plethora of data points relevant to the changes in one particular neighborhood in post-industrial Pittsburgh through closer study of income distribution, schooling populations, poverty, underachievement, crime, housing, joblessness, and other social, political and economic indicators over the last 20 years. Building on the conceptual frameworks of urban inequality in major U.S. metropolises (O'Connor, Tilly & Bobo 2001; Orfield, 2002; Varady, 2005), the paper provides a poignant picture of the causes and consequences of urban promise and urban neglect in Pittsburgh. Compelling vignettes of those residents for whom urban renewal provides and provided promise are presented, as well as vignettes from those for whom urban renewal provides and provided urban neglect, across generational lines.
|
|
A Case Study of the Demise of Public Housing Through Urban Renewal in Chicago, Illinois, 1990-2010
|
| Carol Camp Yeakey, Washington University, St Louis, cyeakey@artsci.wustl.edu
|
|
Because of problems associated with high density poverty, the demise of large scale, high density, public housing for the poor, through urban renewal, has been a goal of national public policy for several decades. For purposes of this presentation, urban renewal is defined as the politics of contested space (Todd & Swanstrom 2009). This study is a preliminary investigation of the demise of public housing, through urban renewal, in Chicago, Illinois, from the period l990-2010, utilizing Chicago's Plan of Transformation as a key point of departure. In particular, this research seeks to examine the dislocation of the public housing residents in the largest public housing experiment in Chicago and in the country, the Robert Taylor projects. The Robert Taylor Projects consisted of twenty-eight high rise buildings built in l962 between 39th Street, on the northern boundary and 57th Street on the southern boundary, stretching for two miles. The last residents moved out in 2005. Utilizing the multi-layered theoretical framework of Hyra (2008), Popkin (2005), Freeman (2006), Gordon (2008), Bradford (2009), and Koval et.al.(2006), the study is guided by the intersect of local (community),city, state, federal and international policies which impact the decisions and non-decisions that cities make to ensure their ability to not just survive, but to thrive in a competitive global economy. This research investigation is designed to answer the generic question, what have been the residential and relocation housing patterns for residents from the Robert Taylor Housing Projects in Chicago? What have been the neighborhood impacts of urban renewal and gentrification on former residents of the Robert Taylor Housing Projects? Preliminary findings suggest disturbing trends. The first trend observed, from preliminary data analysis, suggests that neighborhood poverty may have as strong an impact on future social mobility of children as parents' education, employment status or marital status. A second disturbing trend is the suburbanization of poverty, a geographic re-clustering based on race and class leading to dramatic increases in re-segregation, neighborhood deterioration, crime, poor schools and a host of urban ills formerly associated with public housing projects.
|
|
Session Title: Empowerment Evaluation: Creating Baselines, Goals, Benchmarks, and Documenting Performance
|
|
Multipaper Session 137 to be held in San Simeon A on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Chair(s): |
| David Fetterman, Fetterman & Associates, fettermanassociates@gmail.com
|
| Discussant(s): |
| Abraham Wandersman, University of South Carolina, wandersman@sc.edu
|
| Abstract:
My father used to say: "if you don't know where you are going, any road will get you there." Empowerment evaluation uses many tools to help organizations reach their goals and arrive at their "destination." As empowerment evaluators we help communities establish baselines. We also help them determine specific, measurable goals. In order to reach those goals, we help communities create meaningful benchmarks, which helps them monitor their own performance (and status reports for donors). The bottom line, however, is performance. We work with communities to help them document their actual performance and compare it with their goals. We help them compare their performance with their benchmarks as well to determine if they are on track or need to make mid-course corrections. This session uses a tobacco prevention program in Arkansas (funded by the Minority Initiative Sub-recipient Grant Office), to demonstrate the use of this empowerment evaluation tool.
|
|
Empowerment Evaluation: The Power of Visualization
|
| David Fetterman, Fetterman & Associates, fettermanassociates@gmail.com
|
|
Empowerment evaluation uses many techniques to assist community members accomplish their objectives. Visualization is used at each step including: 1) mission; 2) taking stock; and 3) planning for the future. Community members are asked to visualize what they would like to see at the end of the day in their community, e.g. safe neighborhood, good schools, and affordable housing. This is used to help them crystallize and record their mission. They also use numerical tables and bar charts to present a powerful visual image of their taking stock ratings (current performance. They also use them to highlight where they started, where they want to go, and where they are at the moment, e.g. baseline, goal, benchmark, and performance. Finally, empowerment evaluators encourage community members to visualize their new goals, develop strategies to accomplish those goals, and identify credible evidence to document their efforts and plan for the future.
|
|
Empowerment Evaluation: Creating Realistic Goals and Benchmarks
|
| Beverly Tremain, Public Health Consulting LLC, btremain@publichealthconsulting.net
|
|
We are using an empowerment evaluation approach to help tobacco prevention non-profits in counties throughout the State of Arkansas.
Many grantees are overly ambitious and overpromise because they want to make real and rapid changes in the community. However, this approach can backfire if they expect too much change too quickly in their own communities.
One tobacco prevention agency wanted to: "eliminate second hand smoke in apartment complexes in several counties in Arkansas through a 25-foot perimeter law enacted by 2011 in one county and 2015 in five counties."
The community advocate representing this tobacco prevention agency was surprised to learn that tenants in an apartment complex would not even complete his survey about their smoking habits, never mind help to eliminate second hand smoke.
The problem was that the goal was not realistic. This presentation highlights how we worked with them to set realistic goals and benchmarks.
|
|
Empowerment Evaluation: Using SMARTER Goals, Objectives, and Activities
|
| Linda Delaney, LFD Consulting LLC, linda2inspire@earthlink.net
|
|
We are assisting tobacco prevention agencies align SMART (Specific, Measurable, Achievable, Realistic, and Time-bound) goals, objectives, and activities with their mission. "A Brief Guide to SMART goal setting" may be found at http://www.executive-and-life-coaching.com/support-files/smartgoalsettingworksheet.pdf. After setting SMART goals, we help them evaluate their programs to determine if they are achieving their goals. Working closely with community and agency personnel, we encourage them to log both positive and negative events as they occur to track their activity and identify any problems at early stages. In essence, they are using event logs to allow them to take stock of where they have been, where things are now, and to provide "intelligence" that helps them plan events for the future that smartly align with the mission.
|
|
Session Title: Evaluating Science Programs in K-12 Schools
|
|
Multipaper Session 138 to be held in San Simeon B on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Shira Solomon,
CNA Education, solomons@cna.org
|
| Discussant(s): |
| Kimberle Kelly,
Wisconsin Center for Education Research, kakelly3@wisc.edu
|
|
Polymer Science Education for Middle School
|
| Presenter(s):
|
| Jane Beese, University of Akron, jab128@uakron.edu
|
| Carin Helfer, University of Akron, chelfer@uakron.edu
|
| Abstract:
The Polymer Science Education for Middle School Project provides a model of curriculum and instruction for middle school students, which addresses the need for science career awareness embedded in standards-based science education. The federally funded project spanned a three-year period from 2008-2011. This project involved a learning community of pre-service and in-service science and math teachers; middle school students from a suburban middle school district; professional scientists and engineers; university science education professors, and parents. Project activities included teacher professional development, classroom instruction of science units, Polymer Family Night, career day, and a field trip to a university research laboratory. The purpose of this project and evaluation is to inform members of the science education community interested in interventions and to increase the pool of potential scientists and engineers from the middle school population to enter the higher education pipeline and the professional science community.
|
|
The Maps in Medicine Program: An Evaluation of the Development and Implementation of Life Sciences Curriculum
|
| Presenter(s):
|
| Jennifer O'Malley, Saint Louis University, jomalley@stchas.edu
|
| Abstract:
It is essential for individuals to have an understanding of scientific concepts and many programs have been developed to remedy the dire situation of the lack of science proficiency in the United States. One solution has been provided by the Maps in Medicine (MiM) program at the University of Missouri Columbia, funded by the Howard Hughes Medical Institute. Newly developed MiM curricula are intended to promote positive attitude changes in those students who are currently underrepresented in Science, Technology, Engineering and Mathematics (STEM) fields, with the program also providing professional development to high school science teachers. It was important to determine if MiM's solution to the science education problem has been successful and so the program evaluation piece was integral. A mixed-methods approach was used to evaluate the MiM program. Formative evaluation results indicate a positive response from teachers regarding curriculum and professional development and have identified appropriate improvements.
|
|
Planning and Conducting an Evaluation of an Inquiry-based Science Kit Intervention in Four Public School Districts
|
| Presenter(s):
|
| Sarah Brasiel, Edvance Research Inc, sbrasiel@edvanceresearch.com
|
| Eric Rolfhus, Edvance Research Inc, erolfhus@edvanceresearch.com
|
| Leslie Grunden, Edvance Research Inc, lgrunden@edvanceresearch.com
|
| Abstract:
Evaluating program implementation and related student and teacher outcomes can be challenging. This paper presents the challenges and lessons learned from an external evaluation of a regional program designed to improve science and literacy achievement for high-needs students in grades K- 6. The program's goals were to strengthen teacher understanding of science and quality of instruction in 4 districts, through professional development and coaching that supports inquiry-based science kit instruction. Lessons learned for evaluators include the importance of communication with project researchers to understand the day-to-day implementation issues that could potentially bias analytic findings, the challenges around access to student-level data to create propensity-score matched samples when random assignment is not possible, the use of descriptive data and graphical representations to improve clarity of the findings to local stakeholders, and documenting implementation issues throughout the study to guide researchers in improving implementation in future years.
|
|
A New Method to Measure Elementary Teachers' Pedagogical Content Knowledge in Science
|
| Presenter(s):
|
| Nena Bloom, Northern Arizona University, nena.bloom@nau.edu
|
| Barbara Austin, Northern Arizona University, barbara.a.austin@nau.edu
|
| Sandie Grinnell, Mount Elden Middle School, sgrinnell@fusd1.org
|
| Jane Kirkley, Northern Arizona University, jane.kirkley@nau.edu
|
| Abstract:
Pedagogical content knowledge (PCK) is one of the three frameworks identified by Shulman (1986) as essential knowledge for teaching. Since that time, numerous books, articles, and research projects have been devoted to examining and explaining PCK throughout the K-20 enterprise. PCK is considered inextricably correlated with teaching quality. Although Shulman defined and described this term, consensus as to what this construct means has never been fully reached in the education community. Consistent with the lack of definition of the term has been the lack of an instrument to adequately measure the construct itself or subconstructs of it (Fredrichsen et al., 2011), making evaluation of PCK growth in science education programs challenging. The authors of this paper will present the pilot results of a new method for measuring elementary teachers' application or operationalization of their pedagogical content knowledge in science using a lesson sequencing task.
Friedrichsen, P., Van Driel, J., & Abell, S. (in press). Taking a closer look at science teaching orientations. Science Education.
Shulman, L.S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4-14.
|
| | | |
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Holding up a Mirror: Investing in an Internal Implementation Evaluation as a Vehicle to Document a Funder's Process and Inform Decision-making |
|
Roundtable Presentation 139 to be held in Santa Barbara on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Internal Evaluation TIG
|
| Presenter(s):
|
| Artineh Samkian, First 5 LA, asamkian@first5la.org
|
| Jessica Monge, First 5 LA, jmonge@first5la.org
|
| Abstract:
This presentation documents a funder's process in developing a research and evaluation agenda as we move from a project-based funding strategy to a primarily place-based one. Special attention is given to an implementation evaluation, which will systematically document the process of carrying out our strategic plan so as to inform decisions. Consistent with the growing literature on utilization-focused evaluation (Patton, 1997) and the expressed need to ensure stakeholder buy-in and engagement (Preskill & Caracelli, 1997; Taut & Alkin, 2003; Johnson et al., 2009) it is critical to attend to the context in which the evaluation is taking place and to attend to the role of the evaluator. As a semi-internal, semi-external evaluation that reflects on and scrutinizes our own process as a funder, this case study poses an interesting opportunity to examine how the use of findings is affected by both the context and the evaluators responsible for providing feedback.
|
| Roundtable Rotation II:
Building and Cultivating Internal Evaluation in the Not-for-Profit Sector: A Critical Reflection of Our Roles |
|
Roundtable Presentation 139 to be held in Santa Barbara on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Internal Evaluation TIG
|
| Presenter(s):
|
| Lymari Benitez, Harlem Children's Zone, lbenitez@hcz.org
|
| Kira Krenichyn, Ramapo for Children, kkrenichyn@ramapoforchildren.org
|
| Abstract:
In recent years, the role of evaluators in the not-for-profit sector has changed. Increased delivery of social services, government initiatives and associated demands for reporting and accountability have created the need for nonprofits to build internal capacity for program evaluation. As internal evaluators, we are faced with the challenges of external and internal demands for evaluation data and results. Although our experiences are different (one of us the first and only internal evaluator at a small organization, and the other a part of a large evaluation team), we both agree that successful internal evaluation needs to be participatory and requires support from senior management. This roundtable will be an opportunity for internal and external evaluators at all levels, or anyone whose work involves building internal evaluation capacity, to reflect upon their roles, share and discuss challenges associated with supporting not-for-profit work, and brainstorm strategies to build internal evaluation capacity.
|
|
Session Title: A Comparison of Bias Reduction Methods
|
|
Multipaper Session 140 to be held in Santa Monica on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Laura Peck, Abt Associates Inc, Laura_Peck@abtassoc.com
|
| Discussant(s): |
| Furio Camillo, University of Bologna, furio.camillo@unibo.it
|
| Abstract:
While researchers use a variety of methods to estimate treatment effects in quasi-experimental designs, propensity score adjustments have become increasingly popular. Although they can be effective in reducing selection bias, they are not without their limitations. To improve statistical estimates from non-randomized studies, researchers modify propensity score methods and develop new methods. Recently, Peck, Camillo, and D'Attoma (2010) proposed a new method to reduce selection bias in quasi-experiments using an innovative clustering approach to create balanced treatment-comparison groups. Bias and treatment effects are assessed within each cluster permitting comparisons of equivalent groups without compromising the factorial structure of the data. As with any new statistical method, the popularity of this method depends on its effectiveness when compared to existing methods. Therefore, this panel will compare bias reduction between their new method and propensity score matching across three applications.
|
|
A Comparison of Bias Reduction Methods on Educational Outcomes
|
| M H Clark, University of Central Florida, mhclark@mail.ucf.edu
|
|
This study compares bias reduction in quasi-experiments between Peck, Camillo, and D'Attoma's (2010) method and propensity score matching (PSM) using data from a non-randomized evaluation of a first-year college orientation seminar. Clark and Cundiff (2011) conducted an evaluation of a first-year seminar in which participants self-selected into the course. Without any statistical adjustment, program participants had lower first-year GPAs than non-participants. However, of the 25 covariates measured, 10 were not balanced between the treatment and control groups, suggesting that the treatment effect was biased. After matching cases on propensity scores, the negative effect on GPA that was originally found was nullified. While treatment groups were balanced reasonably well, nearly half of the treatment cases were dropped because they did not have suitable matches. Using the same covariates, I will examine the extent to which similar or better bias reduction and case retention occurs using Peck et al.'s clustering method.
|
|
How Does Welfare Use Affect Charitable Activity? A Tale of Two Methods
|
| Laura Peck, Abt Associates Inc, Laura_Peck@abtassoc.com
|
|
Motivated by Brooks (2002, 2004), prior work (Guo & Peck, 2009) analyzed the extent to which welfare recipients engage in giving money and time to charitable causes. Using the 2003 Center on Philanthropy Panel Study (COPPS) data, we used a difference-in-difference approach to overcome sticky issues of selection bias and found evidence that public assistance receipt tends to suppress monetary donations but may increase volunteer time. The proposed paper updates that work, using the 2005 COPPS data, and applying two other methods to address the same question. Specifically, we use (1) a propensity score matching approach, and (2) a multivariate cluster-based approach to generate useful comparison groups of non-welfare-recipients to estimate the effect of welfare on charity-related outcomes. We explicitly compare the effectiveness of these approaches in reducing selection bias, in an attempt to provide methodological guidance to analysts similarly vexed by selection bias.
|
|
Subjective Choices in Propensity Score Applications: A Comparison With the Multivariate Cluster-based (MCB) Method
|
| Ida D'Attoma, University of Bologna, ida.dattoma2@unibo.it
|
| Furio Camillo, University of Bologna, furio.camillo@unibo.it
|
|
This study evaluates the performance of the Multivariate Cluster Based (MCB) method (Peck, Camillo & D'Attoma, 2010) in terms of bias reduction, case retention, achieved balance and subjective judgment reduction compared to Propensity Score (PS) Stratification using simulated data. Because propensity score requires subjective choices about model specification, we expect that MCB will involve less subjective judgment in developing balance between treatment and comparison cases. The analysis applies a global imbalance (GI) measure introduced in prior work to simultaneously examine balance in pre-treatment categorical covariates. We assert that the GI measure is objective because of its consideration of variance observed in the data along baseline case and the treatment assignment variable. We expect to find better bias reduction and hope to find at least similar case retention using the MCB method compared to conventional propensity score matching.
|
|
Session Title: Assessing Day-to-Day Leader Performance to Drive Organizational Human Capital Strategies: A Software Demonstration
|
|
Demonstration Session 141 to be held in Sunset on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Business and Industry TIG
|
| Presenter(s): |
| William Hinshaw, Applied Leadership Solutions, whinshaw@appliedleader.com
|
| John Thompson, Horizon Performance, jat@horizonperformance.com
|
| Chris Mills, Horizon Performance, chrism@horizonperformance.com
|
| Abstract:
Horizon Performance and Applied Leadership Solutions have partnered to help clients implement personnel performance assessment solutions. The primary tool in these solutions is a software system developed for units within the United States Special Operations Command and NASA to address the challenges of assessing, selecting, and developing leaders for complex and stressful environments. The success of these engagements has led to the creation of a civilian version for corporate talent management. By having the capability to assess performance on mobile devices and/or computers, this software allows organizations to transition talent assessment from occasional anecdotal reports to a behaviorally based, data driven system providing a robust assessment of individual and organizational strengths and weaknesses. This demonstration will show how to make performance assessments using this software and how resulting data can be used for leadership development. Attendees can then discuss potential benefits and drawbacks of organizations using this tool.
|
|
Session Title: Uses of a Statewide Longitudinal Data System to Evaluate and Inform Programs, Policies, and Resource Allocations
|
|
Panel Session 142 to be held in Ventura on Wednesday, Nov 2, 4:30 PM to 6:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Deborah Carran, Johns Hopkins University, dtcarran@jhu.edu
|
| Abstract:
The state of Maryland has been actively working toward the implementation of their Statewide Longitudinal Data System (SLDS) for 15 years. Within the past year, the SLDS has begun to be used by the state to evaluate policy and other issues critical to inform educational reform. This panel proposes to present four papers (1) describing the SLDS, (2) using the SLDS to evaluate the impact of early childhood intervention services (birth to 3) on fall kindergarten assessment scores, (3) using the SLDS to track student performance on fall kindergarten assessment to evaluate grade 3 academic performance, and (4) using SLDS to identify students with disabilities at age 3 (exiting Part C early intervention services) and prospectively track students to grade 3 to evaluate the impact of early childhood (Part C) services.
|
|
The Maryland Longitudinal Data System
|
| Tamara Otto, Johns Hopkins University, tamaraotto@jhu.edu
|
| Jacqueline Nunn, Johns Hopkins University, jnunn@jhu.edu
|
|
The Maryland State Department of Education (MSDE) Division of Special Education/Early Intervention Services in partnership with the Johns Hopkins University Center for Technology in Education (CTE) has positioned itself among the leaders in the Nation in the race to not only develop a high-quality longitudinal data system, but to develop the user-friendly predictive analysis tools to enable the effective use of data by all key stakeholders.
In 2005, MSDE, in partnership with CTE, was awarded a three-year grant from the Department of Education's Institute of Education Sciences to design and implement a statewide longitudinal data system to help strengthen student achievement. Initially, the MD Scorecard was launched at the State and district level to provide a powerful tool in the analysis and use of the longitudinal data for decision making regarding programs, policies, and resource allocations.
The MD IDEA Scorecard has used qualitative and quantitative data to evaluate the effectiveness.
|
|
|
Evaluating the Impact of Early Intervention on Kindergarten Readiness
|
| Helena Mawdsley, Johns Hopkins University, hmawdsley@jhu.edu
|
| Jacqueline Nunn, Johns Hopkins University, jnunn@jhu.edu
|
| Deborah Carran, Johns Hopkins University, dtcarran@jhu.edu
|
| Tamara Otto, Johns Hopkins University, tamaraotto@jhu.edu
|
|
The Maryland Longitudinal Data System was used to evaluate impact of service provided to children (ages 0 to 3) enrolled in Part C early intervention services, on later kindergarten readiness. A total of 5,942 were linked with their subsequent fall kindergarten readiness test (MMSR) scores. Because Part C EIS delivers a wide range and types of services, only high incidence services were examined for this study. Six service types were found to have been provided to more than 15% of the matched participants, with speech/language therapy representing 74% of matched participants.
Evaluation findings support the benefit of Part C early intervention, that Part C EIS has a positive impact on children's readiness to enter elementary school. Regression analysis indicated that the greater the intensity of Part C early intervention services (as measured by maximum minutes per session and age of starting services), the better prepared children are for kindergarten.
| |
|
Data Mining Electronically Linked Grade Three Standardized Assessment Scores from Kindergarten Assessments to Identify Performance Patterns
|
| Deborah Carran, Johns Hopkins University, dtcarran@jhu.edu
|
| Jacqueline Nunn, Johns Hopkins University, jnunn@jhu.edu
|
| Tamara Otto, Johns Hopkins University, tamaraotto@jhu.edu
|
|
Linkage of unique student identifiers across grade levels has generated renewed interest in predicting high stakes test scores at early ages. Data mining, an iterative process using large extant data warehouses to discover meaningful patterns in data, examined the relationship between kindergarten assessments and grade 3 high stakes reading and math assessments. 152,105 kindergarten students were identified as receiving a kindergarten assessment between 2002 and 2005. Of these students, 100,957 were matched with their Grade 3 standardized math score and 100,978 with their Grade 3 Reading score, representing a 66% match rate. Using Classification and Regression Tree modeling analysis results are presented in tree-like figures with branches representing the splitting of cases based on values of predictor attributes. Results indicated that the kindergarten assessment is a moderately successful predictor of later high stakes testing performance; math performance was predicted better than reading
| |
|
Risk of Special Education Services at Age Eight for Children Receiving Early Intervention Services By Age Three
|
| Stacey Dammann, York College of Pennsylvania, sdammann@ycp.edu
|
| Deborah Carran, Johns Hopkins University, dtcarran@jhu.edu
|
|
A prospective risk analysis was used to evaluate whether the provision of early intervention services by age 3 would reduce the need for (intensity) of special education services in early elementary school. Children receiving Part C Special Education services at age 3 were prospectively tracked to determine educational placement in grade 1 (age 6).
Two cohorts of children who had historically received Part C early intervention services were identified; cohorts exited Part C programs in 2006 and 2007. Children were then tracked forward in the SLDS data base to determine their educational placement in first grade. Children were stratified by qualification categories in Maryland: High Probability condition, 25% Delay, and significant atypical development. A risk analysis was used to determine if children had lower risk of special education services if they had received early intervention services.
Results will be presented describing the estimated benefit of Part C services on children's later educational status.
| |
