|
Session Title: Learning From the Consequences of No Child Left Behind on Evaluation (Part 2 of 2)
|
|
Multipaper Session 770 to be held in International Ballroom B on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Rebecca Eddy,
Claremont Graduate University,
rebecca.eddy@cgu.edu
|
| Abstract:
The primary purpose of this panel (Part 2 of 2) is to explore what the field of evaluation has learned since the inception of No Child Left Behind (NCLB), and is a continuation of Part 1. Specifically, we will discuss how the practice of evaluation within the context of NCLB has itself been changed, and as a result, influences the practice of educational evaluation. In light of the reauthorization of NCLB set to occur in 2007, it is clear that this legislation has substantively impacted programs, schools, and districts (Part 1). The current panel extends this discussion in which we will explore challenges related to topic-specific foci, including both curriculum and assessment. In addition, we have included a culminating discussion of changes to evaluation purpose, design, and practice as well as a broader discussion of the challenges and opportunities for the field of evaluation in light of NCLB.
|
|
Measuring Student Progress: Changes and Challenges Under No Child Left Behind
|
| Mariam Azin,
Planning, Research and Evaluation Services Associates Inc,
mazin@presassociates.com
|
| Miriam Resendez,
Planning, Research and Evaluation Services Associates Inc,
mresendez@presassociates.com
|
|
The focus of this presentation will be to examine important issues in the assessment of student progress as a result of the No Child Left Behind Act (NCLB). In particular, NCLB has produced substantial changes on states' accountability systems and facilitated a plethora of assessment data that is potentially available to evaluators to use as 'outcome measures.' Thus, we will provide a brief overview of the current status of state accountability systems, including a description of the variability across state assessment systems in terms of what is being measured and how, and the intended and unintended consequences of such systems. In addition, we will discuss how evaluators can navigate the challenges we face in: a) using state assessment data to measure student progress adequately; and b) conducting educational evaluations in schools that are operating under the accountability umbrella of NCLB.
|
|
Using Research to Inform Educational Curricula
|
| Marcy Baughman,
Pearson Educational Measurement,
marcy.baughman@pearsoned.com
|
|
The No Child Left Behind Act (NCLB) changed the development and publishing process for K - 12 educational materials. Prior to NCLB, scientific research to support product development or to demonstrate product efficacy was not required, and some viewed it as a 'value-added' component to product development. However, NCLB mandated that, 'educational materials purchased with public funds must be proven by scientific research to improve student achievement in the classroom (NCLB, 2002).' Research has been added into the product development cycle with little to no impact on school districts. However, the integration of research into the post-publication product cycle has proven challenging. Publishers need to test their products in the classroom using rigorous research methods to test product efficacy, best practice implementation guidelines, teacher satisfaction, and ease of use. This panel will provide a discussion of the intended and unintended consequences of research on educational curricula and the classroom.
|
|
Consequences of No Child Left Behind on Evaluation Purpose, Design, and Practice
|
| Linda Mabry,
Washington State University, Vancouver,
mabryl@vancouver.wsu.edu
|
|
In 2003, the U.S. Department of Education proposed to prioritize for funding experimental, quasi-experimental, and other causality-focused designs in evaluation. Despite vigorous protest from some in the evaluation community, the priority was established as part of the No Child Left Behind Act of 2001 (NCLB). The priority threatened federal funding for the most frequently reported evaluation designs in Education, innovation in evaluation approach, contextualized formative feedback to program personnel, and ethical practice. It also reignited the "paradigm wars" over the relative merits of quantitative and qualitative methodology. The consequences will be discussed with reference to the analogous restrictions established by the World Bank and their consequences for evaluation design and practice and for societal consequences. In conclusion, an odd instance of contradictory federal initiatives related to NCLB will be discussed as to its implications for recognizing and understanding evaluation more broadly than the federal priority suggests.
|
|
No Child Left Behind and the Discipline of Evaluation: Challenges and Opportunities
|
| Rebecca Eddy,
Claremont Graduate University,
rebecca.eddy@cgu.edu
|
| Tiffany Berry,
Claremont Graduate University,
tiffany.berry@cgu.edu
|
|
This paper explores the challenges faced and opportunities afforded to the discipline of evaluation within the context of No Child Left Behind (NCLB). Specifically, this paper will detail such challenges as
(1) reconciling what evaluation means under NCLB with what it means outside of education;
(2) emphasizing outcomes at the expense of process assessment; and
(3) prioritizing compliance needs ahead of other stakeholder needs.
Opportunity costs associated with these challenges will also be explored. In response to NCLB, the evaluation community has been afforded several opportunities, many of which we should build upon as a growing discipline. For example, specific opportunities include
(1) increased funding for evaluation;
(2) capitalizing on evaluation as a transdiscipline across educational contexts (e.g., within schools, programs, curricula development, etc.);
(3) increased need for evaluator training; and
(4) ample opportunities for research on evaluation to examine empirically how NCLB has impacted the discipline of evaluation.
|
|
Session Title: Building Communities in the Context of Transitional Countries: Challenges and Approaches for Program Evaluation
|
|
Panel Session 771 to be held in International Ballroom C on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Chair(s): |
| Robert Stake,
University of Illinois at Urbana-Champaign,
stake@uiuc.edu
|
| Abstract:
This session will present some methodological cornerstones identified by practicing evaluators with regards to civil society strengthening and community development program. It also explores the roles that evaluation can play for such projects' implementing institutions and beneficiary communities. The panel also presents a hands-on experience of community program evaluation from four transitional countries: Macedonia, Slovakia, Belarus, and Ukraine.
|
|
Re-constructing Program Theory for the Post-ante Evaluation of the Ukraine-Belarus Partnership for Community Development
|
| Kseniya Temnenko,
Eurasia Foundation,
ktemnenko@eurasia.kiev.ua
|
|
Ukraine-Belarus Partnership for Community Development was a three-year joint program of the Eurasia Foundation, the Charles Stewart Mott Foundation, and the International Renaissance Foundation. It was implemented between 2003 and 2006. The goal of the Program was to foster community development initiatives in Belarus through cooperative activities between representatives of Belarusian local communities, such as local councils of deputies, NGO representatives, media and independent analytical centers, and their counterparts in Ukraine. As evaluators discovered there was a multi-faceted definition of the 'community' term: 1) a physical locality; 2) a network of experts/practitioners within the country; 3) cross-border network aimed at some form of civic activity. Beyond that there was a separate understanding of this term among Belarusian beneficiaries. The presentation discusses approaches of re-constructing program theory and undertaking post-ante evaluation in the absence of baseline data, and in restrictive political environment.
|
|
|
Institute for Sustainable Communities' Local Partnership Grantees in Macedonia: An Example of Mid-course Corrections in Evaluation Design as a Result of Evolving Understandings of Community and Capacity
|
| Gretchen Elias,
Institute for Sustainable Communities,
gelias@iscvt.org
|
|
Designed in response to a requirement by the donor agency (USAID), the Local Partnership component of ISC's Civil Society Strengthening Program (CSSP) in Macedonia is intended to cultivate community partnerships at the local level. It takes place in the context of the government's recent decentralization, and donor recognition of the need to build local government capacity and momentum for increased reliance on community-level solutions to local problems
ISC solicited applications for Local Partnership projects involving community-level partnerships without articulating our own definition of 'community' or anticipating the range of possible definitions that our grantees would propose. Therefore, they proposed communities that didn't fit with the measurement tools we'd been planning to use. In addition, the data collection methods and tools we had originally proposed were beyond the capacity of LPP grantees. In this presentation, I discuss how ISC has handled this challenge in redesigning its M&E activities and methods.
| |
|
Enabling Roma Children to Attend Ordinary Schools
|
| Robert Stake,
University of Illinois at Urbana-Champaign,
stake@uiuc.edu
|
|
A Roma settlement in eastern Slovakia, surrounded by the village of Jarovnice, survives despite risk of flooding from the nearby stream, and the persistent perception among villagers in Jarovnice that the Roma are 'separate'. Long after communism, little progress has been made toward assimilation. Perhaps a strong sense of community exists, but ordered, without common definition of membership. One contributing factor was that the settlement's children were consistently found unready--linguistically, culturally, and intellectually-for admission to first grade. To improve equity of educational opportunity, the Wide Open School Foundation, funded by the Open Society Institute, worked to prepare Roma children for elementary school. Despite limited funds, the Foundation found ways to engage Roma mothers in preparing their children, even in areas of which the women themselves were ignorant. The evaluation task was to explore how the foundation stimulated new perceptions and whether the community's own definition of community was changing.
| |
|
Session Title: Successfully Publishing Your Dissertation
|
|
Panel Session 773 to be held in International Ballroom E on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Graduate Student and New Evaluator TIG
|
| Chair(s): |
| Liz McGough,
University at Buffalo,
emcgough@buffalo.edu
|
| Discussant(s):
|
| Shelly Mahon,
University of Wisconsin, Madison,
mdmahon@wisc.edu
|
| Abstract:
The primary purpose of this session is to provide information for graduate students and recent graduates on the process of restructuring a dissertation or thesis into a published journal article or book chapter. Journal editors and authors will discuss the process of getting published. They will review the often implicit rules for writing an article or chapter including editing down a larger project like a dissertation or thesis into a publishable manuscript, targeting a journal, submitting a manuscript, understanding the review process, deciphering the editor's letter, revising and resubmitting the manuscript, and regrouping after rejection.
|
|
Where Do I Begin? Transforming Student Research Into a Publishable Manuscript
|
| Chris Coryn,
Western Michigan University,
christian.coryn@wmich.edu
|
|
Chris Coryn is PhD candidate at Western Michigan University. He has many journal publications and is currently working on a book based on his dissertation research. He will share his experience of the manuscript development and publication process from a student perspective.
|
|
|
From Conference to Publication; Transforming a Conference Presentation into a Volume of New Directions in Evaluation
|
| Michael Hamar,
Claremont Graduate University,
michael.hamar@cgu.edu
|
|
Michael Hamar is a graduate student at Claremont Graduate School. He is part of a team which has proposed and is working on an upcoming volume of New Directions in Evaluation; the proposal grew out of a session at the 2006 AEA conference. He will discuss his experiences with that project.
| |
|
Publishing Qualitative Research
|
| Sandra Mathison,
University of British Columbia,
mathison@interchange.ubc.ca
|
|
Sandra Mathison is a Professor of Education at the University of British Columbia, the Editor-in-Chief or New Directions in Evaluation, and a qualitative researcher. She will discuss her experience and offer suggestions for publishing qualitative research.
| |
|
Submitting Your Manuscript
|
| Jonathan Morell,
New Vectors LLC,
jamorell@jamorell.com
|
|
Jonny Morrell is an organizational psychologist, Senior Analyst with New Vectors, LLC, and the Editor in Chief of Evaluation and Program Planning. He has published many journal articles and will discuss the process of getting an article published in a peer-reviewed journal from initial submission through acceptance.
| |
|
Regrouping and Revising
|
| Robin Miller,
Michigan State University,
mill1493@msu.edu
|
|
Robin Miller is an Associate Professor of Psychology at Michigan State University, and Editor of the American Journal of Evaluation. She will discuss deciphering the editor's letter, regrouping from rejection, and revising and resubmiting the manuscript.
| |
|
Session Title: Community of Learners in Evaluation: The Triple Threat
|
|
Demonstration Session 774 to be held in Liberty Ballroom Section A on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Presenter(s): |
| Chantell Johnson,
TCC Group,
cjohnson@tccgrp.com
|
| Allison Crean,
Informed Educators Consulting Group,
allisoncrean@earthlink.net
|
| Abstract:
Evaluation is frequently performed in response to external contingencies, resulting in one-dimensional efforts that fulfill accountability requirements, but add little value to the growth of an organization. These efforts tend to focus on the "what" of evaluation (e.g., what protocols to follow, what data to collect, what analyses to perform, what to report).
When evaluative learning is the goal, and a Community of Learners process is employed, the "who, how, and why" of evaluation are addressed, and evaluation becomes a systemic triple threat: organizations understand the impact of specific programs, have an internal process and infrastructure for evaluation, and can integrate findings into larger organizational goals, including mission achievement.
Yet the cultural shift a Community of Learners often presents cannot be underestimated. Assessing an organization's readiness, determining baseline evaluation skills, and obtaining feedback on the process itself are key to tailoring the Community of Learners approach and facilitating its evolution over time.
Discover how a Community of Learners works, what TCC Group is learning through practice, and engage in dialogue as we consider where to take this approach in the future.
|
|
Session Title: What is or Should be the Value-added of an Evaluator?
|
|
Panel Session 775 to be held in Liberty Ballroom Section B on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Theories of Evaluation TIG
|
| Chair(s): |
| Melvin Mark,
Pennsylvania State University,
m5m@psu.edu
|
| Abstract:
This session addresses the question, "What is -- or should be -- the value-added of an evaluator?" The session takes as a given that there are many other specialists who will do the same kinds of work as do evaluators -- or at least what we might think of as AEA style evaluators. Alternatively stated, there are professionals available to do what we would call evaluation, but who have little if any familiarity with the theory, practices and professional standards specific to evaluation. For example, economists and strong methodologists are available to do large-scale outcome studies; organizational development specialists will perform work similar to that of many capacity building evaluators; etc. In light of this state of affairs, the session will address the question: What if anything is the value added when evaluative work is done by someone steeped in the evaluation literature, relative to some other kind of specialist?
|
|
Evaluators: Value-added, Value-subtracted
|
| Lois-ellin Datta,
Datta Analysis,
datta@ilhawaii.net
|
|
What is the value-added of evaluators, compared to members other fields, when assessing merit, worth, and value? Evaluators bring at least five potentially beneficial qualities. They more likely (1) begin immediately in helping clarify values, purposes, and questions, (2) are passionate about utilization, engaging stakeholders more, (3) attend to context and, in a systems sense, consider interconnections among the evaluand and the world, (4) expect the unexpected, being systematically concerned with the unpredicted and the emergent, and (5) work with what is, rather than what is predicted, in assessing attribution, contributions, causes. The evaluator further adds value by humility and appreciation, and thus can add to the team the method skills needed. This implies an evaluator trained well enough in allied disciplines to think critically about them, and also multi-lingual in evaluation theory and practice. Lacking these skills or the interpersonal skills of cross-discipline communication, the evaluator may be value-subtracted.
|
|
|
Working with Close Relatives
|
| George Grob,
Center for Public Program Evaluation,
georgeandsuegrob@cs.com
|
|
Several other professional groups carry out functions that are similar to some of those carried out by evaluators. This includes auditors, investigators, policy analysts, management analysts, lawyers, investigative reports, social scientists, program specialists in fields like education, health, and social services, and even lobbyists. This paper will briefly discuss: the nature of the overlap with these close relatives of ours; professional sensitivities; what can be done to minimize professional tensions; and, even more important, how to effectively join forces with them to solve problems and improve social well being. It will also look at the flip side of the equation - what does it take to persuade members of the allied professions who may be trained in methods that are similar to those of evaluators, but who may see things differently because of their professional upbringing.
| |
|
Value-added (or Subtracted) by an Evaluator
|
| William Shadish,
University of California, Merced,
wshadish@ucmerced.edu
|
|
The idea that training in evaluation can add value to evaluation is premised on the assumption that we can define training in evaluation in a way that would be deemed reasonable by the community of colleagues who do evaluations. This premise is questionable because the community of colleagues who do evaluations is multidisciplinary (e.g., economists, psychologists, sociologists). For example, an economist who does evaluation may believe that traditional training in economics is, in fact, training in evaluation, and further training is neither needed or desirable. However, a rephrased version of this question asks whether evaluators who are steeped in the literature common to the American Evaluation Association can add value to evaluation as a result. I will discuss this question in more detail in my talk, commenting on both the value added and subtracted from evaluations that are conducted by AEA evaluators.
| |
|
Value-Added by an Evaluator: What should it be? And so what?
|
| Melvin Mark,
Pennsylvania State University,
m5m@psu.edu
|
|
This paper first puts the question, 'What is - or should be - the value-added of an evaluator', in context. Related but distinct questions have been addressed, such as what are the competencies that evaluators should have. The value-added question includes a comparison standard (evaluative work by those in related fields), and thus will have a more limited answer. But the answer is critical, for our own self-assessment, for training, for communicating with funders, and for understanding the lessons learned about evaluation theory and practice. The paper also sketches a tentative answer to the value-added question. It rests largely on evaluation theory, on the good evaluator's ability to think contingently about evaluation purposes and methods, and on thinking about evaluation in a broader context. This paper also briefly addresses how to proceed in dealing with the value-added question, in light of the varied positions provided in this session.
| |
|
Session Title: Practicing Systemic Methods for Evaluation of Food Systems
|
|
Skill-Building Workshop 776 to be held in Mencken Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Systems in Evaluation TIG
|
| Chair(s): |
| Glenda Eoyang,
Human Systems Dynamics Institute,
geoyang@hsdinstitute.org
|
| Presenter(s):
|
| Ken Meter,
Crossroads Resource Center,
kmeter@crcworks.org
|
| Abstract:
Inner-city children, consumers, and farmers alike are impacted by global forces that are very difficult to understand from local points of view. Yet these broader issues deeply affect local contexts. Many evaluators are not trained to understand the complexity of food systems, though this realm is vital to community health and welfare. As in any evaluation setting, a multitude of issues interact. Participants with diverse points of view may carry diverse self-interests. The presenters work closely with complex systems including local foods initiatives. Using lessons learned in this field work, this workshop offers you both research insights and practical tools. These can be applied in more bounded settings as well. This presentation/experiential session will cover the following topics: - Complex dynamics in local and national food system work - Conceptual approaches to complexity - Identifying linked indicators that distill complex issues into concise measures.
|
|
Session Title: Multiple Methods, Multiple Perspectives
|
|
Multipaper Session 777 to be held in Edgar Allen Poe Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Extension Education Evaluation TIG
|
| Chair(s): |
| Melissa Cater,
Louisiana State University,
mcater@agcenter.lsu.edu
|
|
An Alternative Approach for Narrative Documentation of Extension Programs: Tales of an Internal Evaluator
|
| Presenter(s):
|
| Michael Duttweiler,
Cornell University,
mwd1@cornell.edu
|
| Abstract:
The utility of narrative documentation for program evaluation and accountability and as a tool for organizational learning is well established. In particular, story–based approaches have become widely accepted in both evaluation and organization development settings. Yet many organizations struggle to achieve intentional approaches for story collection that are both valued by practitioners and responsive to organization learning and accountability needs. This paper describes a pilot test in a statewide extension education system of the Most Significant Change Technique as described by Davies and Dart (2005). The test revealed organizational culture, unit size and structure, and perceived value in relation to existing processes as strong influences on acceptance of and commitment to the approach. Because the pilot test was led by an internal evaluator, the case also provides opportunity to explore how the evaluator's role shaped evaluation foci and methods employed.
|
|
Characteristics Associated With Increasing the Response Rates of
Web-based Surveys
|
| Presenter(s):
|
| Thomas Archer,
The Ohio State University,
archer.3@osu.edu
|
| Abstract:
Having a respectable response rate is critical to generalize the results of any survey. Web-based surveys present their own unique set if issues. This research identified web deployment and questionnaire characteristics that could be associated with increasing the response rate to web-based surveys based on a systematic evaluation of over 120 web-based surveys to a variety of audiences during three and a half years. Fourteen web deployment characteristics and nine web-based questionnaire survey characteristics were correlated with response rates. The resultant correlations prompted recommendations: [1] Allow more time between second and third contacts; [2] Potential respondents must be convinced of the potential benefit of accessing the questionnaire; and [3] Do not be overly concerned about the length or detail of the questionnaire - getting people to the web site of the questionnaire is more important to increasing response rates.
|
|
A Stakeholder Valuation Approach to Evaluating a Program's Public Benefits: The University of Minnesota Extension's Master Gardener Program
|
| Presenter(s):
|
| Tom Bartholomay,
University of Minnesota,
barth020@umn.edu
|
| Abstract:
This presentation will focus on the evaluation process used by the University of Minnesota Extension to evaluate the public benefits of its Master Gardener program. With increasing interest in the public benefit of Extension programs, this expert opinion approach is not only useful for identifying a program's public benefits, but assessing the degree to which those benefits are valued across primary stakeholder groups. The results can be used to identify the most valued and robust attributes of a program, which can then be used for program planning, effectively describing the program to counties, and defining future outcome-based evaluations.
This presentation will describe each step in the evaluation process, how the data was reported, the results of the Master Gardener evaluation, and how the information was used by the program.
|
|
Improving the Content of Penn State Cooperative Extension ArcView Geographic Information System Workshops Through Analysis of Participant Evaluations
|
| Presenter(s):
|
| Stewart Bruce,
Pennsylvania State University,
stew@psu.edu
|
| Abstract:
In 2006 the Penn State Cooperative Extension Geospatial Technology Program delivered fifteen introductory and intermediate ArcView GIS workshops to a total of 184 participants. A post workshop evaluation survey was given to each participant asking them to list one or two thing that they learned and to describe how they planned to use this information. Additional quantitative questions dealt with how the program would affect their work and the quality of the actual workshop. In addition, an opened ended question asking for comments and suggestions was also included. A follow-up evaluation was conducted by telephone with a representative sample to gauge how participants were actually using the ArcView software and were there any changes in how they were using the information they learned. The results of this evaluation will be presented along with how the evaluation has modified the format of future ArcView workshops to emphasize areas of greater interest.
|
| | | |
|
Session Title: Qualitative Analyses of Education Reform Efforts
|
|
Multipaper Session 778 to be held in Carroll Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Qualitative Methods TIG
|
| Chair(s): |
| Eric Barela,
Los Angeles Unified School District,
eric.barela@lausd.net
|
| Discussant(s): |
| Bill Thornton,
University of Nevada, Reno,
thorbill@unr.edu
|
|
The Power of Q-methodology: Getting Beyond Perceptions to Uncovering Actions (or Inactions)
|
| Presenter(s):
|
| Matthew Militello,
University of Massachusetts, Amherst,
mattm@educ.umass.edu
|
| Sharon Rallis,
University of Massachusetts, Amherst,
sharonr@educ.umass.edu
|
| Abstract:
This paper explores the use of Q-methodology as a qualitative method for differentiating what participants in an alternative, district-based principal certification program express as their beliefs and demonstrate in their practice. This program trains aspiring principals to become instructional leaders who facilitate their teachers' practices for improved student learning. As evaluators we first looked at the impact of the training on participants' beliefs and understandings about instructional leadership through interviews and surveys. Next, we had to capture whether or not these new understandings changed participants' actions in practice. Resources, however, were not sufficient to allow time for shadowing principals to see how they actualized their learning in practice. We gathered all graduates to date and administered a Q-sort with statements drawn from the literature on effective leadership for learning practices. The paper will describe our methods and reveal our findings.
|
|
Developing Protocols for Qualitative Analysis of Education Sector Reform: Possibilities and Limitations of Research Protocols for Identifying and Informing on Non-linear, Iterative, and Generative Characteristics of Reform Processes
|
| Presenter(s):
|
| David Balwanz,
Academy for Educational Development,
dbalwanz@aed.org
|
| Jessica Jester Quijada,
Academy for Educational Development,
jquijada@aed.org
|
| Abstract:
The EQUIP2 team at the Academy for Educational Development is conducting case study analyses of Education Sector Reform in several different countries. Some of the reforms include: decentralization reforms in Zambia; adaptation of innovative and complementary school models in Guatemala; and school-based reforms in Namibia. The cases will outline the genesis, manifestation and development of specific reforms over a given period of time using an analytical framework highlighting specific factors influencing reform, including: Politics/Leadership; Finance/Resources; Institutional Framework; Institutional Capacity; Civil Society/Participation; and Driving Forces.
The research, in part, seeks to identify non-linear, iterative and generative characteristics of the reform process in each case and link that understanding to broader themes of system change, education quality improvement and roles of international donors.
This paper will outline the development and implementation of research protocols seeking to capture such characteristics and offer a discussion on some of the possibilities and limitations of such protocols in capturing information from the multiple contexts in which education reform takes place.
|
|
Success Case Study: A Retroactive Look at Impact Using the Success Case Method
|
| Presenter(s):
|
| Ravneet Tiwana,
University of California, Los Angeles,
rtiwana@ucla.edu
|
| Abstract:
The Success Case Method, (Brinkerhoff, 2003), was used to qualitatively evaluate how the interventions of professional development and mentorship impacted the processes of successful teaching practices in the classroom. By using the program logic model to determine indicators of success, professional development and classroom observations as well as interviews were used to retroactively look at impact of an intervention. During a four-year evaluation descriptive and outcome data had been collected without examining how the intervention had successfully impacted social processes that led to outcomes. Hence, the Success Case Method was used to determine how the program's contextual factors were influenced by the professional development and mentorship, which lead to a more in-depth, focused, and complex understanding of quantitative outcome data.
|
| | |
|
Session Title: Expert Evaluation of Federal Agencies' Program Portfolios in the Program Assessment Rating Tool (PART) Environment
|
|
Panel Session 779 to be held in Pratt Room, Section A on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Government Evaluation TIG
|
| Chair(s): |
| Raymond Sinclair,
Centers for Disease Control and Prevention,
rsinclair@cdc.gov
|
| Abstract:
With the advent of PART, independent expert evaluation of federal programs has received more attention. OMB's guidance for PART reviews of federal programs requires regular assessment of program relevance, quality, and impact using expert evaluation. Expert review may be an appropriate and cost-effective evaluation method, especially for research programs that often have uncertain results. However to date, there has been little analysis of the use of independent evaluation for federal programs in the PART environment.
This panel session provides different perspectives on concurrent evaluations of a federal agency's research portfolio by the National Academies, a respected independent science advisory body. National Academies' evaluators were asked to assess both relevance and impact of the agency's programs. Up to 13 reviews are either underway or planned. The purpose of this panel is to better understand the strengths and weaknesses of independent evaluation methods, especially for research organizations subject to PART.
|
|
Expert Evaluation of the National Institute for Occupational Safety and Health's Portfolio of Research Programs in the Program Assessment Rating Tool (PART) Environment
|
| Raymond Sinclair,
Centers for Disease Control and Prevention,
rsinclair@cdc.gov
|
|
The National Institute for Occupational Safety and Health (NIOSH) is engaged in an evaluation of its research portfolio by the National Academies (NA). Multiple studies are completed (2), underway (7), or planned (4) for its workplace sector-focused research programs (e.g., agriculture, mining) and research programs that cross those sectors (e.g., traumatic injuries, respiratory diseases). The NA is recognized for high-quality, independent, expert studies of science questions and government programs. OMB's PART evaluation process requires agencies to use expert judgment review methods, yet there is much to be learned about the process and how to use such reviews to improve program management. This presentation reviews the NIOSH approach to securing useful expert evaluations by the NA, the overall review plan, the review process, results so far, communications with OMB and stakeholders, and the responses of the reviewed programs.
|
|
|
Assisting Federal Programs in Expert Evaluation: Tools and Processes
|
| Valerie Williams,
RAND Corporation,
valerie_williams@rand.org
|
|
The current emphasis on assessing program effectiveness has required many federal programs to re-think how to identify and measure program impact. This has often led to a shift from focusing on program outputs to program outcomes. Yet, for many federal programs there are few tools available to assist them in determining and demonstrating program outcomes. We report our experiences in developing a framework to assist NIOSH in its external review by the National Academies. The methods that we developed rely on the use of logic models to communicate how the program operates and identify its intended outcomes. We developed an outcome-based worksheet to categorize and trace the causal linkages from outcomes back to program activities. Finally, we have developed a structure for articulating the program impact through the use of narratives that build from the outcome worksheets. We discuss the development, implementation and some of the challenges associated with these three tools.
| |
|
The National Academies Review of the Programs of the National Institute for Occupational Safety and Health
|
| Evan Douple,
The National Academies,
edouple@nas.edu
|
|
The National Academies has a long tradition of providing independent expert advice to the nation on issues of science, technology, and medicine, including reviews of research programs. Credibility has been built on the process for the selection of committee members with emphasis on recruiting appropriate expertise (including experience in program evaluation) and avoidance of potential conflicts of interest, providing opportunities for input from affected parties and public participation in the evaluation process, ensuring an evidence-based nature in the deliberations, and a thorough review process prior to the release of the final assessment reports. In 2005, the Academies was asked by NIOSH to evaluate the impact and relevance of up to 15 research programs at NIOSH. To begin that process, the Academies appointed a framework committee that set parameters and offered guidance for the evaluations in order to promote efficiency and consistency of methods and standards across the 15 individual studies. The resulting framework document set the template for the research program evaluations, two of which have been completed and a number of others are underway. The experiences of the framework and evaluation committees will be shared.
| |
|
The Use of Independent Evaluation in the Program Assessment Rating Tool Process (PART)
|
| Norris Cochran,
United States Department of Health and Human Services,
norris.cochran@hhs.gov
|
|
Independent program evaluations play a fundamental role in PART review. As an assessment tool, PART asks programs to provide evidence of effectiveness. Independent evaluations are an important source of evidence. In the Strategic Planning and Program Results and Accountability sections of PART, independent evaluations are explicitly requested to demonstrate that programs are (1) routinely evaluating their effectiveness and relevance to the problem, interest or need, and (2) operating as intended and achieving results. In this presentation, we present the perspectives of a former OMB staff member on the task of evaluating evidence. Special emphasis is placed on the contribution of independent evaluations as a source of evidence.
| |
|
Session Title: Connecting Evaluation Theory and Practice via Experiential Learning
|
|
Multipaper Session 780 to be held in Pratt Room, Section B on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Teaching of Evaluation TIG
|
| Chair(s): |
| Vanessa Dennen,
Florida State University,
vdennen@fsu.edu
|
|
Learning Evaluation and Applied Research Consultants
|
| Presenter(s):
|
| Gargi Bhattacharya,
Southern Illinois University, Carbondale,
gargi@siu.edu
|
| Kristin Pankey,
Southern Illinois University, Carbondale,
knpankey@siu.edu
|
| Joel Nadler,
Southern Illinois University, Carbondale,
jnadler@siu.edu
|
| Abstract:
The Applied Psychology Program at Southern Illinois University at Carbondale (SIUC) has been providing training in program evaluation since 1981. Perhaps the most unique part of this training is the opportunity to conduct actual professional evaluation as a part of Applied Research Consultants or ARC, a student managed consulting firm. Based on extensive qualitative research (structured interviews of both senior graduate students and alumni), this presentation discusses some of the major evaluation projects conducted by senior graduate students in ARC. The challenges they faced, and the strategies they used to overcome these problems. It also highlights some of the important skills and strategies in conducting evaluation learned in ARC, considered as invaluable skills by the alumni of this program who are conducting professional evaluation projects as the part of their post-academic careers.
|
|
Extending the Teaching of Evaluation Through Experiential Learning: Connecting Evaluation Theory and Planning With Grant-Making
|
| Presenter(s):
|
| Melissa Kelly,
University of Illinois, Chicago,
mkelly27@uic.edu
|
| Dan Kaczynski,
University of West Florida,
dkaczyns@uwf.edu
|
| Abstract:
This study, which presents an approach for educating future evaluators, connects evaluation theory and practice, grant-making, and experiential learning. We position this discussion upon the conceptual framework of a newly developed graduate-level evaluation course: Advanced Program Development and Evaluation. The course offers a promising means for applying experiential learning to evaluation education and is designed to encourage graduate students to critically examine and discuss current and emerging variations in theoretical orientations of evaluation. Students analyze these theoretical relationships through an applied research perspective involving the development of an actual grant proposal. This type of approach to evaluation education serves multiple functions by (a) deepening students' knowledge of evaluation issues, (b) fostering critical examination of diverse evaluation orientations, (c) illuminating the relationships between evaluation and organizational development, and (d) actively engaging students in developing and submitting a grant proposal. It also promotes civic engagement as students confront current, meaningful social issues.
|
|
The Evaluation Internship: Engaging Undergraduate Students in Applied Learning Opportunities
|
| Presenter(s):
|
| Kristi Lekies,
The Ohio State University,
lekies.1@osu.edu
|
| Abstract:
Throughout their undergraduate careers, students are exposed to a knowledge base that has resulted from the findings of research and evaluation studies. Many are eager to learn first-hand how knowledge is generated, apply the knowledge they've learned, and see for themselves how programs and policies, such as educational programs for children, are effective. Evaluation internships, either for pay or academic credit, can provide a way for students to learn more about evaluation process and design under the direction of an experienced mentor. This presentation will give an overview of a model used to engage over 50 undergraduates in research and evaluation opportunities. Benefits include building evaluation skills and knowledge, learning more about the program or subject matter being investigated, engaging with schools and community organizations, developing a relationship with faculty, opportunities to pursue independent work, and presenting findings through presentations and publications. Effective strategies will be discussed.
|
| | |
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Evaluation of an In-class Versus an Online Alcohol Diversion Program to Reduce Alcohol Consumption and Negative Consequences Among College Students |
|
Roundtable Presentation 781 to be held in Douglas Boardroom on Saturday, November 10, 12:10 PM to 1:40 PM
|
| Presenter(s):
|
| Peggy Glider,
University of Arizona,
glider@health.arizona.edu
|
| David Salafsky,
University of Arizona,
salafsky@health.arizona.edu
|
| Carlos Moll,
University of Arizona,
cmoll@email.arizona.edu
|
| Abstract:
This program evaluated an in-class versus online alcohol education course designed to: a) decrease alcohol use and negative consequences; b) increase student's stage of change; c) change attitudes regarding alcohol and d) increase accuracy of perceptions of AOD use norms. Students were randomly assigned to type of course following an alcohol infraction. The in-class program consisted of three 2-hour sessions over consecutive weeks. Students in the online course completed Third Millennium's “Under the Influence”, a self-paced, 3-hour alcohol program. Students completed baseline and 3-month post surveys that measured a range of alcohol indicators including consumption, negative consequences, perceptions and attitudes. To date, 396 matched (baseline and follow-up) cases have been analyzed. Preliminary results have shown consistent decreases in several alcohol consumption measures including drinks per week, drinks when partying and nights per week partying from both in-class and online methods, suggesting that both interventions are effective for this population.
|
| Roundtable Rotation II:
Pearls and Pitfalls in Evaluating Student Assistance Programs: A Seven-Year Longitudinal Study |
|
Roundtable Presentation 781 to be held in Douglas Boardroom on Saturday, November 10, 12:10 PM to 1:40 PM
|
| Presenter(s):
|
| Sharon Wilburn,
University of North Florida,
swilburn@unf.edu
|
| Kenneth Wilburn,
University of North Florda,
kwilburn@unf.edu
|
| Dax Weaver,
Health-Tech Consultants Inc,
htci@comcast.net
|
| Kathy Bowles,
Duval County Public Schools,
bowlesk@educationcentral.org
|
| Abstract:
This study is a longitudinal evaluation of the ZIP student assistance program including challenges to conducting evaluation in school-based programs. Evaluation questions were posed to determine program impact on substance use, behavior, academic success, and internal control orientation. During the study period, 9,978 students at 12 middle and 10 high schools participated in some phase of the program. Results indicate that drug use declined and significant improvement in internal locus of control occurred each year; average number of discipline referrals was less than the year prior to program admission; and there was no significant change in academic performance. Levels of significance for amount of time in program, referral types, intervention category, school level, and participant characteristics are also addressed. Evaluation pitfalls and recommendations for continued and improved evaluation practice with school-based prevention programs are discussed.
|
|
Session Title: The Illogic of Privileging Western Mainstream Ways of Knowing and Evaluation Practice in Indigenous and Other Non-western or Non-mainstream Contexts
|
|
Panel Session 782 to be held in Hopkins Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Indigenous Peoples in Evaluation TIG
|
| Chair(s): |
| Morris Lai,
University of Hawaii,
lai@hawaii.edu
|
| Discussant(s):
|
| Nan Wehipeihana,
Research Evaluation Consultancy Ltd,
nanw@clear.net.nz
|
| Abstract:
Since the latter part of the 18th century, Westerners have foisted much upon the indigenous peoples of Hawai'i and Aotearoa and later upon other non-mainstream groups. That foisting and colonization were based on the assumption that Western ways were an improvement on our own. The role of research and evaluation in this process is less explicit than the guns and religion also used. The privileging of Western ways of knowing within research, evaluation, and pedagogy is equally illogical. This privileging denies the knowing indigenous and other minority communities have. We illustrate this point with examples of community storytelling about historical events, educational initiatives, and evaluation strategies that meet the demands for Western-preferred evidence. Evaluators then become caught in the middle as translators who must attempt to bridge this knowledge divide. We have learned to use all the relevant available tools such as theory, ethical protocols, and community action.
|
|
Dastardly Deeds and Words Should Lead to Loss of Privileges
|
| Morris Lai,
University of Hawaii,
lai@hawaii.edu
|
|
Much land and other property belonging to the Hawaiian and Maori people were illegally (even by Western standards) taken by Western colonizers near the end of the 18th century. Such dastardly deeds continued throughout the 19th and 20th centuries, and many other acts of cruel prejudice against non-Westerners are occurring at the present time. Among the perpetrators of horrendous behavior against indigenous and other minority peoples are U.S. presidents, a U.S. Supreme Court Chief Justice, presidents of Harvard University, a renowned historian, a state governor, Christian ministers who led indigenous churches, law makers in many states, a prominent general in the U.S. Army, and a national museum. It would be foolish, not proper, and illogical for any group of people, grossly mistreated by the aforementioned Western/mainstream 'experts' and prominent leaders, to turn around and then privilege Western/mainstream approaches to matters such as evaluation, pedagogy, and ways of knowing.
|
|
|
Managing for Maori Outcomes
|
| Fiona Cram,
Katoa Ltd,
fionac@katoa.net.nz
|
|
Government departments in Aotearoa New Zealand are moving toward a Managing for Outcomes' (MfO) environment in which evaluative activity is seen as essential to a department's ability to learn from successes and failures within an environment more focused on performance and outcomes. These moves are impacting on departments' contracted providers of services and programmes, including Maori (tribal) Provider Organizations (MPOs). The potential for MfO to re-colonize MPOs cannot be ignored, especially if relationships are not present between departments and contracted MPOs and/or mainstream values dominate legitimated departmental outcomes. The application of learnings from indigenous evaluations (with, for, and/or by MPOs; often using qualitative methods) can forestall this re-colonization by privileging indigenous values and successful models of practice. Challenges to this happening include demands for 'hard' evidence, unrealistic expectations, and racism. This paper examines evaluation lessons that have been learned and asks whether anyone is open to them.
| |
|
Kumu o ka 'Aina (Teachers of Our Land): Home Grown New Teacher Development
|
| Alice Kawakami,
University of Hawaii,
alicek@hawaii.edu
|
|
Western educational settings and teacher education programs are dominated by accountability mandated by the national accreditation agencies, teacher licensing boards, and student content and performance standards. These standards define parameters for teacher education programs, new teacher licensing, and K-12 student achievement as defined by knowledge, skills, and dispositions. Thus, mainstream views are privileged even in learning communities with multicultural and indigenous student populations. The Ka Lama o ke Kaiaulu Elementary Teacher Education Cohort and the Ka Lama Education Academy empower the indigenous community through recruitment of pre-service teacher candidates, school community partnerships and curriculum integration of guidelines for Culturally Healthy and Responsive Learning Environments (CHARLE) also known as Na Honua Mauli Ola Hawaii. This presentation will describe work that elevates the indigenous community within the process of new teacher development through recruitment, field experience partnerships, and teacher candidate values clarification. It will also identify future directions for program development.
| |
|
Assets Based Inquiry: Culling Tenets of Success From Promising Practices in Hawaiian Education
|
| Kanani Aton,
Kapuahi,
k-aton@hawaii.rr.com
|
|
In Hawaii's public school population, 26% are Native Hawaiians. Statewide, Hawaiian education community stakeholders and the State Department of Education (DOE) collaboratively design specific strategies for improving the quality of their learning. This initiative recently pulled together successful characteristics, or Tenets of Success of Hawaiian educational promising practices from the community using Assets Based Inquiry (adapted from Appreciative Inquiry), a practice that identifies what is working for broader replication, rather than focusing on fixing what doesn't work. The Tenets of Success underpin promising Hawaiian education practices and are an effective starting point for planning/piloting in the DOE system. This process also supported strong relationship building at the stakeholder base, another significant outcome that may help overcome the challenges of the DOE system's lukewarm response to the usefulness of the Tenets thus far as well as keep this largely volunteer effort sustained.
| |
|
Session Title: Fun and Games With Logframes
|
|
Skill-Building Workshop 783 to be held in Peale Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Presenter(s):
|
| Scott Chaplowe,
American Red Cross,
schaplowe@amcrossasia.org
|
| Abstract:
In the international development community, logframes have become the industry standard to summarize a project/program's design and intended results. At best, logframes are tools that help project design, monitoring, and evaluation (DM&E). At worst they can straightjacket a project, imposing an outside, technocentric method that alienates rather than fosters local participation in project design, monitoring, and evaluation. This skills building workshop is aimed at helping to minimize the alienation of local partners in the logframe process. The workshop's purpose is threefold. First and foremost, it will introduce the experienced M&E practitioner or trainer to fun, innovative exercises (games) to reinforce local partner understanding of logframes. This is important not only for local ownership of a project, but for reliable reporting as local partners often gather monitoring data. Second, the workshop will reinforce an logframe understanding for those new to them. Third, the workshop will show how logframes can indeed be fun!
|
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Can Using Communities of Practice Explicate Informal Knowledge in Evaluation Practice? |
|
Roundtable Presentation 785 to be held in Jefferson Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
| Presenter(s):
|
| Shannon Coulter,
University of Tennessee, Knoxville,
scoulte1@utk.edu
|
| Abstract:
This paper raises the issue of the evaluation community's ability to regulate the informal knowledge of the field. With the increasing proliferation of evaluation manuals, texts, and information, the gap has widened between what one knows and understands conceptually and one's tacit, everyday knowing. Moreover, informal knowledge consists of information embedded in norms that are not easily identifiable, which makes the knowledge difficult to communicate to others; however, the evaluation community's ability to innovate depends on its level of informal knowledge, and its ability to make informal knowledge explicit. Specifically, this paper questions whether a community of practice organized around dialogue could balance the field's reliance on both formal and informal knowledge, and whether the field of evaluation would support a community approach to explicating informal knowledge.
|
| Roundtable Rotation II:
Empowering the Individual: Considerations for Evaluation in a Flat World |
|
Roundtable Presentation 785 to be held in Jefferson Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
| Presenter(s):
|
| Stacey Hoopes,
Brigham Young University,
staceylea@gmail.com
|
| David Williams,
Brigham Young University,
david_williams@byu.edu
|
| Abstract:
This roundtable session will focus on some of the ideas discovered in readings of Friedman's The World is Flat and Evaluation Practice Reconsidered, written by Schwandt. Topics discussed will relate to empowerment evaluation as it relates to individuals. Specifically, attention will be given to how individuals might be empowered in a technologically advanced, ever-changing world. This session will also consider the practical role of the evaluator in such a world, including how that might need to be changed as the flattening of the world continues and quickens over time.
|
|
Session Title: Evaluating Health Improvement Collaboratives Using a Systems Thinking Developmental Evaluation Approach
|
|
Multipaper Session 786 to be held in Washington Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Health Evaluation TIG
|
| Chair(s): |
| Tessie Catsambas,
EnCompass LLC,
tcatsambas@encompassworld.com
|
| Abstract:
The USAID-funded Quality Assurance Project that aims to develop and apply methods for improving healthcare quality in developing countries. The session shows how a combination of internal and external developmental evaluation helped to document the emerging model of rapid collaborative improvement of healthcare quality in developing countries. Developed by the Institute for Healthcare Improvement in the mid-90s, the improvement collaborative is a major new approach for rapidly improving the quality and efficiency of healthcare. By applying systems thinking, the evaluation tested program assumptions, definitions, and mental models to facilitate learning throughout the evaluation. A -lessons learned week- brought together managers of 17 collaboratives from Russia, Rwanda, Niger, Uganda, Tanzania, Eritrea, Benin, Honduras, Nicaragua, Ecuador, and South Africa. Joint client/evaluator teams collected data in 6 countries where they visited health sites, government, and other stakeholders. The evaluation report explored the added value of collaboratives, factors for collaborative readiness and sustainability.
|
|
Applying the Collaboratives Model to Developing Country Settings
|
| Jorge Hermida,
University Research Company LLC,
jhermida@urc-chs.com
|
| Mary Gutmann,
EnCompass LLC,
mgutmann@encompassworld.com
|
|
The improvement collaborative, a major new approach for rapidly improving the quality and efficiency of healthcare, was developed by the Institute for Healthcare Improvement in the mid-90s. This model underwent its first adaptation by QAP when before it was introduced to developing country settings. Up to that point, collaboratives had been applied only in wealthier US and European countries with decentralized and privatized healthcare systems. By contrast, in developing countries, collaboratives were set up in the context of resource poor, centralized healthcare systems. As the project evolved, major lessons emerged for managers of individual collaboratives resulting in further adaptations of the model. Challenges, lessons and good practices were shared among collaborative managers resulting in further adaptations still. Key concepts include: demonstration and spread collaboratives, norms and standards, change package, learning sessions, measuring and reporting, coaching, PDSA cycles, and continuous quality improvement.
|
|
Investing in Collaboratives
|
| Lynne Franco,
University Research Company, LLC,
lfranco@urc-chs.com
|
|
Strategically, the United States Agency for International Development decided to invest in collaboratives based on the work of the Quality Assurance Project over a 10-year period. Quality improvements that focused on creating specific healthcare improvements were unable to help the world tackle larger more universal and acute problems related to maternal and child health (HIV, TB, Malaria, and others). USAID made a strategic decision to support approaches such as collaboratives that promised rapid, evidence-based quality improvement. Lessons from the work of the Quality Assurance Project and this evaluation have important implications for developing country healthcare improvement.
|
|
Evaluating Collaboratives
|
| Tessie Catsambas,
EnCompass LLC,
tcatsambas@encompassworld.com
|
|
An outside evaluation team was engaged to implement the evaluation of collaboratives early on. The evaluation team worked alongside implementers helping to document the emerging model, insights from implementation, changes in definitions and approaches, challenges, and breakthroughs. Many things evolved during project implementation e.g. the use of experts for agreeing on quality standards, coaching, spread strategies, nature of team-based improvement. System thinking was used to test assumptions with individual collaboratives managers and the group as a whole. In a lessons learned week, the evaluation team facilitated -depth conversations on achievements, challenges and lessons. To ensure that learning occurred within the project, evaluation team members paired up with collaboratives managers to conduct field visits. Collaborative managers visited countries where there were not working for independence, deepening collaboration and learning among project staff. The resulting evaluation report explored questions related to the added value of collaboratives, factors for collaborative readiness, and sustainability.
|
|
Session Title: Multiple Levels of Evaluation: Organizational Learning in a Foundation-Supported Initiative
|
|
Panel Session 787 to be held in D'Alesandro Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Chair(s): |
| Andrea Berger,
American Institutes for Research,
aberger@air.org
|
| Discussant(s):
|
| Victor Kuo,
Bill & Melinda Gates Foundation,
victor.kuo@gatesfoundation.org
|
| Abstract:
How can a foundation encourage participant learning among school sites, foundation grantees, and external evaluators in one large initiative? This panel will discuss a nationwide initiative's unique organization for using data and evaluation findings. A lead evaluator will describe how multiple levels of evaluation activities, including monitoring, formative, and summative evaluation efforts, work together to meet different audience needs. A grantee representative will discuss how school data are used for accountability purposes within a network and for sharing with the foundation. A representative from the initiative's coordinating organization will discuss its unique role in connecting everyone around data sharing and learning from the evaluation. Finally, a foundation representative will discuss how all of these organizations meet the foundation's grant oversight needs. She will also talk about this initiative's unique organization of participants, how that differs from other initiatives, and what the benefits and drawbacks have been.
|
|
Collaboration Within a National Evaluation
|
| Andrea Berger,
American Institutes for Research,
aberger@air.org
|
|
Two organizations, the American Institutes for Research (AIR) and SRI International (SRI), are partnering to conduct the ECHSI national evaluation. Andrea Berger has been the AIR evaluation director since its inception in 2002. She will discuss how collaborations with ECHSI participants have informed the evaluation. First, the evaluation team collaborates with JFF on two data collection activities: a school survey and the SIS, which houses student-level data. The evaluation team also formally and informally discusses the status of the ECHSs and the initiative overall with JFF. This close collaboration eases participants' reporting burden and keeps the evaluation team informed about the activities and challenges that are occurring. Annually, the evaluation team works with the foundation to reflect on data collection and reporting activities and develop approaches to keep the evaluation integral to the ECHSI.
|
|
|
Participating in and Learning From Evaluation
|
| Linda Campbell,
Center for Native American Educational Advancement,
lcampbell@antiochsea.edu
|
|
Linda Campbell has overseen this grantee's work since the first round of ECHSI grants were made in 2002. Having been involved with this initiative since its inception, she has a perspective on how the evaluation work and interaction between the participants has grown and changed over the years. She will discuss what information needs are met by the national evaluation and by the work done by JFF. In particular, she will discuss her organization's involvement in the development of the SIS and in the accountability activities facilities by JFF (e.g., annual reviews with the foundation). She will also discuss her own organization's data collection and reporting from the perspective of working with rural and under-funded schools and districts, and with Native American students. Finally, Dr. Campbell will discuss the benefits and drawbacks of these multiple levels of collaboration.
| |
|
Connecting all of the Evaluation Pieces to Facilitate Initiative Learning
|
| Leslie Haynes,
Jobs for the Future,
lhaynes@jff.org
|
|
Leslie Haynes will discuss the many ways that JFF supports ECHSI participant information sharing and collaboration. Most of these supports stem from JFF's role as technical assistance provider. This role has evolved since 2002. JFF facilitates all of the annual reviews between the grantees and the foundation. These reviews represent an opportunity for foundation staff to learn about each intermediary's progress, collectively reflect on the work, and strategize about future plans. JFF also oversees the SIS and other initiative-wide data collection activities. These activities contribute to JFF's understanding of the overall initiative and its participants, placing JFF in a strong position to provide technical assistance and coordinate peer learning. JFF has also used its role as a technical assistance provider to support the activities of the national evaluation and to build learning opportunities around findings from the national evaluation.
| |
|
Facilitating Connections Across the Initiative to Meet Funder Information Needs
|
| Yee-Ann Cho,
Bill & Melinda Gates Foundation,
yee-ann.cho@gatesfoundation.org
|
|
Yee-Ann Cho is the foundation program officer overseeing the ECHSI. She will present her perspective as a consumer of the multiple levels of evaluation work. She will discuss how the foundation's evaluation efforts have evolved to include monitoring and formative and summative evaluation activities, and how the ECHSI evaluation exemplifies this multi-pronged approach. She will also discuss how these efforts ultimately meet the needs of the grantees, JFF, the foundation, and other audiences. The foundation supports the use of evaluation by grantees and JFF for learning and the annual review of grantee progress. Ms. Cho will also describe how the evaluation addresses learning goals articulated by the foundation's leadership. Finally, Ms. Cho will discuss the benefits of designing evaluations and program initiatives in tandem and will offer suggestions for how program officers and evaluators can work together to maximize evaluation use and learning.
| |
|
Session Title: Longitudinal/Growth Curve Analysis of Program Impacts
|
|
Multipaper Session 788 to be held in Calhoun Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Lihshing Wang,
University of Cincinnati,
leigh.wang@uc.edu
|
|
The Long Road of Longitudinal Studies: Learning What to Do and Not to Do Along the Way
|
| Presenter(s):
|
| Bruce Yelton,
Praxis Research Inc,
praxis1@att.net
|
| Paula Plonski,
Praxis Research Inc,
pmplonski@carolina.rr.com
|
| Grant Morgan,
Praxis Research Inc,
praxisgm@aol.com
|
| Mary Beth Gilbert,
Praxis Research Inc,
marybethgilbert@bellsouth.net
|
| Abstract:
The use of longitudinal studies to demonstrate program effects over time presents evaluators with challenges that are immediately obvious and frustratingly difficult to anticipate. Evaluators considering the use of longitudinal studies should consider both the theoretical implications of implementing a longitudinal study and the more frequently mentioned methodological barriers to this type of investigation. These and other issues are discussed in relation to four longitudinal studies of North Carolina's “SMART START” initiative to prepare young children for school. These studies were conducted between 1998 and the present and involved following young children from their childcare/preschool years into public school.
Themes for discussion in the presentation include: 1) Stakeholder Expectations, 2) Theory Driven Evaluation, 3) Data, Technology, and Access, and 4) Attrition. Results, examples, and practical solutions will be offered.
|
|
Using Cox Regression Modeling to Predict Recidivism for Youth Departing Out-of-home Care: Implications for Program Evaluation and Treatment of At-risk Youth
|
| Presenter(s):
|
| Jay Ringle,
Girls and Boys Town,
ringlej@girlsandboystown.org
|
| David Kingsley,
University of Kansas,
gridave@sunflower.com
|
| Stephanie Ingram,
Girls and Boys Town,
ingrams@girlsandboystown.org
|
| Beth Chmelka,
Girls and Boys Town,
chmelkab@girlsandboystown.org
|
| Ron Thompson,
Girls and Boys Town,
thompsonr@girlsandboystown.org
|
| Abstract:
Although high-quality treatment programs for troubled youth reduce the risk of serious life-problems, some youth will nevertheless experience post-treatment difficulties. Arrest for illegal activity is one common risk factor addressed in most youth treatment programs. This study examined risk factors associated with youth being arrested up to five years post-discharge from an out-of-home residential treatment setting. Using a Cox Regression Proportional Hazards modeling procedure, results indicate two protective factors against future recidivism: an absence of a criminal background at admission and a positive departure from care. Using this modeling procedure, organizations serving at-risk youth can use data they are already collecting to evaluate outcomes and identify risk factors associated with those youth most likely to re-offend in order to modify and improve their treatment appropriately.
|
|
Evaluating Impacts of Place-based Initiatives: An Application of a Spatially Improved Interrupted Time Series Design
|
| Presenter(s):
|
| Julia Koschinsky,
University of Illinois at Urbana-Champaign,
koschins@uiuc.edu
|
| Abstract:
Evaluators frequently apply the interrupted time series (ITS) design with an added nonequivalent, no treatment control group time series to assess whether targeted neighborhood revitalization investments generate spillover effects to surrounding areas (Accordino et al. 2005; Galster et al. 2004). Although this literature begins to incorporate advances in spatial analysis/econometrics, no comprehensive applications exist to date.
To address this research gap, the purpose of this paper is to increase the “spatial intelligence” of the best current adjusted ITS models. Using spillover effects from low-income rental housing in Seattle, WA as an example, this is done by 1) applying exploratory spatial data analysis techniques, including geographically weighted regression (with aerial images); 2) by modeling the spatial segmentation of housing markets; 3) by testing for the presence of spatial autocorrelation; and 4) by comparing the performance of the adjusted ITS models to state-of-the-art spatial regression models (Anselin 1988).
|
|
Evaluating Value-added Methodology for Standards-based Accountability Assessment
|
| Presenter(s):
|
| Lihshing Wang,
University of Cincinnati,
leigh.wang@uc.edu
|
| Kent Seidel,
University of Cincinnati,
kent.seidel@uc.edu
|
| Suzane Franco,
Wright State University,
suzanne.franco@wright.edu
|
| Abstract:
As stakeholders continue to push for longitudinal assessment of learning growth under the No Child Left Behind Act, value-added methodology (VAM) emerges as a promising tool for evaluating teacher quality and school effectiveness. This study critically examines the epidemiological basis of value-added scores and their psychometric sensitivity to demographic confounding and scaling operation. Using recent statewide assessment data to substantiate our claims, we found that VAM is both epidemiologically inconsistent with the standards-based accountability ideology and psychometrically unstable for assessing teacher/school effects. Furthermore, the value-added scores were found to be divergent from the Adequate Yearly Progress index currently in place for measuring school performance. We conclude by issuing cautions on applying and interpreting VAM for standards-based accountability assessment and proposing an alternative conceptual framework for measuring student growth that combines the merits of VAM and other longitudinal models.
|
| | | |
|
Session Title: Consortium-wide Evaluation of a Minority Science, Technology, Engineering, and Mathematics Retention Project
|
|
Panel Session 789 to be held in McKeldon Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Multiethnic Issues in Evaluation TIG
and the Assessment in Higher Education TIG
|
| Chair(s): |
| Jing Zhu,
The Ohio State University,
zhu.119@osu.edu
|
| Abstract:
The content of this session deals with evaluation and research conducted under the aegis of the Ohio Science and Engineering Alliance (OSEA), a National Science Foundation-sponsored consortium of universities. The purpose of OSEA is to improve minority retention in science, technology, engineering, and mathematics (STEM) fields. The challenges and strategies of conducting a consortium evaluation are discussed followed by summary of multiple years of evaluation results. Then there will be two short presentations on current research projects now underway in the OSEA context. Four brief presentations of approximately 12 minutes each will comprise the main part of the panel with ample time allotted for discussion and participation of those in attendance.
|
|
Problems and Issues in Conceptualization and Implementation
|
| Jeffry L White,
Ashland University,
jwhite7@ashland.edu
|
| James W Altschuld,
The Ohio State University,
altschuld.1@osu.edu
|
| Yi-Fang Lee,
National Chi Nan University,
ivanalee@ncnu.edu.tw
|
|
Increases in minority enrollment in science, technology, engineering, and mathematics (STEM) have not produced more degrees for African-Americans, Hispanics, and Native-Americans. The attrition rates for these groups are significantly higher than those of White or Asian college students. In response, the National Science Foundation Louis Stokes Alliances for Minority Participation has formed partnerships with universities to develop programs aimed at addressing the issue. In this presentation, problems and issues in conceptualizing and implementing the evaluation of a major retention project are presented. Recommendations and strategies for others interested in the evaluating programs designed to improve the retention rates of minorities in the STEM fields will additionally be offered.
|
|
|
Comparisons of Multiple Years of Data
|
| James W Altschuld,
The Ohio State University,
altschuld.1@osu.edu
|
| Deborah H Kwon,
The Ohio State University,
kwon.59@osu.edu
|
| Jing Zhu,
The Ohio State University,
zhu.119@osu.edu
|
| Jeffry L White,
Ashland University,
jwhite7@ashland.edu
|
| Yi-Fang Lee,
National Chi Nan University,
ivanalee@ncnu.edu.tw
|
|
The Ohio Science and Engineering Alliance (OSEA) coordinates and provides several unique activities designed to enhance the experience of underrepresented minorities in STEM disciplines. OSEA evaluators have developed and administered surveys in 2005 and 2007 to groups of students and faculty involved in the statewide effort. Results from the surveys will be compared across the time periods and groups surveyed in terms of perceptions of services provided and satisfaction with same. Implications of the findings for future endeavors in this area as inferred from survey data will also be given.
| |
|
Capturing the Impact of Academic Self-efficacy and Career Perceptions
|
| Deborah H Kwon,
The Ohio State University,
kwon.59@osu.edu
|
| James W Altschuld,
The Ohio State University,
altschuld.1@osu.edu
|
| Jing Zhu,
The Ohio State University,
zhu.119@osu.edu
|
| Jeffry L White,
Ashland University,
jwhite7@ashland.edu
|
|
In response to the low level of minorities in the (STEM) workforce, the United States has focused on improving minority participation in STEM in universities. How do academic learning and self-efficacy translate into careers for minority undergraduate students? How should these constructs be measured? What are the components of these variables? These questions are at the core of this research study and they along with the structure of the study will be described. If preliminary results are available, they will also be discussed.
| |
|
The Effects of Interaction on Decisions to Persist
|
| Jing Zhu,
The Ohio State University,
zhu.119@osu.edu
|
| James W Altschuld,
The Ohio State University,
altschuld.1@osu.edu
|
| Deborah H Kwon,
The Ohio State University,
kwon.59@osu.edu
|
| Jeffry L White,
Ashland University,
jwhite7@ashland.edu
|
|
Aligned with the other research study just discussed, a crucial question is how to motivate minority students to stay in STEM disciplines. Here the focus will be on student interaction as a factor important for retention. Most retention models contain an interactive component. There is an abundance of literature supporting the notion that the frequency and quality of student interaction with faculty and peers influence the intention to persist. This study characterizes the interaction construct in the STEM environment based on minority students' perceptions of their interaction with STEM faculty, peers, and non-minority student groups, as well as of faculty interactions with both minority and non-minority students. The effects of interaction are explored in relation to persisting, being at risk of not persisting, or leaving a STEM field.
| |
|
Session Title: Advancing Personnel Evaluation in Business & Industry
|
|
Multipaper Session 792 to be held in Calvert Ballroom Salon B on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Business and Industry TIG
|
| Chair(s): |
| Liliana Rodriguez-Campos,
University of South Florida,
lrodriguez@coedu.usf.edu
|
| Abstract:
As a major category within the -Big 6- evaluation fields, personnel evaluation has a clear role to play within the evaluation discipline. However, evaluation and social science scholars have given relatively little attention to this specialty, leaving practitioners to wrestle with issues related to personnel selection and evaluation. In this multi-paper session, three aspects of personnel evaluation are explored. The first considers personnel selection systems and how using a well-designed and properly operated selection system can impact employee turnover and organizational performance. The second paper considers the implementation and evaluation of the five-step career planning process used in professional human resource management. The third paper addresses the topic of evaluating employee performance and presents a metaevaluation of a personnel evaluation system.
|
|
Getting to Good: Evaluating the Impact of Personnel Selection Systems
|
| Wes Martz,
Western Michigan University,
wes.martz@wmich.edu
|
|
The reliability and validity of personnel selection approaches have been a topic of interest for practitioners and scholars alike. Research on recruitment, selection interviews, assessment tests, criterion measures, and a variety of subtopics fills the personnel and industrial psychology literature. However, less attention has been given to evaluating the essential components of a selection system and the selection system's impact on organizational performance. This paper explores the evaluation of personnel selection systems, the criteria that defines a 'good' system, and the impacts on organizational performance. A case study is presented to illustrate the practical elements associated with evaluating a personnel selection system and its outcomes.
|
|
Evaluating the Effectiveness of the 5-step Career Planning Process
|
| Willis Thomas,
Western Michigan University,
willis.h.thomas@wmich.edu
|
|
Career planning is made up primarily of five action steps: performance planning, development planning, coaching, compensation discussion, and evaluation. Performance planning concerns the roles and responsibilities of the employee, and integrates job training needs to meet job requirements. Development planning encompasses the professional aspirations of the employee. It extends beyond required training, and involves learning opportunities designed to help the employee exceed job requirements. Coaching is an iterative process that reinforces job performance and encourages professional development. Compensation discussion is the financial reward that the employee realizes for their contribution. Evaluation is the last step in the process that assesses the effectiveness and efficiency of the prior steps. This paper will discuss strategies for effective implementation of this 5-step HR process.
|
|
Splitting the Atom: A Metaevaluation of a Nuclear Power Plant's Personnel Evaluation System
|
| Otto Gustafson,
Western Michigan University,
ottonuke@yahoo.com
|
|
Employee performance management processes seek to link an organization's key performance indicators with its personnel evaluation system. This association is intended to clearly communicate business goals and personnel expectations, as well as tie overall business unit, department and individual performance to merit compensation increases. A metaevaluation of one such system was recently conducted for a Midwestern Nuclear Power Plant with 600 employees. This paper will provide an overview of the personnel evaluation system, a discussion of the metaevaluation methodology and results, and suggestions for future improvements to the employee performance management process.
|
|
Session Title: Learning From Quality Assurance and Improvement Processes
|
|
Panel Session 794 to be held in Calvert Ballroom Salon E on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Government Evaluation TIG
|
| Chair(s): |
| Rakesh Mohan,
Idaho State Legislature,
rmohan@ope.idaho.gov
|
| Discussant(s):
|
| David J Bernstein,
Westat,
davidbernstein@westat.com
|
| Abstract:
Quality assurance and improvement processes can serve as a learning tool for evaluators and others. Drawing from private, non-profit, and government sectors, panel members will discuss different approaches to quality assurance and improvement and how we can learn from those processes to improve quality of our work in the future. These approaches vary widely depending on the nature of business one is involved with and the organizational context. However, for these processes to serve as an effective learning tool, we must clearly define their purpose.
|
|
Quality Assurance and Quality Improvement Fosters Quality Management
|
| Stanley Capela,
HeartShare Human Services,
stan.capela@heartshare.org
|
|
This presentation will focus on how one organization utilizes external reviews conducted by a state funding agency as part of their quality improvement process that leads to enhancing quality management and thereby organizational learning.
HeartShare Human Services is a non-profit organization that provides children and family and developmental disabilities services. Unlike children and family services where the emphasis is on performance measurements, the developmental disabilities service component focuses on quality assurance. This presentation will focus on how one organization uses information from state funding reviews to identify programmatic strengths and weaknesses through an analysis of statements of deficiencies. Based on information from these reviews, the organization utilizes a quality improvement committee to analyze data, target program areas and develop systems for improving performance.
Overall, this process has led to a significant reduction in programmatic deficiencies and has created a quality improvement process that fosters an environment of organizational learning.
|
|
|
Rabbit Season? Duck Season? Proposal Season!
|
| Jennifer Dewey,
Macro International Inc,
jennifer.d.dewey@orcmacro.com
|
|
This presentation will describe how a for-profit organization organizes proposal production to enhance cooperation and channel resources among staff in order to provide coordinated, methodologically sound, cost-effective, and utilization-focused evaluations in the public health, mental health, and prevention fields. Macro International Inc.'s Center for Evaluation Excellence (CEE) assists staff in:
1) capitalizing on and channeling staff's evaluation strengths and skills during development of evaluation proposals, including proposed staffing for evaluation projects;
2) providing a forum for discussions about evaluation across program areas by way of special meetings, special guests, and evaluation-focused staff training, which is then incorporated into proposals; and,
3) organizing and articulating our collective understanding of, and our range of approaches and skills for, conducting health promotion and health care/services evaluation.
Macro's processes, including the CEE, contribute to an organized, coordinated system for proposal development that fosters cross-team collaboration, organizational learning, and a track record of success.
| |
|
Ensuring Quality of Evaluations on a Shoestring Budget
|
| Rakesh Mohan,
Idaho State Legislature,
rmohan@ope.idaho.gov
|
|
One good way to ensure quality of our evaluations is to learn from quality control/assurance (QC/QA) processes that we employ in conducting those evaluations. Depending on the size of an evaluation agency and available resources, these QC/QA processes can vary in scope-from using a simple checklist that shows policies and procedures were followed to doing a thorough assessment of methodologies used and analyses conducted. For those evaluation offices that operate on a shoestring budget, the challenge lies in designing a QC/QA system that not only helps improve the quality of evaluations but also allows evaluators to learn from it. If we learn from the QC/QA process, we are likely to not repeat the same mistakes and do a better job on the next evaluation. This presentation will discuss different QC/QA methods used by the evaluation office of the Idaho Legislature.
| |
|
Session Title: The Challenges and Opportunities of Evaluating Mathematics and Science Partnership (MSP) Projects: Michigan's Design, Strategies and Instruments
|
|
Panel Session 795 to be held in Fairmont Suite on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Shannan McNair,
Oakland University,
mcnair@oakland.edu
|
| Abstract:
The Michigan Department of Education (MDE) Mathematics and Science Partnership (MSP) programs are to strengthen K-12 mathematics and science education in high-need school districts. Each funded program includes an evaluation model that uses scientifically-based research methods to evaluate whether program objectives were attained. Each participant will present their evaluation design, strategies, instruments and findings. Dialogue about current evaluation expectations and the challenges this presents in school settings will be facilitated. The objective of each MSP project is to achieve a measurable increase in the academic performance among students instructed by teachers who participated in a MSP project. Evaluation findings from the first cohort and preliminary findings from the second cohort will be discussed.
|
|
Western Michigan University: Looking at the Michigan Mathematics Rural Initiative and Muskegon Middle School Mathematics Improvement Projects
|
| Sandy Madden,
Western Michigan University,
sandra.madden@wmich.edu
|
|
A major focus of three Michigan Mathematics and Science Partnership projects--two serving urban schools and the other rural schools--has been on developing mathematics content knowledge of middle and high school teachers to help them improve the teaching and learning of mathematics in their classrooms. An important element of the evaluation has been on gathering information about the impact of the program on teacher content knowledge through use of project-developed content tests, an existing measure of pedagogical content knowledge, teacher self-report, evaluator observations of lessons in classrooms of participating teachers, interviews, student surveys, and MEAP trend data. A discussion of the professional development interventions, a key component of which has been school-based learning communities, will be followed by presentation of data sharing effects on teacher content knowledge and student content knowledge. Analysis of the strengths and limitations of the various methodologies will be presented.
|
|
|
Oakland Schools Math Science Partnership Project
|
| Wendy Tackett,
iEval,
wendolyn@mac.com
|
| Valerie Mills,
Oakland Schools,
valerie.mills@oakland.k12.mi.us
|
|
Project Purpose: The purpose of MERC is to provide a comprehensive professional development experience for teachers and administrators focused on good instruction through mathematics. Teachers participate in curriculum-based workshops, content-focused classes, coaching in the classroom with expert teachers, and lesson-planning meetings. Principals participate in a course designed to improve their leadership skills while focusing on math content. This two-tiered approach helps create a collaborative school environment focused on the pursuit of academic success for students.
Methods of Evaluation: The evaluation is also a multi-tiered approach. Teacher pedagogical content knowledge is determined through the testing and retesting of teachers, use of instructional techniques by teachers is determined through classroom observations, perceptions of changes and impact are determined by teacher and administrator interviews, and student academic growth is determined through the state standardized test. Data is disaggregated using various variables including building climate, length of participation in the project, etc.
Key Findings: Teachers are creating lessons with more substantive student-student interaction, appropriate pacing for all student levels, and adequate reflection time. It was also determined that building climate may not have an impact on teacher content knowledge but it does have a significant impact on instructional practices and classroom management.
| |
|
Sustained Professional Development and Achievement: Washtenaw Intermediate School District
|
| Frederica Frost,
Wayne County Research Educational Service Agency,
frostf@resa.net
|
| Naomi Norman,
Washtenaw Intermediate School District,
nnorman@wash.k12.mi.us
|
|
Lesson Study was evaluated over a two-year period with the same participants each year. Only six teachers participated consistently, but results appear meaningful. The amount of participation time seemed to make a difference. Not all teachers were involved at the same level of effort; the number of hours of participation varied widely.
For the first year, the evaluation indicated some level of reluctance to repeat the experience, particularly by those who actually developed and taught the lesson. They seemed to feel that the amount of time they had to commit was excessive. Further, limited differences could be detected between the Lesson Study and No Lesson Study groups. The additional time spent did not appear to result in greater mathematical content learning or in greater knowledge of math pedagogy.
Following the second year of Lesson Study, participating teachers were compared to their non-Lesson Study colleagues on mathematical content knowledge, pedagogical knowledge, and classroom practice. The design replicated that of Elizabeth Fennema's group in Cognitively Guided Instruction, in which teachers were found to continue their growth during the year following their professional training. Although Lesson Study was the only professional development offered to the Washtenaw teachers that year, they continued to improve in both mathematical content and instructional practice. Their non-participating colleagues improved their practice as well, but performed less well overall in content knowledge.
| |
|
The Role of Statewide Evaluators for Math Science Partnership (MSP) Projects
|
| Dennis W Rudy,
Lakehouse Evaluation Inc,
drudy@lakehouse.org
|
| Shannan McNair,
Oakland University,
mcnair@oakland.edu
|
|
Each funded program includes an evaluation model that uses scientifically-based research methods to evaluate whether program objectives were attained. A team of evaluation consultants, contracted by MDE, works to provide technical support and consultation on evaluation to MDE/MSP staff and to individual projects. Their tasks include, but are not limited to, meeting with project staff in large groups, by cohort group and individually, collecting summary information on program implementation and evaluation, reviewing the evaluation section of all grant proposals and making recommendations specific to evaluation, reviewing calls for proposals, assisting with technical assistance on preparing proposals and suggesting criteria. In addition, the Michigan evaluation team conducts site visits of each program, attends the U.S. D.O.E. meetings, participate in the conference calls, and review yearly and final reports before they are submitted.
A particular aspect that these consultants have added to their list of tasks is to explore ways in which the question of impact of the partnership on STEM faculty can be identified and measured. This appears to be a question of importance nationally, but was not routinely included in MSP project evaluation plans. Questions raised regarding this aspect of MSP evaluation and others will be raised for discussion among all session participants. What is the role of Statewide Evaluators in other states? How is the relationship between the evaluators and the State Department of Education team negotiated? Does partnership at all levels emerge from such projects?
| |
|
Session Title: Assessment Issues in Education
|
|
Multipaper Session 796 to be held in Federal Hill Suite on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Zandra Gratz,
Kean University,
zgratz@aol.com
|
|
Proximal Assessment for Learner Diagnosis (PALD): A Dynamic Approach to Classroom Assessment aimed at Reducing Learner Gaps in Subject Area Domains
|
| Presenter(s):
|
| Madhabi Chatterji,
Teachers College Columbia University,
mb1434@columbia.edu
|
| Edmund W Gordon,
Teachers College Columbia University,
egordon@exchange.tc.edu
|
| Abstract:
The Proximal Assessment for Learner Diagnosis (PALD)is a dynamic approach to classroom assessment aimed at reducing learner gaps in subject area domains, starting from the individual learner upwards to classrooms and schools. Implemented by classroom teachers, the PALD approach involves cyclic use of frequent, learner-centered diagnostic assessments, coupled with error analysis, mediation and intensive practice for learners in specific domains. Research and development on the PALD model is presently underway with a focus on mathematics domains in 14 classrooms at four New York schools, with support from the National Science Foundation. This presentation will detail the theoretical bases of the PALD model, and show results from the first year's research on classroom implementation, focusing on processes and early outcomes.
|
|
Objective Criteria for Assessing the Validity of Traditional and Performance-based Classroom Assessment
|
| Presenter(s):
|
| Bruce B Frey,
University of Kansas,
bfrey@ku.edu
|
| Vicki L Schmitt,
University of Kansas,
vschmitt@ku.edu
|
| Abstract:
School-based evaluations typically include the evaluation of student academic outcomes. Valid measures of student learning and performance can provide critical evidence of program success and frequently appear on the “wish lists” of funders, stakeholders and evaluators. Theoretically, teacher-made classroom assessments provide a more valid alternative than standardized test scores. The problem for many evaluators, though, might be that while teacher-made tests might seem clearly to be the most proximal measure of a student's learning, how does one verify the validity of the assessment? This paper brings together the results of two studies which identified the technical criteria for quality when evaluating classroom assessments, whether they are traditional (e.g. multiple-choice) tests or performance-based assessments (e.g. portfolios). These criteria provide objective characteristics for judging the base validity of a classroom assessment.
|
|
Evaluation of the Elementary Grading and Reporting Policy
|
| Presenter(s):
|
| Kecia Addison-Scott,
Montgomery County Public Schools,
kecia_l_addison-scott@mcpsmd.org
|
| Abstract:
In an effort to ensure uniform and consistent grading practices within and across schools, the Montgomery County Public Schools (MCPS) has implemented a Grading and Reporting policy that is being piloted in Grades 1 and 2. The three major components of the policy include academic meaning of a grade, homework, and learning skills. As part of the evaluation process, interviews, surveys, and focus groups were conducted. This paper describes the process of the evaluation for the multi-method evaluation design. The paper begins with background information on the policy followed by the evaluation methodology. Then a summary of the findings are presented followed by a detailed discussion of the findings. The concluding section of the paper offers recommendations based on the findings from various research methodologies employed.
|
|
An Analysis of the School-based Assessment for Youth (S-BAY) in Ohio's Alternative Education Program: A Focus on Gender Sensitive Evaluation
|
| Presenter(s):
|
| Aundrey Somerville,
Bureau of Research, Training and Services,
asomerv2@kent.edu
|
| Abstract:
The School-based Assessment for Youth (S-BAY) tool is an instrument used to assess students entering the Ohio Alternative Education Challenge Grant Program by identifying their risks/needs and strengths/assets. The program serves youth who are habitually truant, disruptive in the school environment, may in addition be students who have not met academic proficiency goals of their grade level. The results of the S-BAY assessment are used to guide intervention and rehabilitation plans for the participants. Research has shown that although girls may appear to occupy complicity roles in harmful social settings, they are by their participation demonstrating delinquency. This paper will focus on how the tool may be refined to capture critical indicators of risk with regard to gender sensitive issues among the program participants. Moreover, this presentation seeks to encourage an exchange of ideas pertaining to program learning and the use of data to enhance the evaluation capacity building process.
|
| | | |
|
Session Title: Using Logic Models to Evaluate Research and Technology Diffusion Results: Two Cases
|
|
Panel Session 797 to be held in Royale Board Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Jeff Dowd,
United States Department of Energy,
jeff.dowd@ee.doe.gov
|
| Abstract:
Two major challenges in evaluating Research, Technology, and Development and Deployment Programs are summarizing research progress for large diverse research programs in a handful of measures, and getting credible impact numbers for program deployment activities such as training that only indirectly influence adoption of a technology or practice. This panel will present results of multiple, related metrics and evaluation projects at the U.S. Department of Energy that have tackled both these challenges. We will present logic models and then the use of those models. Logic modeling has been useful in both cases to make the "magic in the middle" explicit. One example is an exercise to improve metrics for research and technology development programs. The other is a training program for industrial energy efficient technologies which, because it is based on the social science theory of diffusion, applies to the diffusion of anything, not just technology. Both cases provide generic templates for others to follow.
|
|
Cutting Edge Logic Models for Research and Technology Programs
|
| Gretchen Jordan,
Sandia National Laboratories,
gbjorda@sandia.gov
|
|
During the past year we have been working to develop logic models that U.S. Department of Energy program managers and their stakeholders who make investment decisions find useful for describing the programs and their intended outcomes. From the investors' point of view that means the logic models need to be able to link the progress of many projects to intermediate points of research progress that then link to longer term outcomes. From the program managers' point of view this means the logic models have to build on technology roadmaps, Stage Gate questions & answers, Gantt charts, and systems integration analysis. For both audiences, these logic models suggest the most important intermediate performance measures to include in strategic, multi year, and annual plans.
|
|
|
Linking Projects to Program Outcomes in Metrics for Technology Development Programs
|
| John Mortensen,
Energetics Inc,
jmortensen@energetics.com
|
|
Using the logic models just described, we identified candidate metrics by mapping existing metrics against the logic, improving upon existing metrics, and filling in gaps. Gaps included elements of the logic models that had no metrics and gaps in the logical linkages, that is, the 'performance story' that the set of metrics tells. Key stakeholders were then offered opportunity to improve and a consensus was reached. We will show the metrics we found for a variety of program types. These build on and provide information for other assessments such as stage gate and peer review. The metrics help the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy programs respond to multiple requests for performance information, many of which are externally driven and to do so in a streamlined manner that is readily available to management.
| |
|
The Logic of Indirect Programs to Diffuse Technologies: The Example of Training
|
| John Reed,
Innovologie,
jreed@innovologie.com
|
|
This presentation will provide a specific example of the logic of technology diffusion for technology-related training activities, tracing that difficult path to outcomes tied to program participants, differentiated from responses of others who are not influenced by the program. The U.S. Department of Energy's Impact Evaluation Framework for Deployment Programs (presented at AEA in 2006 and set to be released in summer 2007) is under girded with ideas drawn from E. Rogers (2003) diffusion of innovations which has been widely used. The framework has identified detailed logic model templates for diffusion in four domains: End User, and three Infrastructure domains - Government/ Policy, Business, and Knowledge. The logic suggests that to be successful training needs to address the product characteristics as well as the characteristics of all four domains in order to meet the needs of the trainees. The logic also accounts for communication patterns, replication effects and sustainability.
| |
|
Getting From Training to Credible Energy Savings: An Evaluation Template
|
| Harley Barnes,
LM Business Process Solutions,
harley.h.barnes@lmco.com
|
|
EERE has developed an evaluation plan and generic survey for credibly determining the impacts of training programs. The logic-model approaches described by the other participants in this panel were used to guide the selection of questions designed to determine (1) whether energy-efficiency measures related to the training were implemented, or are planned, subsequent to the training, and (2) if so, the degree to which the training influenced the decision to implement the measures in comparison to other potential influences. Multiple questions will be used to assess the validity of the second of these two batteries of questions. We will explain why this approach was chosen to isolate the effect of the training rather than other approaches. We will describe how these results can be combined with an estimate of energy savings to enable the evaluator to calculate net energy savings.
| |
|
Session Title: Helping Communities Improve: Evaluation as Feedback for Learning
|
|
Multipaper Session 798 to be held in Royale Conference Foyer on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG
|
| Chair(s): |
| Carolyn Lichtenstein,
Walter R McDonald & Associates Inc,
clichtenstein@wrma.com
|
| Discussant(s): |
| Brigitte Manteuffel,
Macro International Inc,
bmanteuffel@macrointernational.com
|
| Abstract:
This multi-paper session describes how the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program (CMHI) operates as a feedback mechanism for federally-funded grantees funded by the Substance Abuse and Mental Health Services Administration's Center for Mental Health Services. These grant communities are funded to develop, maintain, and evaluate systems of care for children's mental health services. Much of the data compiled and analyzed by the national evaluation are fed back to grant communities using various reporting mechanisms designed to support sites in their efforts to improve. Session papers emphasize different perspectives on this theme, including the Federal impetus for this model, an overview of different components of the national evaluation, and two examples of data reporting from particular national evaluation components.
|
|
The National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program (CMHI): The Federal Perspective
|
| Sylvia Fisher,
United States Department of Health and Human Services,
sylvia.fisher@samhsa.hhs.gov
|
|
This presentation describes how CMHI national evaluation reporting activities promote continuous self-improvement activities at the grantee level and facilitate federal-level decision-making and provision of technical assistance. Data collection and reporting mechanisms include the (1) Annual Report to Congress, which describes whether participating children improved on clinical and functional outcomes and whether these improvements endure over time, and (2) System-of-Care Assessments (SOCA), which are administered three times during the 6-year grant cycle, yielding reports that help grantees more effectively allocate personnel and funding and prioritize activities to conform to system of care service delivery principles. Both inform decision-making and continuous program improvement activities by identifying gaps in system development, barriers to collaboration, and effective program components and interventions with participating children and their families. This process is described in greater detail from a federal perspective.
|
|
National Evaluation Results as a Mechanism for Continuous Quality Improvement in Grantee Communities
|
| Carolyn Lichtenstein,
Walter R McDonald & Associates Inc,
clichtenstein@wrma.com
|
|
Integral to the national evaluation of the CMHI are local evaluation capacity-building and use of evaluation data for local program development, improvement, and sustainability. Data are collected for this evaluation using several different methods, including descriptive and longitudinal interviews with caregivers of children and youth served by the system of care, site-visit interviews with key informants in each grant community, and Web-based surveys of providers and administrators. Multiple processes are used to feed data and findings back to the grant communities. This paper describes the national evaluation's research questions, the data collected to answer each question, and current feedback mechanisms.
|
|
The Evidence-based Practices and Cultural and Linguistic Competence Studies: Complex Topics for Complex Systems
|
| Kurt Moore,
Walter R McDonald & Associates Inc,
kmoore@wrma.com
|
| John Fluke,
Walter R McDonald & Associates Inc,
jfluke@wrma.com
|
|
This paper shares data analyses from the CMHI national evaluation's Evidence-Based Practices (EBP) and Cultural and Linguistic Competence (CLC) Studies focusing on characteristics of providers, administrators, and systems that are implementing new treatments and CLC practices. Initial findings are also presented from the first CLC sub-study on implementation factors based on in-depth interviews and focus groups with four 2005-funded CMHI grant communities. The adoption of evidence-based treatment practices may entail profound and potentially risky changes in system structure and behavior and the availability of rapid, high-quality evaluative feedback may facilitate this organizational learning/change. The use of evidence-based practices raises questions about cultural and linguistic competence, as well. The CLC Study examines grantee self-assessment approaches, implementation of CLC practices, and the intersection of EBP and CLC.
|
|
Data Reporting: Information to Guide Learning
|
| Susan Drilea,
Walter R McDonald & Associates Inc,
sdrilea@wrma.com
|
| John Fluke,
Walter R McDonald & Associates Inc,
jfluke@wrma.com
|
|
This paper describes two reporting mechanisms used to feed back descriptive and longitudinal interview data to grant communities to support continuous quality improvement for the national evaluation of the CMHI. Reports produced for the grant communities include:
(1) Data Profile Reports (DPRs), and
(2) Continuous Quality Improvement (CQI) Reports.
DPRs are produced three times a year at the aggregate and community level. The national evaluation statistically summarizes data from the interviews and produces PowerPoint slides of graphs and tables for communities to use in presenting this data to their stakeholders. Grant communities can download their own data file for additional analysis to support local program needs. The national evaluation also produces CQI Reports four times a year at the aggregate and community level. Grant communities use CQI Reports to identify areas needing programmatic technical assistance.
|
|
Session Title: Using Empowerment Evaluation to Facilitate Organizational Transformation: A Stanford University Medical Center Case Example
|
|
Multipaper Session 799 to be held in Hanover Suite B on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Chair(s): |
| David Fetterman,
Stanford University,
profdavidf@yahoo.com
|
| Discussant(s): |
| Abraham Wandersman,
University of South Carolina,
wandersman@sc.edu
|
| Abstract:
Empowerment evaluation is guiding evaluation efforts throughout the Stanford University Medical Center. Empowerment evaluation is a collaborative approach and designed to build evaluative capacity, engaging people in their own self-assessment and learning. The process typically consists of three steps: 1) mission; 2) taking stock; and 3) planning for the future. Strategies are monitored and information is fed back to make mid-course corrections and/or build on successes. The process depends on cycles of reflection and action in an attempt to reduce the gap between theories of action (espoused) and theories of use (observed behavior). The approach relies on critical friends to help facilitate the process. This is an important case example organizationally because the effort represents a rare opportunity to align and build on medical student education, resident training, and the education of fellows. The data generated are used to inform decision making, improve curricular practices, and enhance critical judgment.
|
|
Using Empowerment Evaluation to Engage Stakeholders and Facilitate Curriculum Reform
|
| Jennifer Berry,
Stanford University,
jenberry@stanford.edu
|
| David Fetterman,
Stanford University,
profdavidf@yahoo.com
|
|
When Stanford University School of Medicine undertook a major reform of its curriculum, the School adopted an empowerment evaluation approach to help monitor and facilitate implementation of the new curriculum. Empowerment evaluation is collaborative, engaging faculty, students, and administration in the cyclical process of reflection and action. Empowerment evaluation relies on the theory of process use. Empowerment evaluation theories and tools were used to facilitate organizational transformation at the course level. Our process included: using the School's mission as a guide; taking stock by holding focus groups and developing new survey instruments, including learning climate assessments; and planning for the future by facilitating discussions about evaluation findings with key stakeholders and having the faculty and teaching assistants revise specific courses. We also established a feedback loop to measure the success of reforms and revisions from one year to the next. Case examples highlight measurable evidence of curricular improvement.
|
|
Organizational Learning Through Empowerment Evaluation: Improving Reflection Skills With a 360 Degree Evaluation
|
| Kambria Hooper,
Stanford University,
khooper@stanford.edu
|
|
This study explores the impact of a 360 degree empowerment evaluation system in one of Stanford School of Medicine's required classes for Stanford medical students.
This evaluation system has three levels of reflection and improvement. The first is the individual member's performance. The second level of reflection and improvement is small group performance. The final level is organizational learning; the course directors and staff reflect on data, looking for group variability or patterns, to create new goals for the course structure or curriculum.
Organizational learning is dependent on each member's ability to give and receive constructive, formative feedback. In response to resistance and confusion around the new evaluation system, we developed several interventions to improve the ability of students, faculty and simulated patients to give and receive constructive feedback.
This evaluation demonstrates how organizational learning is improved when the organization's members have opportunities to reflect on individual and team performance.
|
|
Overestimation of Skills in Medical School: The Need to Train Students How to Self-assess
|
| Andrew Nevins,
Stanford University,
anevins@stanford.edu
|
|
Stanford's School of Medicine used standardized patients (SP) to help assess medical students' skills. This study focuses on students at the preclinical or course level.
Clinical skills were assessed by checklists compiled from a consensus of faculty experts. Students also rated their perception of patient satisfaction on a 1 (low) to 9 (high) scale. SPs completed a matching questionnaire, rating their satisfaction with the student. Student and SP satisfaction ratings were paired and correlated, consistent with empowerment evaluation practices.
Overall, students over-rated their performance by 0.75 points. The lowest quintile overestimated performance by 1.57 points, while the highest quintile underestimated performance by 0.003 points (p<0.01).
The most significant finding is that lower-performing medical students consistently overestimate their clinical skills. This study highlights the significance of properly training students to conduct more accurate self-assessments. Curricular efforts to improve student self-reflection may improve both clinical skills and patient interactions.
|
|
Empowerment Evaluation: The Power of Dialogue
|
| David Fetterman,
Stanford University,
profdavidf@yahoo.com
|
| Jennifer Berry,
Stanford University,
jenberry@stanford.edu
|
|
Empowerment evaluation has 3 steps including mission, taking stock, and planning for the future. However, the middle stage is not always explored in depth. One of the central features of the taking stock step is dialogue. Program participants rate how well they are doing at this step in the process, using a 1 (low) to 10 (high) rating systems. They are also required to provide evidence to support their ratings. However, it is the generative dialogue that is most characteristic of this part of the process and critical to authentic learning, on the community of learners as well as organizational learning levels. Each participant explains why they gave their rating, using documentation to build a culture of evidence. Three examples of dialogue (and norming) are provided:
1) engaged scholarly concentration directors;
2) faculty, administrators, and students grappling with curricular problems; and
3) committed clerkship directors guiding student learning in hospitals.
|
|
Using Principles of Empowerment Evaluation to Build Capacity for Institutional Learning: A Pilot Project at Stanford Hospital
|
| Heather A Davidson,
Stanford University,
hads@stanford.edu
|
|
Residency education is rapidly changing from an apprentice-based to a competency-based model where performance outcomes must guide individual feedback and continuous program improvement to meet new accreditation standards. This change represents a cultural shift for teaching hospitals and a management shift that must support systems of assessment. Many faculty members do not have the tools needed to design and implement these goals. Since institutional accreditation requires that all residency programs undergo a peer-led internal review process, Stanford Hospital has created a new protocol to build evaluation capacity.
Utilizing principles of empowerment evaluation, the pilot project formalizes feedback loops needed at both program and institutional levels. By combining performance benchmark and portfolio techniques with a mock accreditation site visit, the new protocol provides a more comprehensive assessment of overall program needs; evidence of program quality across the institution; and supports a learning culture where faculty share educational initiatives.
|
|
Sixth Presenter
Alice Edler
6507236412
edlera@aol.com
Stanford University
Empowerment Evaluation: A Catalyst for Culture Change in Post Graduate Medical Education.
Pediatric anesthesia requires special skills for interacting with small patients, not required in general anesthesia training. Empowerment evaluation was used to assess these behaviors in a Stanford pediatric anesthesia fellowship.
Trainees, faculty and aggregate data revealed the need for more clinical decision- making opportunities in the fellowship. Clinical judgment ranked the lowest.
The role of administrative chief fellow emerged from the self-assessment. It allowed for more opportunities for decision making in day-to-day schedules, curriculum, and disciplinary decisions. This position was rotated over all the fellows.
Individual and group improvements were evidenced. Fellows assumed the responsibility for creating new rotations and revising their schedules based on perceived curriculum needs. Faculty evaluations of clinical judgment significantly increased in the clinical judgment item (see table 1).
Information from the EE has allowed fellows to model self determination, form a more cohesive group, and provide opportunities for high stakes clinical decision-making.
|
|
Session Title: Is There Anything Left to Say About Logic Models?
|
|
Panel Session 800 to be held in Baltimore Theater on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| John Stevenson,
University of Rhode Island,
jsteve@uri.edu
|
| Discussant(s):
|
| Bob Williams,
Independent Consultant,
bobwill@actrix.co.nz
|
| Abstract:
This session provides several perspectives on the ways logic models can be beneficial in evaluation and the limitations on their utility. Examples are provided for the federal government context, the large-scale multi-site program context, and the local context. Although there is an extensive literature on how to develop logic models (e.g. Renger & Titcomb, 2002; Rush & Ogborne, 1991) and numerous examples of their application in evaluation studies, this session will offer insights on the circumstances best suited to their use, the limits of the conceptual foundations, and alternative choices for developing and applying these diagrammatic devices. An important theme is the tension between an inductive, emergent, developmental approach and a formally imposed structure. Each can have its place, as examples in the papers will show. Methods for deciding on layout and complexity will be discussed, along with their advantages and disadvantages.
|
|
When Does Linear Logic Help?
|
| John Stevenson,
University of Rhode Island,
jsteve@uri.edu
|
|
The advantages of applying logic models in small-scale evaluations of local-level programs may seem obvious, but there are a number of important concerns. This paper examines some advantages of using logic models in this context and some reservations about their use. Some advantages: these models can guide the development of measures for both short-term (mediating) effects and longer term outcomes in a convincing story of the causal effects of the program; logic models can be used to bring various stakeholders together on the purposes and processes of a program. Some reservations: linear causal logic may not fit the understanding of important stakeholders; there may be dramatic areas of disagreement among stakeholders; imposed logic may please no one and inductive development of logic may seem an artificial exercise. Examples will be drawn from local evaluations, considered singly and from the perspective of state-level grant-makers trying to foster coherent local programming.
|
|
|
Multi-site Evaluations and Logic Models:
Development Strategies, Uses, and Cautions
|
| Debra Rog,
Westat,
debrarog@westat.com
|
|
This paper discusses the role of logic models in multi-site evaluations. Given that multi-site evaluations can be implemented in a variety of ways, from directed by a central evaluation team to highly collaborative, the process of developing logic models and their role in the evaluations varies. These different development processes and roles will be reviewed, including the use of overall models and single site models, and the use of models for program development, evaluation design, measurement development, analysis, and writing. Examples from these different types of studies, including both quantitative and qualitative multi-site evaluations, will be highlighted. Some of the ways in which models can be 'overused' or misapplied also will be addressed.
| |
|
A Developmental Approach to Using Logic Models in Evaluation
|
| George Julnes,
Utah State University,
gjulnes@cc.usu.edu
|
|
It is well established that different evaluation methods are generally most useful at different points in the life-cycle of projects. Some have discussed a similar life-cycle dimension to different uses of logic models in supporting effective evaluations. This presentation will discuss the use of logic models in a random assignment experimental study of a policy innovation for the Social Security Administration, with an emphasis on their use in guiding the design of the evaluation, the analyses of resulting data, and the dissemination of findings.
| |
|
Constructing Logic Models of Impact to Guide Evaluation Designs of Multi-level Programs
|
| Robert Orwin,
Westat,
robertorwin@westat.com
|
|
This paper will present the logic model of impact developed for the cross-site evaluation of the Strategic Prevention Framework State Incentives Grant (SPF SIG) program. The model depicts the chain of activities that logically links funding of SPF SIG states to community and statewide outcomes, and articulates a broader theory of impact, not only of SPF SIG elements and the relationships among them, but also of non-SPF factors that potentially influence the same processes and outcomes as SPF (i.e., threaten internal validity).
This was critical to identifying the design and data elements needed to address how the flow of state- and community-level activities will lead to systems change and epidemiological outcomes in the uncontrolled “open system” where prevention operates, including measures of processes and outcomes, baseline status, and post- baseline contextual change, each assessed at both state- and community-level. Implications for evaluating multilevel programs in general also will be discussed.
| |
|
Session Title: Place Randomized Trials: Design, Implementation and First Results From Evaluating Ambulatory Psychotherapy
|
|
Panel Session 801 to be held in International Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Lee Sechrest,
University of Arizona,
sechrest@u.arizona.edu
|
| Discussant(s):
|
| Fred Newman,
Florida International University,
newmanf@fiu.edu
|
| Abstract:
The suitability of randomized clinical trials or randomized control group designs as an assumed gold standard has been heavily debated in evaluating educational programs and interventions. Recently we also experienced a recommendation split and some controversy within our organization. It is our opinion that evaluators should tailor the best design, best data analysis and best evaluation/decision tools to an evaluation project, considering the circumstances and constructs, which cannot be easily altered.
Place randomized trials are a type of design with potential of high internal(causal)validity and high external validity (generalizability)in many applied setting where randomization at the individual level is not possible. This symposium will report about our strategies, namely the five data-box conceptualization we use as a comprehensive framework, our experiences and first results of a large scale program evaluation focusing on the impact of ambulatory psychotherapy in Germany.
|
|
Designing a Place Randomized Trial in a Field Setting
|
| Werner Wittmann,
University of Mannheim,
wittmann@tnt.psychologie.uni-mannheim.de
|
|
Evidence based research to improve and evaluate the quality and impact of interventions in different areas, e.g. the health system has reached Germany. We have been asked by a leading health insurance company, the 'Techniker Krankenkasse, Hamburg' (Technicians health insurance company,Hamburg) to do the evaluation of a computerized assessment system, giving feedback to the therapist about the progress of clients seeking help in ambulatory psychotherapy. We chose to randomize at the level of the therapist, because randomization at the client level was not feasible. The control group therapists had no access to the computerized feedback system. The whole evaluation design was conceptualized according to the Northwestern path of the five data box conceptualization, developed as a comprehensive evaluation system by the author. The different steps in implementing design, assessment and the necessary power to satisfy different stakeholder interests will be described in detail.
|
|
|
Tailoring Assessment Tools to Different Stakeholder Groups, Randomization and Treatment Integrity Checks
|
| Andros Steffanowski,
University of Mannheim,
steffanowski@tnt.psychologie.uni-mannheim.de
|
| David Kriz,
University of Mannheim,
krizdavid@yahoo.com
|
|
An overview about the various assessment tools recommended and implemented to map the different data boxes is given. Three different treatment schools, namely behavior therapy, analytical therapy and depth psychology participate in the study. The common, the school specific and the disease specific assessment instruments are described along with the randomization checks, the number of treatment sessions and their integrity. Multiple act criteria as the fairest approach to map stakeholder interest and symmetry between treatment and outcome criteria in this project are discussed.
| |
|
Evaluating Ambulatory Psychotherapy in a Field Setting With a Place Randomized Trial, First Results
|
| Manuel Voelkle,
University of Mannheim,
voelkle@rumms.uni-mannheim.de
|
|
A priori power analysis in detecting effect sizes just above the break-even point forced us to assess more than 2000 clients and the main stakeholder complied to this number. So far we have more than 800 clients assessed. There are at least five assessments of the clients progress during therapy and a one year follow-up. It will be demonstrated how to analyze such a repeated measurement or time series design with classical and modern data analytic tools. First results in terms of pre-post effect sizes mapping progress are reported. First results are very promising, medium effect sizes right after the first sessions bolstering late Ken Howard's research and medium to large effect sizes later on.
| |
|
Session Title: Educational Technology: Evaluation Opportunities
|
|
Multipaper Session 802 to be held in Chesapeake Room on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Distance Ed. & Other Educational Technologies TIG
|
| Chair(s): |
| Shani Reid,
Macro International Inc,
shani.a.reid@orcmacro.com
|
|
It's not Just Fun and Games: Evaluating Games in the Classroom
|
| Presenter(s):
|
| Shani Reid,
Macro International Inc,
shani.a.reid@orcmacro.com
|
| Helene Jennings,
Macro International Inc,
jennings@macroint.com
|
| Betsy Peisach,
Maryland Public Television,
bpeisach@mail.mpt.org
|
| Abstract:
There has been increased interest in educational computer games as educators, researchers, and scientists work together to take advantage of the opportunities games provide to enrich the learning experiences of the multimedia generation. One factor hindering the expansion of educational games to the classroom however is the lack of evaluative data that demonstrates the effectiveness of these games.
Macro International has the opportunity to advance research in this area through evaluating Maryland Public Television's 5-year grant to develop a series of games focused on pre-algebra skills. In this presentation we will highlight the integral role evaluation plays in educational computer games designed for use in the classroom. We will discuss techniques and findings from various evaluation methodologies employed, including observations of teachers using games in the classroom and prototype testing. We will also discuss our plans for Randomized Controlled Trials in subsequent years.
|
|
Evaluative Learning and Technology Use in Educational Innovation Programs
|
| Presenter(s):
|
| Juna Snow,
InnovatEd Consulting,
jsnow@innovatedconsulting.com
|
| Abstract:
This paper discusses evaluation design issues for three process evaluations of technology programs in educational settings. These innovations, idealizations of betterment typically based on pre-existing processes, intended to change prior processes. New programs, as well as new technologies, necessitate formative, diagnostic evaluation because designers can not predict fully in what ways participants will make use of and benefit from the program innovations.
In each of the three cases, the programs implemented technology innovations with goals to create teacher change and transform practice. Specific evaluation design and methods used are discussed to illustrate the framework introduced in this paper. Moreover, this paper demonstrates how the technological nature of the innovations provided the medium and tools for data collection. The evaluative learning about the evaluand was fostered and shared with clients and stakeholders in each case as part of the methodology framework presented in the paper.
|
|
Evaluation of the Computer's Efficacy in Education
|
| Presenter(s):
|
| Paul Lorton Jr,
University of San Francisco,
lorton@usfca.edu
|
| Abstract:
Does the computer as an educational tool help or hinder the educational effort? We know that the answer really depends on the substantive question being asked – and evaluation of efficacy in any domain depends on how we define efficacy. In this survey, the focus is on how the evaluation of this at once too costly and too necessary educational technology has been shaped over the almost fifty years of effort in the field, and what we have learned as evaluators and from the evaluations. Starting with the early efforts in the 1960's where the bar was set at a fairly low self fulfilling level to the most recent efforts to tease out the always subtle but real effects, the discussion will follow threads in the affective and effective domains while glancing often at the parallel state of other technologies used in education.
|
| | |
