| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Using a Checklist Methodology to Evaluate E-Learning In Organization |
|
Roundtable Presentation 618 to be held in the Boardroom on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Distance Ed. & Other Educational Technologies TIG
|
| Presenter(s):
|
| Anne-Marie Oulai, Western Michigan University, anne.oulai@wmich.edu
|
| Abstract:
Recent advances and innovations in information technology (IT) have shifted learning methods from the face to face traditional classroom environment to an on demand environment allowing learning to take place from a distance and at any time. With all the expenditures on e-learning programs, it is necessary for evaluation to be conducted to determine effectiveness in the organization. E-learning evaluation is conducted through multiples methods such as Kirkpatrick four level model of evaluation and Phillips's five level model of evaluation.
The purpose of this study is to conduct a comprehensive review of existing methods/models used by organizations to evaluate e-learning and develop a framework to enhance the process of e-learning evaluation in organizations. The E-learning Evaluation Checklist (ELEC) uses the Key Evaluation Checklist (KEC) framework and addresses elements required for evaluation of e-learning in organizations. The study will add to the existing body of knowledge geared toward e-learning evaluation.
|
| Roundtable Rotation II:
Evaluating Change in Practice Through Online Coursework |
|
Roundtable Presentation 618 to be held in the Boardroom on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Distance Ed. & Other Educational Technologies TIG
|
| Presenter(s):
|
| Amy Gaumer Erickson, University of Kansas, aerickson@ku.edu
|
| Abstract:
The goal of professional development is to improve professional practice, but evaluating this change is often an elusive endeavor. Online modalities of professional development are considered effective in delivering content across geographical distances, but they also produce an additional barrier to evaluating change in practice. This roundtable presentation will provide evaluation methods implemented in an online course series, including competency surveys, curriculum-reference assessments, goal attainment scaling, and document review of assignments. Participants will discuss these measures and identify additional measures and strategies that provide evidence of change in practice.
|
|
Session Title: International Advocacy Evaluation: Coordinating Comprehensive Evaluation Across Seven Countries
|
|
Panel Session 619 to be held in Panzacola Section F1 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Advocacy and Policy Change TIG
|
| Chair(s): |
| Laura Ostenso, Innovation Network Inc, lostenso@innonet.org
|
| Discussant(s):
|
| Julia Coffman, Harvard Family Research Project, jcoffman@evaluationexchange.org
|
| Abstract:
The ACTION project, working in 7 countries worldwide, developed an internal evaluation system, complimented by external evaluation, to support its advocacy work to fight tuberculosis (TB). In this session, ACTION project partners from donor and high-TB burden countries, along with it's external evaluation partner, Innovation Network, will demonstrate ACTION's evaluation processes and the successes and setbacks in designing and implementing an advocacy evaluation system that is successful in guiding a multi country advocacy campaign. The session will provide an overview of ACTION's evaluation system and will examine the following: How is advocacy evaluation implemented across a seven country project? How does cultural context impact how M&E is implemented? How is it integrated into everyday operations? How is it best leveraged for decision making? How did an advocacy organization bridge internal evaluation, designed for mapping internal advocacy strategies, and an external evaluation, designed to link strategies to long-term goals?
|
|
Developing and Coordinating Monitoring and Evaluation for International Advocacy
|
| Paul Jensen, RESULTS Educational Fund, pjensen@results.org
|
|
Victoria Treland is the Senior Project Manager for ACTION (Advocacy to Control Tuberculosis Internationally) at RESULTS Educational Fund. Victoria will explain the process of developing and refining a Monitoring and Evaluation (M&E) system flexible enough to meet the unique needs of a multi-country advocacy campaign. She will address how ACTION's internal M&E system helped ACTION to strategically align its activities and objectives from the first three years of the project into its second five-year phase. The presentation will address how ACTION effectively coordinates among seven international partners to create a streamlined system tailored to each country's specific needs. Victoria will also describe how ACTION used external evaluation findings in order to comprehensively understand its overall strengths and weaknesses and modify its work accordingly. Finally, she will discuss how ACTION's internal and external evaluations will impact its work moving forward.
|
|
|
Using a Monitoring and Evaluation System for Strategic Decision Making
|
| Aldwyn Hamilton, RESULTS Educational Fund, ahamilton@results.org
|
| Evelyn Kibuchi, AIDS Non Governmental Organization (NGO) Consortium, action@action.org
|
|
Aldwyn Hamilton and Evelyn Kibuchi have been a part of the ACTION project since its inception. As ACTION partners, they work extensively with ACTION's M&E system. Chris and Evelyn will discuss the use of the M&E system in their daily work, and showcase examples of how their use of the M&E system has informed particular areas of their advocacy work. Each presenter will offer their experiences of balancing M&E with everyday operations, which is often easier said than done, and the challenges of learning to use and integrate M&E into strategic decision making. They will also discuss the differences between ACTION's M&E system in donor and high-tuberculosis (TB) burden countries. The presentation will shed light on advocacy evaluation as a learning tool both for ongoing strategic decision making, and for communicating progress to partners and funders.
| |
|
From the Outside In: Integrating Internal Monitoring and Evaluation (M&E) With an External Evaluation
|
| Laura Ostenso, Innovation Network Inc, lostenso@innonet.org
|
|
Laura Ostenso, an Associate at Innovation Network, coordinated the external evaluation for the ACTION project in 2008-2009. In designing the external evaluation, which consisted of thirty interviews, one partner-wide focus group, and the research and writing of four advocacy case studies, the Innovation Network evaluation team built on ACTION's internal evaluation system to delve into the "how" and "why" questions of ACTION's advocacy successes. Laura will demonstrate how the Innovation Network team pulled appropriate information from ACTION's comprehensive internal M&E system, and she will relay the external findings in order to shed light on framing external advocacy evaluation questions. Laura will also discuss the usefulness of case studies in exploring specific focus areas important to ACTION and its primary funder.
| |
|
Session Title: Using Multiple Sources of Evidence to Evaluate Policy Changes: Examining the Impact of the 2007 Smoke-Free Arizona Act
|
|
Panel Session 621 to be held in Panzacola Section F3 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Health Evaluation TIG
|
| Chair(s): |
| Michele Walsh, University of Arizona, mwalsh@u.arizona.edu
|
| Abstract:
Adequately evaluating public policy changes requires several important elements, including an understanding of the context in which the policy is applied; a general effect theory of what the desired outcomes are, and how change should occur; and multiple sources of data to determine whether the policy was actually implemented, and what the outcomes were. This session outlines how multiple sources of evidence can be used to evaluate the impact of larger-scale public policies, addressing both the initial context of the policy and examining outcomes. The evaluation of the adoption of a statewide smoking ban will be used to illustrate the described approach.
|
|
Understanding the Context of Policy: Setting the Stage For Evaluation
|
| Michele Walsh, University of Arizona, mwalsh@u.arizona.edu
|
|
Policy change is generally a long, drawn-out process that occurs in a highly-charged environment involving multiple stakeholders, with sometimes-conflicting agendas. Developing an evaluation of the outcome of such a process requires that multiple perspectives are taken into account in defining what outcomes to examine and what sources of data should be used. This paper sets the stage for the methodological approaches to follow by outlining the contextual background of a statewide smoking ban and describing how that affected some of the methods chosen.
|
|
|
Evaluating New Policies: General Principles of Multi-method Approaches - Exemplified With Arizona's 2007 Smoke-free Law
|
| Frederic Malter, University of Arizona, fmalter@email.arizona.edu
|
|
New policies usually have a multitude of effects. An evaluation of new policies begins with careful examinations of the changes to be expected in the social, political and psychological reality of people affected by it. The necessity of a general effect (causal) theory becomes evident when evaluators try to answer questions around why a given change should occur. In the case of Arizona's 2007 smoking ban, a causal model was developed that specified anticipated effects in several domains, such as compliance, reductions in air pollution, changes in smoking behavior and impacts on hospitality's business revenue. Estimates of anticipated effects were counterbalanced against a careful assessment of the share of the population already covered by smoke-free provisions before the state-wide ban to obtain realistic projections. The presentation will give an overview of the multiple data sources that were identified and the data-analytic methods chosen to best answer questions about time-dependent changes.
| |
|
The Effect of the Smoke-Free Arizona Act on Cessation and Other Behaviors
|
| John Daws, University of Arizona, johndaws@email.arizona.edu
|
|
The campaign to pass Arizona's state-wide smoking ban focused on the reduction of second-hand smoke. As a matter of policy, the campaign did not directly promote smoking cessation. Nonetheless, it would be a desired indirect outcome of the ban that some smokers decide to quit. Calls to the state's telephone cessation service were used as a proxy for intentions to quit in the general population. We examined how call volumes responded to ban-related health messages, to the ban's passage in November 2006, and to its implementation seven months later. We used regression analysis to isolate the effects of the ban from the effects of other events, such as the cigarette-tax increase in December 2006. Other measures of public response to the ban included (a) reduced particle pollution inside bars and restaurants, (b) public-opinion surveys of smokers and non-smokers both before and after implementation, and (c) public complaints about non-compliant locations.
| |
|
Public Health and Economic Impact of the New Smoke-free Law
|
| Patricia Herman, University of Arizona, pherman@email.arizona.edu
|
|
One of the main arguments for the statewide smoking ban was evidence of the detrimental health effect of secondhand smoke (SHS) on those exposed, and many smoking ban opponents cited the potential for it to cause serious economic losses to establishment owners. In order to measure the impact on health of the ban we compared statewide hospital admission data before and after the ban for diagnoses known to be associated with SHS exposure. In order to better imply causation we also looked at admissions for diagnoses not likely to be associated with SHS exposure and we split the analysis between counties with and without previous smoking bans. To determine the economic impacts we analyzed three types of data: self-report patronage of smoking and non-smoking establishments before and after the ban, statewide taxable monthly sales for bars and restaurants, and statewide monthly employment in the food services and drinking places sector.
| |
|
Session Title: Action Research: A Formative Evaluation Strategy in Times of Complexity and Chaos
|
|
Panel Session 622 to be held in Panzacola Section F4 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Systems in Evaluation TIG
and the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Daniel Folkman, University of Wisconsin Milwaukee, folkman@uwm.edu
|
| Abstract:
How do we evaluate programs that are designed and implemented in complex, rapidly changing, and indeterminate environments? Join our panel presentation to learn how action research was employed as a formative evaluation strategy that is well suited for programs operating in complex and/or chaotic environments. The panel members will highlight the action research experience of a group of middle school teachers and administrators as they implement a physical fitness and nutrition education program in their schools. Emphasis will be placed discussing how this group of individuals were able to identify and respond successfully to a myriad of unanticipated issues that emerged as they implemented their projects. The presentation concludes with a matrix of critical learning that has resulted from the action projects. This matrix can serve as a blueprint for evaluation practitioners who are interested in applying action research as a formative evaluation strategy in complex and chaotic program environments.
|
|
Complexity Theory and Action Research as Formative Assessment
|
| Daniel Folkman, University of Wisconsin Milwaukee, folkman@uwm.edu
|
|
This presentation will provide an overview of the action research strategy and its connection to evaluation practice in complex environments. Emphasis will be placed on how the action research process was organized and facilitated as both a formative evaluation strategy as well as professional development training for program staff and other stakeholders. The end result of this process is a collection of action projects that document the implementation process, the opportunities and challenges that were encountered, and the results that were produced. In addition, the participants gathered stories and anecdotes that captured the dynamic essence of the programs being implemented. The presentation will highlight the use of anecdotes as a data collection strategy and its use in formative assessment, problems solving, and continuous program improvement.
|
|
|
Action Inquiry: From Project Implementation to Learning in Times of Complexity and Chaos
|
| Sheryl Gotts, Family Leadership Academy, sherylgotts@sbcglobal.net
|
|
This presentation will highlight several of the action research projects that were undertaken by the different team members. Each action project represents a different strategy that was employed as the group of teachers and administrators worked they way toward achieving program goals that included a) increasing the level of physical activity among the students in school and home, and b) improving the eating habits of students and families through nutrition education. The presentation concludes with a matrix of critical learning that has resulted from the action projects. This matrix can serve as a blueprint for evaluation practitioners who are interested in applying action research as a strategy that combines formative assessment with staff training and development.
| |
|
Learning to Making Exercise Fun: An Emergent Outcome
|
| Mary Laubenheimer, Milwaukee Public Schools, laubenme@milwaukee.k12.wi.us
|
|
This presentation is an in-depth case study of one action research project. The goal was to increase student physical activity by organizing an exercise session before lunch. Challenges came in all directions including resistance among some of the students and finding time and space in the school to hold the exercise sessions. The case study highlights the trial-and-error sequence in which a solution was eventually found. The presentation concludes with a discussion on how the process represents movement within the complexity framework. Some of the activities represent simple and predictable solutions while the "WOW" moment of success occurred as student receptivity emerged spontaneously and unpredictably. What followed was a cascade of min-successes including increases in student exercise, improved classroom and academic behavior, and growing interest in the exercise session school wide.
| |
|
A Birds Eye Perspective: Seeing Systemic Issues Emerge From Individual Action Research Projects
|
| Brett Fuller, Milwaukee Public Schools, fullerba@milwaukee.k12.wi.us
|
|
This presentation takes a bird's eye view of the combined action research projects and reflects on their value from a district wide, administrative perspective. Several of the action research projects uncovered a variety issues that call for a larger, system wide response from school district administrators. One such issue revolves around the challenge of improving the nutritional value of school lunches and snacks. One action project involved documenting the choices that students make in the school lunch line in terms of vegetables versus fries and other high fate menu items. It turns out that the range of choices is being limited by the school lunch menu, which is a central office and district policy issues. Learn how this school administrator responded to the emergent information from an action research project and framed a larger systemic problem solving strategy around nutrition and school lunch menus.
| |
|
Session Title: Is Evaluation Meeting its Purpose: Examples From the Health Sector
|
|
Multipaper Session 623 to be held in Panzacola Section G1 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Evaluation Use TIG
, the Health Evaluation TIG, and the Organizational Learning and Evaluation Capacity Building TIG
|
| Chair(s): |
| Lennise Baptiste,
Kent State University, lbaptist@kent.edu
|
| Discussant(s): |
| David Hough,
Missouri State University, davidhough@missouristate.edu
|
|
Evaluation In Context: Developing Standardized Measures of Success for Tobacco Cessation Quitlines in North America
|
| Presenter(s):
|
| Jessie Saul, North American Quitline Consortium, jsaul@naquitline.org
|
| Tamatha Thomas-Haase, North American Quitline Consortium, tthomas-haase@naquitline.org
|
| Sharon Cummins, University of California San Diego, scummins@ucsd.edu
|
| Lawrence An, University of Minnesota, lcan@umn.edu
|
| Stephen Michael, University of Arizona, smichael@email.arizona.edu
|
| Michael Luxenberg, Professional Data Analysts Inc, michael@pdastats.com
|
| Anne Betzner, Professional Data Analysts Inc, abetzner@pdastats.com
|
| Linda Bailey, North American Quitline Consortium, lbailey@naquitline.org
|
| Abstract:
Realizing a need for a common evaluation framework that incorporates the unique economic, environmental, and political situation of individual tobacco cessation quitlines, the North American Quitline Consortium has begun a process of developing standardized measures of quality and outcomes. This paper reviews the process for initiating and implementing the Quality Improvement Initiative as well as its initial outcomes of measuring and reporting reach and quit rates. The particular value of the Initiative is that it was designed within the larger context of tobacco control and limited funding resources in general, but also within a community of practice where many different stakeholders have a voice in creating the goals and final products of that Initiative. The final measurements have been standardized, and allow for reporting of key variables that emphasize the specific context of each quitline. Implications for other communities of practice and evaluation efforts will be discussed.
|
|
Apples, Oranges, and Bananas: A Case Study of the Relationship Between Context and Use in Three Statewide Smoking Cessation Program Evaluations
|
| Presenter(s):
|
| Anne Betzner, Professional Data Analysts Inc, abetzner@pdastats.com
|
| Lija Greenseid, Professional Data Analysts Inc, lija@pdastats.com
|
| Julie Rainey, Professional Data Analysts Inc, jrainey@pdastats.com
|
| Abstract:
This presentation compares and contrasts three statewide smoking cessation program evaluations to explore the complicated relationship between evaluation context and use. The three evaluations vary in terms of the size of the interventions, external stakeholder pressure for evaluation, clients' levels of knowledge and experience with evaluation and tobacco cessation programming, and clients' desire and availability to manage the intervention and evaluation. Together these contextual factors interact to affect both the potential and actual use of evaluation processes and findings. This presentation will describe the three evaluation cases, highlight factors that theoretically should help or hinder use, explain which of these factors play the largest role in affecting the ultimate impact of the evaluations, and share the attempts by the evaluators to increase the use of the evaluation in order to improve state health outcomes.
|
| |
|
Session Title: An Evaluative Approach to Quality Assurance in Higher Education
|
|
Multipaper Session 625 to be held in Panzacola Section H1 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Presidential Strand
and the Assessment in Higher Education TIG
|
| Chair(s): |
| William Rickards, Alverno College, william.rickards@alverno.edu
|
| Discussant(s): |
| Michael Scriven, Claremont Graduate University, mjscriv@gmail.com
|
| Jean A King, University of Minnesota, kingx004@umn.edu
|
| Abstract:
What happens when an entire sector is required to shift from a compliance/audit model of quality assurance to an evaluative one, one that asks and answers explicitly evaluative questions about the value of outcomes and key contributing processes? Is it possible to build an evaluative culture across an entire sector? Can meaningful self-evaluation thrive alongside an external evaluation function?
New Zealand has embarked on a program of reforms to strengthen higher education provision and outcomes. A significant element is a new system for 'evaluative' quality assurance. This session will describe challenges and learnings encountered in: (1) the development of an evaluative approach that meets the needs of diverse providers and other stakeholders; (2) the capacity building efforts to shift from an audit/compliance/monitoring to an evaluative model; and (3) the change management and culture change strategies used to create both mindshift and behaviorshift in participating government agencies and across the sector.
|
|
The Challenges of 'Evaluative Quality Assurance': Why, How, and Why Now
|
| Syd King, New Zealand Qualifications Authority, syd.king@nzqa.govt.nz
|
|
Why an evaluative approach to quality assurance? This paper outlines the context, the rationale and the challenges associated with the most significant change in quality assurance in higher education in New Zealand in the last 50 years.
Topics covered include the drivers for change, what needed to be done and why (what was "the problem"?), the needs addressed and the emerging benefits of the new approach.
The sector affected includes New Zealand's eight universities, three wananga (Maori universities), 20 Institutes of Technology and Polytechnics, approximately 750 Private Training Establishments, 39 Industry Training Organizations and a handful of Government Training Establishments.
This paper considers the factors underpinning the shift to an 'evaluative approach focusing on outcomes and key contributing processes' - a highly significant shift for providers and the agency staff responsible for evaluating the programs and organizations.
|
|
Methodological Challenges in Evaluative Quality Assurance: Shining Lights In Dark Corners
|
| E Jane Davidson, Davidson Consulting Ltd, jane@davidsonconsulting.co.nz
|
|
What does an 'evaluative' approach to quality assurance look like? What evaluation theories and methodologies does it draw on, and what does that look like in practice? What was the rationale for the design?
There are numerous political and methodological challenges in striking a practical balance among competing stakeholder needs and requirements. For example, diverse organizations need evaluations to be tailored to align with their distinctive contributions, their learners, and their contexts. They also need flexibility in the way 'quality' and 'value' are defined, and in the nature of the data used to answer important evaluation questions. At the same time, education providers find comparable data from within the sector useful for benchmarking, and funding bodies need information that will allow them to compare providers and/or programs.
This presentation outlines how evaluation logic and methodology were used to forge a realistic balance among these competing needs and concerns.
|
|
Applying the Evaluative Approach in Indigenous Contexts, Challenges and Opportunities
|
| Nan Wehipeihana, Independent Consultant, nanw@clear.net.nz
|
| Kate McKegg, The Knowledge Institute Ltd, kate.mckegg@xtra.co.nz
|
|
Over the last two decades, a key development in the New Zealand higher education sector has been the growth of Maori providers, delivering education grounded in Maori pedagogy and philosophy. Also, mainstream higher education providers now have an increased focus on supporting the retention and achievement of Maori students.
This paper will consider the ways in which the new approach is able to take account of the distinctive contributions of these Maori providers, as well as the needs of Maori students in mainstream situations. Drawing on the experiences of Maori providers and feedback from other providers, the paper will discuss key issues such as the extent to which the approach has credibility and validity in Maori contexts.
|
|
Evaluation Capability Building for Changes in the Quality Assurance of a Whole Education Sector
|
| Sue Walbran, New Zealand Qualifications Authority, sue.walbran@nzqa.govt.nz
|
| Phillip Capper, WEB Research, phillip.capper@webresearch.co.nz
|
|
This paper will outline the early initiatives used by the New Zealand Qualifications Authority to begin developing evaluation capability with over 750 diverse educational organizations. This includes the use of multiple strategies to first develop an understanding and willingness to adopt the change and then to encourage the use of evaluative tools and techniques to strengthen organizational self-evaluation.
Initiatives covered in the presentation include the provision of regional workshops to over 1600 participants, the use of a facilitated web-based discussion forum and intensive workshops on the theory and practice of evaluation. Each will be visited in terms of how well each worked in relation to what it set out to achieve as well as some advice for others wanting to use these strategies.
The understanding required by government agencies in order to support and progress the change is also explored particularly in terms of how the system must engage with capability building initiatives that will take several years to mature.
|
|
Manipulations, Manoeuvrings and Machinations: Getting Buy-in, Come Hell or High Water
|
| Tim Fowler, New Zealand Qualifications Authority, tim.fowler@nzqa.govt.nz
|
|
What is required to achieve big shifts in understanding and garner support for an evaluative approach that is not necessarily understood or welcomed by all players?
An evaluative approach directly challenges conventional thinking about quality assurance in higher education. It requires all stakeholders - the diverse parts of the sector, the evaluators themselves (both internal and external), the other agencies involved, national politicians, and communities alike - to reconsider what constitutes quality in higher education and how that quality should be recognized, valued and improved.
The challenge is to convince stakeholders the change is worth attempting even if not all of their concerns can be allayed in advance, nor success in all circumstances guaranteed at first attempt.
To change an entire system with many diverse interests requires courage and commitment - along with the ability to win some battles while keeping a careful eye on the strategy to win the war.
|
|
Session Title: Exploring Methods and Tools in Higher Education Evaluation
|
|
Multipaper Session 627 to be held in Panzacola Section H3 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Assessment in Higher Education TIG
|
| Chair(s): |
| Stanley Varnhagen,
University of Alberta, stanley.varnhagen@ualberta.ca
|
| Discussant(s): |
| Stanley Varnhagen,
University of Alberta, stanley.varnhagen@ualberta.ca
|
|
The Use of Confirmatory Factor Analysis to Support the Use of Subscores for a Large-Scale Student Outcomes Assessment
|
| Presenter(s):
|
| Rochelle Michel, Educational Testing Service, rmichel@ets.org
|
| Towanda Sullivan, Educational Testing Service, tsullivan@ets.org
|
| Abstract:
This paper presentation will use confirmatory factor analysis to provide validity evidence to support the use of the various skill- and context-based subscores that are currently reported for a large-scale student outcomes assessment. The assessment is designed to provide data to accomplish five major goals: 1) to measure and document program effectiveness, 2) to assess student proficiency in core academic skill areas, 3) to allow comparisons of performance with other programs nationwide, 4) to conduct benchmark and trend analyses, and 5) to meet the requirements of the Voluntary System of Accountability (VSA). In order to achieve these goals, test users are provided with skill- and context-based subscores. Users of large-scale outcomes assessments will find the paper presentation useful when making decisions about how to use information reported on an assessment's subdomains.
|
|
The Impact of 'No Opinion' Response Options on Data Quality in High-Stakes Evaluations
|
| Presenter(s):
|
| Dorinda Gallant, The Ohio State University, gallant.32@osu.edu
|
| Tiffany Harrison, The Ohio State University, tkw97@aol.com
|
| Aryn Karpinski, The Ohio State University, karpinski.10@osu.edu
|
| Jing Zhao, Ohio State University, zhao.195@osu.edu
|
| Abstract:
Universities and colleges have traditionally used students' evaluation of instruction as part of the tenure and promotion process for faculty. The purpose of this study is to investigate the impact of 'no opinion' response options in high-stakes evaluations. Specifically, this study investigates 'no opinion' response options on students' evaluation of faculty instruction at a large Midwestern university. The sample consists of over 65,000 undergraduate students who completed evaluation of instruction instruments in courses taught by faculty during autumn quarter of 2005. Data analyses will include comparing responses for each item across college classification, comparing means for each item with and without the inclusion of the 'no opinion' response, and examining patterns in item responses. Policy implications for the results of this study will be discussed.
|
|
Using Cognitive Interviewing to Grasp Cultural Context: Evidence of Construct Validation Among Under-represented Populations
|
| Presenter(s):
|
| Valerie K York, Kansas State University, vyork@ksu.edu
|
| Sheryl A Hodge, Kansas State University, shodge@ksu.edu
|
| Christa A Smith, Kansas State University, christas@ksu.edu
|
| Abstract:
Cognitive interviewing was utilized to better understand unexpected survey results between two groups of traditionally under-represented students in the Engineering field. These students had been randomly assigned into groups of scholarship recipients and matched-cohort controls. Unexpectedly, survey results indicated more positive outcomes for the control group over time in terms of their perceptions of both job expectations and STEM (Science, Technology, Engineering, and Mathematics) field efficacy. Speculating that interpretation of survey items may have played a part in these findings, the evaluation team engaged in cognitive interviewing of the students to gather their feedback on the items. These results were also compared to those obtained through cognitive interviewing conducted with a group of evaluators to determine the necessity of conducting the cognitive interviewing process with the sample of interest. Implications are discussed related to the utility of cognitive interviewing when evaluating programs targeting traditionally under-represented populations.
|
|
Maintaining High Quality Responses and Response Rates Over Time: The Challenges and Solutions in Evaluations of Undergraduate Summer Science Research Programs
|
| Presenter(s):
|
| Courtney Brown, Indiana University, coubrown@indiana.edu
|
| Rebekah Kaletka, Indiana University, rkaletka@indiana.edu
|
| Abstract:
This paper will discuss methods used to successfully increase both short- and long-term response rates of college students participating in a summer research program. Specifically, the paper will focus on methodologies that engage participants even after they leave the program and become increasingly occupied and mobile. Examples from both a nationwide multi-site and a single-site undergraduate summer science research will be discussed. Personal interaction and relationship building with participants and program directors will be presented as methods for gaining engagement. The use of technology such as web-based surveys, social networking websites, text messaging, and program and participant profile web pages will be addressed. These will be discussed as they are effective methods to maintain contact with this transient population. These technologies allow for distribution of information, maintenance of up-to-date contact information, and collection of follow-up data which together help to increase both the rate and quality of responses.
|
|
Proxies and the Autoptic Proference: Using a Logic Model to Construct and Deconstruct the Teacher Education Accreditation Council (TEAC) Accreditation of an Educational Leadership Program
|
| Presenter(s):
|
| John Hanes, Regent University, jhanes@regent.edu
|
| Glenn Koonce, Regent University, glenkoo@regent.edu
|
| Abstract:
We use an elaborated general program logic model to construct an overview of an Educational Leadership Program in its university and societal contexts. This model reflects the relationships among the standard inputs, processes, outputs, and outcomes as well as the Teacher Education Accreditation Council (TEAC) Quality Principles and Inquiry Brief components. The six main Interstate School Leaders Licensure Consortium (ISLLC) standards provide both a set of status claims and an initial group of outcome variables for the model. We place the 20 pieces of evidence from Appendix E of the TEAC Inquiry Brief within the structure of the model, and we also integrate the other chapters and appendices from the Brief.
This initial model prompts some questions regarding the nature of the evidence and what is actually being measured and assessed. In most cases, the evidence that supports the status claims represents proxies for what we really want to know about our graduates. Utilizing Wigmore's idea of the autoptic proference, we explore the model with the intent of addressing a broader array of outcomes that relate to our ultimate customers and to our goals as a school of education. From an extended perspective based upon a three year engagement with the accreditation process, we also inquire into the nature of value-added and causal claims within the realm of available evidence, now and in the future.
|
| | | | |
|
Session Title: The Value of Shared Metrics Approaches
|
|
Think Tank Session 628 to be held in Panzacola Section H4 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Presenter(s):
|
| Mark Kramer, FSG Social Impact Advisors, mark.kramer@fsg-impact.org
|
| Discussant(s):
|
| Hallie Preskill, FSG Social Impact Advisors, hallie.preskill@fsg-impact.org
|
| Becca Graves, FSG Social impact Advisors, becca.graves@fsg-impact.org
|
| Abstract:
Over the past decade, the field of nonprofit and foundation evaluation has experienced slow and difficult but inexorable progress toward the development of metrics. The prevailing paradigm, however, focuses on the results of specific grants or individual organizations, obscuring the potential for learning and alignment through which these evolving measurement processes could most powerfully increase social impact.
In this think-tank, we will explore the potential for shared metrics approaches (defined as systems in which multiple organizations use a common set of measures to monitor performance or track progress toward goals) to increase efficiency, improve knowledge-sharing, and advance strategic alignment among similarly-focused organizations. Drawing on FSG's research and participants' experiences, we will consider questions such as: Under what conditions are shared metrics approaches most relevant and useful to the field? What would contribute to increased use of shared metrics approaches? What barriers must be broken down? Who should take the lead?
|
|
Session Title: Harnessing the Challenges of Conducting Public Education (K-12) Evaluations
|
|
Panel Session 629 to be held in Sebastian Section I1 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Government Evaluation TIG
and the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Rakesh Mohan, Idaho Legislature, rmohan@ope.idaho.gov
|
| Discussant(s):
|
| Nancy Zajano, Learning Point Associates, nancy.zajano@learningpt.org
|
| Abstract:
Policy debates involving public education (K-12) are often charged with strong emotions and are fueled by conflicting and competing interests of well-organized stakeholders. For evaluators to be successful in this environment, they have no choice but to fully understand the evaluation context and manage the politics of that context while avoiding getting entangled in the politics. This understanding of evaluation context includes knowing the sphere of the authorizing environment, knowing who are the sponsors and stakeholders, and understanding the relationships among those stakeholders. Managing the politics of the context means that evaluators should try their best to maximize both their independence from and responsiveness to sponsors and stakeholders. This panel session will use examples of controversial evaluations from Idaho, Texas, and Washington to illustrate how to effectively navigate through the politically-charged world of K-12 evaluations.
|
|
The Hot Potato That Cannot Be Dropped: Evaluations That No One Wanted but Someone Had to Do
|
| Bob Thomas, King County Auditor's Office, rthomasconsulting@msn.com
|
|
Legislative evaluation agencies often receive study mandates whose motivations may have little to do with an interest in promoting good public policy. They may result from deal-making, such as appeasing a certain legislator, or in an effort to pander to interest groups or avoid litigation. The problem for the evaluators is that those in power may have little or no real interest in the study; and there could be the possibility that no matter how well the evaluation is carried out, it will be considered dead on arrival. Understanding the context that gave rise to the evaluation, and in which it will be received, can help evaluators find a path through the political minefield and, ultimately, produce a study of lasting value. This presentation will draw upon experiences with two evaluations in the field of K-12 education, one involving a study of Special Education funding in Washington State, and another study that examined the adequacy of public education funding in Idaho.
|
|
|
Managing the Politics of K-12 Evaluations and Finding Workable Solutions
|
| Rakesh Mohan, Idaho Legislature, rmohan@ope.idaho.gov
|
|
It seems that everyone has an opinion about public education (K-12); often those opinions are charged with strong emotions. Key stakeholder groups are generally well-organized and are not shy about letting their views be heard in public and lobbying for their interests with state legislatures behind the scene. Such an environment presents both challenges and opportunities for evaluators working in state legislative organizations. The two toughest challenges they face are understanding the political context and building a working relationship with policymakers and stakeholders through mutual respect and trust. Once these two challenges are mastered, opportunities lie in conducting evaluations that would produce workable solutions for improving the K-12 public education system. This presentation will use examples from nine K-12 evaluations conducted in Idaho during a six-year period. These evaluations covered a broad range of topics including virtual schools, pupil transportation, school district services consolidation, and adequacy of public education funding.
| |
|
Miracle or Madness: Understanding and Managing the Dynamics of Evaluation in Texas' K-12 Public Education
|
| Maria Whitsett, Moak, Casey and Associates, mwhitsett@moakcasey.com
|
|
Parameters for educational evaluation and accountability exist in the form of law and rule at the federal and state levels. These drive district policies and practices, including gathering and managing a vast array of data on students to feed into accountability systems that typically have predetermined performance measures with targets or standards. In a world dominated by tweets, local evaluation efforts seldom garner the same bursts of public attention as the publication of standards met or missed in educational accountability. Two aspects of accountability seem to particularly distress K-12 systems, beyond the test scores, per se: 1) instability in the measures and/or standards used to determine results and 2) administration of sanctions relative to performance. These issues strike at the heart of perceived fairness to the accountability system, and they must be addressed in a political environment. State and local evaluators can and should contribute to possible resolution and/or management of such issues in politically charged contexts.
| |
|
Session Title: Reviewing Three Statistical Techniques and Their Applications for Evaluation Research: Using Propensity Score Matching (PSM), Hierarchical Linear Models (HLM), and Missing Data Techniques
|
|
Multipaper Session 630 to be held in Sebastian Section I2 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Allan Porowski, ICF International, aporowski@icfi.com
|
| Discussant(s): |
| John Hitchcock, Ohio University, jhitchcoc@ohio.edu
|
| Abstract:
Evaluation research is often only as good as the data. That said, this multi-paper session introduces the use of three statistical techniques that provide alternatives when traditional experimental studies break down. These techniques also help evaluators to tame bad or incomplete data. Propensity Score Matching (PSM) can help evaluators to create an equivalent control groups in situations where they do not have natural control groups. Hierarchical Linear Models (HLM) help evaluators overcome issues of dependence and allow evaluators to examine their subjects while controlling for group level membership (e.g., schools, neighborhoods). Finally, missing data techniques can help to preserve data in cases where records would be lost while also promoting statistical power. These statistical techniques are commonly used in evaluation research today, and the presenters will focus on the benefits and drawbacks of each technique in examples of evaluation research.
|
|
The Use of Several Propensity Score Matching Techniques in the Evaluation of Educational Interventions
|
| Aikaterini Passa, ICF International, apassa@icfi.com
|
| Jing Sun, ICF International, jsun@icfi.com
|
|
An evaluation for the Communities In Schools (CIS) program in Texas implemented two types of propensity score matching techniques for constructing comparison groups. These methods allowed us to conduct a highly rigorous study at the school and the student level, respectively. In this evaluation, we sought to quantify the impact of the Communities in Schools' network in Texas on several academic and behavioral outcomes across elementary, middle and high schools by (a) matching CIS schools with other schools on several school characteristics using optimal matching, and (b) matching students that were exposed to CIS model to comparable students that did not, combining nearest-neighbor and exact matching on 12 student characteristics. This presentation will provide an overview of our methods and the implications for future research on school-based interventions for at-risk students.
|
|
Examining School-Based Intervention Programs in a Multilevel Context: Using Hierarchical Linear Modeling (HLM) in Evaluation Research
|
| Frances Burden, ICF International, fburden@icfi.com
|
| Kazuaki Uekawa, ICF International, kuekawa@icfi.com
|
|
This presentation examines student outcomes in students enrolled in Communities In Schools (CIS) in Texas using hierarchical linear models (HLM). HLM is a statistical set of models that enables one to understand student level effects, while controlling for the different contexts of schools. Using the CIS of Texas evaluation, this presentation focuses on several HLM models, including matched student comparisons, CIS-only student models, and growth curve models. This presentation compares and contrasts the benefits and weaknesses of these three types of models, their outcomes, and discusses the different insights each model provided for the CIS evaluation.
|
|
The Application of Missing Data Techniques in a School-based Intervention Evaluation: Evaluating Multiple Imputation, Maximum Likelihood and Listwise Deletion in an RCT context
|
| John Hitchcock, Ohio University, jhitchcoc@ohio.edu
|
| Frances Burden, ICF International, fburden@icfi.com
|
| Kelle Basta, ICF International, kbasta@icfi.com
|
|
In the course of completing a school-based intervention evaluation for the National Institute of Justice, it became necessary to address the substantial levels of missing data across a lengthy student survey. Although the response rate for students was high, approximately 40 percent of the data would have been lost through standard methods of deleting any student records that were not complete (i.e., listwise deletion). Deleting 40 percent of the respondents from the final analyses would have considerably reduced power and threatened internal and external validity; therefore, both multiple imputation and maximum likelihood missing data techniques were applied to the data to conserve the number of respondents. This presentation focuses on the merits and drawbacks of each of these missing data techniques and compares multiple imputation, maximum likelihood, and listwise deletion in the context of this school-based evaluation.
|
|
Session Title: Adaptation to Local Contexts
|
|
Multipaper Session 631 to be held in Sebastian Section I3 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG
|
| Chair(s): |
| Martha Ann Carey,
Maverick Solutions, marthaann123@sbcglobal.net
|
|
The Forest and the Trees: Evaluating Complex and Heterogeneous Initiatives
|
| Presenter(s):
|
| Shimon Spiro, Tel Aviv University, shispi@post.tau.ac.il
|
| Anastasia Gorodzeisky, Juan March Institute, agorodzeisky@ceacs.march.es
|
| Abstract:
This paper discusses the difficulties encountered in evaluating large scale, complex initiatives, directed at one or more geographical areas, and comprised of many distinct and unique programs. Following a concise review of the literature, we present an approach which attempts to evaluate the forest through the trees. This approach was applied to the evaluation of Tel Aviv University's "Price-Brody Initiative in Jaffa". As part of this initiative, various departments of the University offered dozens of different programs to the population of Tel Aviv's poorest area. We developed a set of instruments that were applied to the evaluation of each program separately. These instruments were sufficiently standardized to allow for comparisons between program, and for an evaluation of the initiative as a whole. We monitored the implementation of the evaluation, and elicited feedback from stakeholders. The paper concludes with a discussion of the potential and limitations of this methodology.
|
|
Consistent Protocol, Unique Sites: Seeking Cultural Competence in a Multisite Evaluation
|
| Presenter(s):
|
| Carolyn Sullins, Western Michigan University, carolyn.sullins@wmich.edu
|
| Ladel Lewis, Western Michigan University, ladel_lewis@yahoo.com
|
| Abstract:
Evaluating one site of a federally funded, multi-site initiative to improve services for children with mental health issues and their families presents numerous challenges. Perspectives of all consumers must be heard, understood, and acted upon, but many people are understandably reluctant to participate in an evaluation concerning such sensitive issues. Further, not all the sites fit neatly into the same 'one size fits all' evaluation protocol that must be used at all the sites. Cultural competence is crucial regarding: (1) breaking the barriers to participation; (2) balancing the traditional perspectives of 'informed consent' and 'confidentiality' with those of the participants; (3) balancing the need for consistent measures in our national study with the local realities of our participants; (4) interpreting and reporting the results. Seeking input from stakeholders at each step of the evaluation helps evaluators recognize and overcome these barriers.
|
|
Findings From the National Evaluation of Systems Transformation (ST) Grantees: Contextual Factors That Influence Grantees' Efforts to Change Community Long-term Support Service Delivery Systems
|
| Presenter(s):
|
| Yvonne Abel, Abt Associates Inc, yvonne_abel@abtassoc.com
|
| Deborah Walker, Abt Associates Inc, deborah_walker@abtassoc.com
|
| Meridith Eastman, Abt Associates Inc, meridith_eastman@abtassoc.com
|
| Abstract:
Between FY2005 and FY2006, the Centers for Medicare and Medicaid Services (CMS) awarded Systems Transformation (ST) Grants to 18 states to support states' efforts to transform their infrastructure to promote more coordinated and integrated long-term care and support systems that serve individuals with disabilities of all ages.
With more than half of the ST Grantees' five-year grant periods completed, findings from the national evaluation reveal that a common set of contextual factors (e.g., leadership support) can positively and negatively influence grant implementation. Within the current economic environment, we present how the role of these factors may be changing and how budget deficits and fiscal constraints are affecting specific strategies and outcomes of the grant. A factor of particular interest is how grantees are integrating with other grant initiatives active in their states (e.g., Money Follows the Person) to augment their ability to achieve and sustain grant goals.
|
|
The Impact of Context on Data Collection: A Comparison of Two Sites
|
| Presenter(s):
|
| Amy Orange, University of Virginia, ao2b@virginia.edu
|
| Walt Heinecke, University of Virginia, heinecke@virginia.edu
|
| Edward Berger, University of Virginia, berger@virginia.edu
|
| Chuck Krousgrill, Purdue University, krousgri@purdue.edu
|
| Borjana Mikic, Smith College, bmikic@smith.edu
|
| Dane Quinn, University of Akron, quinn@akron.edu
|
| Abstract:
This paper explores the difficulties of data collection due to site contexts. Two sites from a multisite study will be contrasted: one site has a high level of student involvement in class, through the course blog, and in focus group participation while the other has little student involvement in these areas. Focus groups conducted at each site had different outcomes. At the first site, multiple invitations were extended to students with a range of available times. There was no student response to the invitations. At the second site, with short notice and two invitations to participate, many students came to share their viewpoints with the evaluator. The varying levels of student participation have created difficulties in the evaluation, which seeks to understand how technologies used in the course impact students' learning and collaboration. We provide illustrative study results to highlight some issues that may arise when context impacts data collection.
|
| | | |
|
Session Title: The Historical Context of Evaluation: The Forgotten Post-war Years
|
|
Multipaper Session 632 to be held in Sebastian Section I4 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Research on Evaluation TIG
|
| Chair(s): |
| John Gargani, Gargani + Company, john@gcoinc.com
|
| Abstract:
In this presentation, we set out to broaden our understanding of the historical context for evaluation by presenting new scholarship related to the field's development in the years following World War II. From a US perspective, this was a period of rapidly expanding research institutions and enormous international aid efforts, both of which accelerated the growth of the field. From an international perspective, it was an unprecedented period of exchange as social scientists began crossing disciplinary and national boundaries in large numbers. However, mounting fears of communism, internationalism, and radicalism tempered that exchange and forced many evaluators to pay a heavy price for their pioneering work. In this presentation we will provide an overview of the period, introduce a transnational framework for analyzing the historical development of evaluation, and offer three historical case studies of pioneering evaluators in the post-war years.
|
|
Evaluation Theories, Evaluation History, and Theories of Evaluation History
|
| Christina Christie, Claremont Graduate University, tina.christie@cgu.edu
|
|
Much of the work related to the history evaluation has been undertaken to shed light on the development of evaluation theories. This work has natural boundaries, namely individuals, institutions, and disciplines, that provide a context for investigation and analysis. In considering the development of evaluation as a field, however, do we benefit from expanding that context? In this presentation I will consider what might be called the "traditional" view of post-war evaluation, describe a competing "transnational" perspective for historical research, and critique both. My purpose is to raise questions about how we collectively arrive at an understanding of our field and offer suggestions on how we can enrich that understanding.
|
|
J. Thomas Hastings (1911-1992): Consequential Validity Before its Time
|
| Gabriel Della-Piana, University of Utah, dellapiana@aol.com
|
| Connie Kubo Della-Piana, National Science Foundation, cdellapi@nsf.gov
|
|
We present a historical case study of J. Thomas Hastings, whose work in the 1950s presaged later debates by evaluation theorists on topics such as theory-based evaluation and consequential validity. An analysis of interviews, presentations, publications, and other documents is used to shed light on the contexts around which these ideas emerged, were taught, and more broadly disseminated. These contexts include courses, disciplinary professional organizations, the measurement community, and collegial individual and group dialogue. His personal manner and his professional perspectives were intrinsically linked, and through a variety of means became embedded in his students, the Center for Instructional Research and Curriculum Evaluation (CIRCE) that he helped found, and the field as a whole.
|
|
Jack Elinson, Laszlo Radvanyi, and the FBI
|
| John Gargani, Gargani + Company, john@gcoinc.com
|
|
Historical case studies of Jack Elinson and Laszlo Radvanyi, pioneers of evaluation who are little known to our field, are presented as windows into the post-war development of evaluation and the ways in which the state sought to control it through coercive means. Elinson, a major figure in the field of medical sociology, partnered with Edward Suchman in groundbreaking research and teaching that led to the latter's seminal text Evaluative Research. Radvanyi founded the first national survey research center in Mexico and conducted one of the earliest evaluations of a UNESCO funded aid project. The way in which these two men crossed paths by crossing disciplinary and national boundaries was indicative of the field at that time. It was also considered suspicious by colleagues, journalists, and governments, which led to FBI surveillance of both men and a heavy price paid in their professional and personal lives.
|
|
Session Title: Evaluating Emergency Preparedness
|
|
Multipaper Session 634 to be held in Sebastian Section L2 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Disaster and Emergency Management Evaluation TIG
|
| Chair(s): |
| Liesel Ritchie,
University of Colorado at Boulder, liesel.ritchie@colorado.edu
|
|
Multiple-perspective Self-report Surveying: A Simple Technique for Improving Accuracy in Self-report Based Emergency Preparedness Competency Assessment
|
| Presenter(s):
|
| Lisle Hites, University of Arizona, lhites@email.arizona.edu
|
| Jessica Wakelee, University of Arizona, wakeleej@email.arizona.edu
|
| Abstract:
With the exception of Homeland Security's Exercise Evaluation Program, the primary tool utilized for assessing Public Health Emergency Preparedness is self-report surveys. This session will begin by discussing the weaknesses inherent in self-report surveying, followed by a discussion of alternatives to self-report surveying. Keeping the previously mentioned limitations in mind, the session will continue with a discussion of improving self-report measurement of Emergency Preparedness and will present an approach that integrates components of 360 degree feedback to help improve accuracy of self-report surveys. Following the discussion of this approach, results will be presented from an application of this methodology for a Project Public Health Ready needs assessment.
|
|
Getting to Outcomes for Emergency Preparedness: A Journey
|
| Presenter(s):
|
| Karen Pendleton, University of South Carolina, ktpendl@mailbox.sc.edu
|
| Victoria Chien, University of South Carolina, victoria.chien@gmail.com
|
| Duncan Meyers, University of South Carolina, meyersd@mailbox.sc.edu
|
| Abstract:
Federal agencies have developed National Preparedness Guidelines for emergency preparedness which are primarily a blend of military and business planning models. Because emergencies occur at the local level, it is important that the national guidance be translated into an organized, practitioner-friendly guide. Getting To Outcomes for Emergency Preparedness (GTO-EP) - a guide which translates these Federal guidelines for local public health and other stakeholders - has been created for use within the context of team-based preparedness trainings facilitated by the University of South Carolina Center for Public Health Preparedness. GTO-EP has been continuously revised and refined over three generations of team-based trainings, and the current version is being utilized for preparedness in a diverse range of areas (e.g., mass fatality planning, environmental surveillance, hospital evacuation planning). Presenters will discuss: (1) the updated GTO-EP; (2) its role in the training; and (3) why GTO-EP is a vital tool for emergency preparedness planning.
|
|
Assessing Disaster Preparedness Among Community-Based Organizations in the San Francisco Bay Area
|
| Presenter(s):
|
| Liesel Ritchie, University of Colorado at Boulder, liesel.ritchie@colorado.edu
|
| Brandi Gilbert, University of Colorado at Boulder, brandi.gilbert@colorado.edu
|
| Abstract:
This paper explores the Bay Area Preparedness Initiative (BayPrep) project, an evaluation of the disaster preparedness of 90 community- and faith-based organizations in the San Francisco Bay Area. The goal of the project was to evaluate readiness and response capabilities of organizations that serve the most vulnerable populations of the city, such as racial and ethnic minority groups, the elderly, people living in poverty, persons suffering from chronic illness, and people with disabilities. Despite the important roles that they fill in assisting vulnerable communities in the event of a disaster, there has been little evaluation focusing on such organizations. The evaluation involved development and administration of a disaster preparedness checklist, as well as structured face-to-face interviews. Findings from this study indicate that community-based organizations are under prepared for disasters and that the next major disaster will cause serious strains on the organizations, causing vulnerable populations to suffer greatly.
|
| | |
|
Session Title: Building Evaluation Capacity, Learning and Use Through Communities of Practice
|
|
Think Tank Session 635 to be held in Sebastian Section L3 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Organizational Learning and Evaluation Capacity Building TIG
|
| Presenter(s):
|
| Beverly Parsons, InSites, bparsons@insites.org
|
| Leslie Goodyear, National Science Foundation, lgoodyea@nsf.gov
|
| Discussant(s):
|
| Patricia Jessup, Independent Consultant, pjessup@insites.org
|
| Mallary Tytel, Healthy Workplaces, mtytel@healthyworkplaces.com
|
| Alyssa Na'im, Education Development Center Inc, anaim@edc.org
|
| Abstract:
The session explores ways of defining and using communities of practice (CoPs) to build evaluation capacity, learning, and use when carrying out evaluations of complex change initiatives in complex contexts. Examples of three types of CoPs will be provided to stimulate thinking about how to apply the concept of CoPs to one's own evaluation work: (1) CoPs within an organization (with a supporting internal infrastructure) designed to integrate inquiry with learning and practice; (2) Multi-organizational CoPs that engage in evaluative inquiry with a supporting external structure but no supporting structure within the individual organizations; (3) CoPs with varying degrees of emphasis on learning, inquiry and/or practice with both internal and external support structures. Through the presentations and facilitated dialogue, participants will explore the possibilities for using CoPs in their own evaluation work.
|
|
Session Title: Evaluation Theories Grounded in Social Justice
|
|
Panel Session 636 to be held in Sebastian Section L4 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Theories of Evaluation TIG
|
| Chair(s): |
| Donna Mertens, Gallaudet University, donna.mertens@gallaudet.edu
|
| Discussant(s):
|
| Marvin Alkin, University of California Los Angeles, alkin@gseis.ucla.edu
|
| Abstract:
Alkin's Evaluation Theory Tree provides a graphic depiction of the diverse theories that contribute to our understanding of theoretical groundings in evaluation. His tree has received both praise and criticism: Praise because it captures the US version of theory up to a point; criticized because its original form primarily reflected the white, male roots of evaluation theory. Alkin subsequently revised the tree to be more inclusive of theories related to social justice. This panel will provide additional possibilities for inclusion in the theoretical tree by examining theoretical frameworks that grounded in social justice and hence promote a salient aspect of ethics to evaluation theory. The panel presentations will include Mertens' transformative paradigm and its commensurate theories, Critical Theory and implications for furthering social justice (Thomas), Indigenous Theory and implications for social justice in the Maori community (Cram), and theory-based evaluations in marginalized communities (Bledsoe). Marv Alkin will serve as discussant.
|
|
Evaluation Theories Grounded in Social Justice: A Transformative View
|
| Donna Mertens, Gallaudet University, donna.mertens@gallaudet.edu
|
|
The transformative paradigm is a metaphysical framework that is defined by a set of basic belief systems that prioritize the promotion of social justice and the furtherance of human rights. Theoretical frameworks that are commensurate with the transformative paradigm include feminist theories, critical race theory, disability theory, queer theory, and indigenous theories. This presentation will focus on the underlying beliefs of the transformative paradigm and how they support the use of theoretical perspectives that are grounded in social justice for evaluation. This presentation will provide an overall framework in which to situate the presentations that follow. Mertens brings a lengthy history of work in marginalized communities that is evidenced in her scholarly work related to the transformative paradigm: Transformative Research and Evaluation (Mertens, 2009) and the book she co-edited with Pauline Ginsberg (Handbook of Social Research Ethics (2009).
|
|
|
Grounding Evaluation Theories in Social Justice: A Critical Theory View - Defining Moments
|
| Veronica Thomas, Howard University, vthomas@howard.edu
|
|
"Defining moments" are those critical periods that penetrate our consciousness and provide us with the opportunity to change our filter and look at things differently. As evaluators we are confronted with an ongoing series of "defining moments" of choices and decisions. It is during these periods that we have an opportunity to define and redefine our ethical responsibilities in dealing with issues such as racism, sexism, classism, and other forms of oppression; the hierarchical relationship between evaluators and stakeholders; considerations of what counts as "knowledge"; and strategies for advancing equity and social justice through evaluative practice. This presentation will discuss Critical Theory (CT) as part of a larger transformative paradigm that can guide evaluators through those "defining moments" and offer a framework for ethical decision-making that better identifies interests, harms, and benefits, with specific examples illustrative of how evaluators can promote equity, democracy, and social justice.
| |
|
Grounding Evaluation Theories in Social Justice: An Indigenous View
|
| Fiona Cram, Katoa Ltd, fionac@katoa.net.nz
|
|
For the past nine years in Aotearoa New Zealand, the policy context for evaluation has stressed the importance of social inclusion to reduce inequalities and create a more just society. Evaluators of government-funded programs should have, on first glance, been delighted to have been part of this agenda. However, tensions between 'inside' and 'outside' ethics exist. 'Inside' ethics ensures that all stakeholders are treated with respect - a crucial perspective for Maori (indigenous) evaluators because the production of knowledge within Maori society is performative; i.e., the way knowledge is gathered is as important as the knowledge itself. 'Outside' ethics is about being politically astute, knowing that Maori are often marginalised by policies bearing social justice labels. Maori evaluators face this challenge: How to critique a social justice 'look' that is unjust for Maori while maintaining credibility as an evaluator in the eyes of both the government and the community?
| |
|
Grounding Evaluation Theories in Social Justice: A Community-Based View
|
| Katrina Bledsoe, Walter R McDonald and Associates Inc, katrina.bledsoe@gmail.com
|
|
The dynamic demographics of communities make it imperative to understand how to conduct evaluations that are sensitive to the context of participants situated in community-based settings. This understanding is especially important in working with communities with hard to reach populations (e.g., marginalized groups), and where evaluations, if not tailored to communities' context, can easily overlook the mores of its members and violate the rights of participants. The presentation will examine theoretical perspectives that facilitate the examination of issues that might arise in evaluative settings between evaluators and key stakeholders that impact the development and execution of ethical and "contextually competent" evaluations (e.g., power dynamics; institutional structures that intentionally or unintentionally promote the "isms"). The presenter discusses these concerns within the context of a social justice framework addressing theoretical issues, as well as ethics; methodology; translation of evaluation terminology; and using strategies to garner community involvement, buy-in, and trust.
| |
|
Session Title: Building Consensus on Nationwide Outcomes and Indicators for Extension
|
|
Panel Session 637 to be held in Suwannee 11 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Extension Education Evaluation TIG
|
| Chair(s): |
| Bart Hewitt, United States Department of Agriculture, bhewitt@csrees.usda.gov
|
| Discussant(s):
|
| Nancy Franz, Virginia Cooperative Extension, nfranz@vt.edu
|
| Abstract:
With today's budgets at both the Federal and State levels being tight there is much competition among Federal agencies and State Land-Grant Universities for dollars. Congress and the White House budget decisions makers are looking for evidence of performance and public value for the dollars appropriated to Federal agencies. Extension has provided evidence of individual State performance and public value, but there is a lack of evidence of performance and public value that can be aggregated to tell a national story. This panel presents both the Federal and State partner perspectives on the need, the value, and the feasibility of developing national outcomes and indicators to provide the evidence Congress, the White House, and other decision makers need to assess the national value of Extension to its citizens. A discussion will explore the next steps evaluators and administration can take to develop national outcomes and indicators and the commitment necessary.
|
|
Federal Partner Data Needs for Reporting National Outcomes
|
| Bart Hewitt, United States Department of Agriculture, bhewitt@csrees.usda.gov
|
|
This presentation describes the Federal Partner budget process with emphasis on reports on evidence of performance and public value to Congress, the Program Assessment Rating Tool (PART), Office of Management and Budget, the agency Budget Office, and USDA. The agency performance-based budget cycle will be discussed. Examples of outcomes used in these reports will be shown.
|
|
|
A National Program Leader's Perspective on Developing National Outcomes and Indicators: Environment and Natural Resources- Renewable Resources Extension Act
|
| Eric Norland, United States Department of Agriculture, enorland@csrees.usda.gov
|
|
Ineffective accountability and broad discretion for program directions are two issues that are often cited as reasons for which formula funds are under constant scrutiny. The Renewable Resources Extension Act (RREA) program has overcome both of those issues. Through a strategic planning process where the state and federal partners "came to the table" with no agenda to push, the partners agreed and commited to a specfic set of strategic issues and a reporting process that included quantitative and narrative information. With facilitation and evaluation expertise from an Extension Evaluation Specialist, national level indicators were developed and adopted.This is one of the few formula programs at CSREES where both partners have agreed to limit program scope to an identified set of issues.
| |
|
A National Program Leader's Perspective on Developing National Outcomes and Indicators: 4-H and Youth
|
| Suzanne LeMenestrel, United States Department of Agriculture, slemenestrel@csrees.usda.gov
|
|
This presentation will provide an overview of efforts that National 4-H Headquarters has engaged in to develop national outcomes for 4-H. These include participation in the PART process, as well as specific examples from the work of the Science, Engineering, and Technology mission mandate (SET) evaluation team. Specifically, the presentation will describe the process of developing a national logic model for SET, identification of and measurement of indicators at the national level, and how state 4-H programs are using these resources.
| |
|
A Land-Grant University Perspective on Developing National Outcomes and Indicators: University of New Hampshire
|
| Lisa Townson, University of New Hampshire, lisa.townson@unh.edu
|
|
Extension programs vary from state-to-state, based on local needs and available resources. Although all states are required to submit federal plans and reports in a similar format, there are important differences in accountability systems and program evaluation support available to staff. While there is a need for flexibility in program planning and the selection of state outcome indicators, state Extension systems would benefit from the ability to select from a menu of common national indicators, particularly those with well-designed evaluation plans/tools.
Such an effort would provide program flexibility yet assist states/regions in prioritizing their evaluation efforts, focusing on fewer, well-defined metrics that are important locally and nationally. It would also facilitate collecting and reporting more credible data that is not only useful to share with local stakeholders, but also to aggregate regionally and nationally to communicate program impacts and public value.
| |
|
Session Title: Hearing Silenced Voices: Using Visual Methods to Include Traditionally Disenfranchised Populations in Evaluation
|
|
Skill-Building Workshop 638 to be held in Suwannee 12 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Special Needs Populations TIG
|
| Presenter(s):
|
| Linda E Lee, Proactive Information Services Inc, linda@proactive.mb.ca
|
| Denise Belanger, Proactive Information Services Inc, denise@proactive.mb.ca
|
| Larry K Bremner, Proactive Information Services Inc, larry@proactive.mb.ca
|
| Abstract:
While evaluators understand the importance of multiple stakeholder perspectives, many struggle with how to ensure the participation of those traditionally 'without a voice,' vulnerable or disenfranchised populations. Children and youth, persons with disabilities, or those having emerging literacy in the majority language(s) hold important views regarding the programs, services, and situations which affect them, but their perspectives are not always included. This workshop will be grounded in theory, but will be high participatory and interactive. Stemming from a rights-based approach, the workshop will explore the why and how of including traditionally disenfranchised populations in evaluation. Through their work in Canada and numerous countries in Europe, the facilitators will share a variety of visually based techniques and how these can be coupled with more traditional methods. Participants will be introduced to the use of drawing, icons, photography, and mapping, as tools for eliciting often-silenced voices. Ethical considerations will also be discussed.
|
|
Session Title: Evaluating United States Department of Education and National Science Foundation Programs for Improving Mathematics and Science Education: Rigor, Relevance, Context, and Challenges
|
|
Panel Session 639 to be held in Suwannee 13 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Robert Yin, COSMOS Corporation, ryin@cosmoscorp.com
|
| Abstract:
The U.S. faces the challenge of increasing its competitiveness in science, technology, engineering, and mathematics (STEM) fields. Federal initiatives, including the National Science Foundation's Math and Science Partnership Program and the U.S. Department of Education Math and Science Partnerships funded under Title IIB of No Child Left Behind, focus on teacher professional development as a means to improve student outcomes in mathematics and science. The panel for this session is comprised of educational researchers who lead national and local evaluations of these programs. Panelists will present evaluation designs, methodologies, and findings from local and national evaluations of these programs. The session will provide an opportunity to discuss issues related to these complex evaluations, including: 1) ensuring rigor and relevance of project and program evaluations; 2) recognizing contextual factors that influence implementation and outcomes; 3) addressing challenges encountered in current evaluation approaches; and 4) identifying needs for future evaluation.
|
|
Comparing Academic Performance between Program Participating Schools and Non-Program Participating Schools
|
| Kenneth Wong, Brown University, kenneth_wong@brown.edu
|
| Megan Boben, Brown University, megan_boben@brown
|
|
Study leaders will present information about this project that examines academic performance between schools that participate in the National Science Foundation's Math and Science Partnership (NSF MSP) Program and their non-participating peers. Because NSF MSP activities primarily involve teacher training and professional development in multiple grade levels, we examine school-level achievement. We address the question: When schools in a state participate in the NSF MSP Program, do their students perform better than they would have if they had not participated in the NSF MSP Program? Student achievement is measured in terms of performance on state-administered assessments in mathematics and science for specific grades for five years in a sample of over 800 schools in 9 math and science partnerships across 6 states. To control for a number of demographic variables, we employ the Mahalanobis distance matching to define an appropriate comparison school group before conducting our multivariate analysis.
|
|
|
Teacher and Student Outcomes in Nebraska's Math in the Middle Institute Partnership
|
| Stephen Meyer, RMC Research Corporation, meyer@rmcdenver.com
|
| John Sutton, RMC Research Corporation, sutton@rmcdenver.com
|
|
The Math in the Middle Institute Partnership is designed to improve student mathematics achievement by creating sustainable partnerships among individuals at the University of Nebraska-Lincoln, regional Educational Service Units (ESUs), and local school districts. Partnerships are designed to educate and support teams of outstanding mathematics teachers in grades 5-8 who become intellectual leaders in their schools, districts, and ESUs. RMC Research's quasi-experimental evaluation compares outcomes for M2 participants and their students to those of teachers and students in a comparison group. Qualitative components include interviews and focus groups; observations; and analysis of documents. Results from analyses of teacher pre/post survey data and student mathematics achievement data will be presented by directors of the external evaluation. Methodological challenges and limitations will also be discussed, including attribution of program effects, conflicts between strong evaluation designs and program implementation goals, challenges identifying valid comparison groups, and data collection challenges in rural settings.
| |
|
Balancing the Call for Evidence-Based Research Designs With Formative Evaluation to Improve Implementation of Inquiry-Based Science Teaching
|
| Catherine Callow-Heusser, EndVision Research and Evaluation LLC, cheusser@endvision.net
|
| Kenneth Wareham, Lewis-Clark State College, klwareham@lcsu.edu
|
|
The Teaching for Excellence in Science and Literacy Achievement (TESLA) project (Idaho Mathematics-Science Partnership Grant) integrates science and literacy instruction, and provides ongoing support from university faculty to classroom teachers. The evaluation included both quantitative and qualitative evidence for formative program improvement and evaluation of program effectiveness. Gains in science and language scores in participating classrooms were large and statistically significant. More importantly, grade equivalent changes for science scores on the SAT-9T standardized test (a form of the Stanford Achievement Test designed to reduce student reliance on memorized factual information and emphasize unifying concepts and themes in science and literacy) were larger than 1-year expected gains. This presentation will include an overview of the TESLA program, evaluation, and findings. Project directors will discuss how the evaluation design was influenced by contextual factors including the focus on program improvement, faculty involvement in collecting evaluative data, and a small evaluation budget.
| |
|
Meta-Evaluation in Title IIB MSP Projects and Programs: What Counts as Standards of Quality?
|
| Edith Gummer, Northwest Regional Educational Laboratory, gummere@nwrel.org
|
| Judith Devine, Northwest Regional Educational Laboratory, devinej@nwrel.org
|
|
The Montana Title IIB MSP program evaluation focused on the nature of the partnerships and the quality of project evaluation for the six projects that made up the second cohort in Montana. The nature of the partnerships has been examined using a survey that was developed based on the work of Gordon Kingsley. Stakeholders were asked questions that provided a contextual profile of their partnership, including the motivation behind stakeholder involvement, the perceived need for the partnership, and the resources that helped to sustain their efforts. The survey provided evidence on the extent to which stakeholders perceived that a collaborative model of professional development has been effective in achieving program goals. The quality of the evaluations was assessed using national guidelines and standards (i.e., Guskey, 2000; Joint Committee on Standards for Educational Evaluation, 2008). Project leaders will present results and discuss their implications.
| |
|
Session Title: Evaluating Educational Materials in the Profit-Making World: Issues Associated With Evaluating Curricular Materials for Textbook Publishers
|
|
Panel Session 640 to be held in Suwannee 14 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Jerry Jordan, University of Cincinnati, jordanjm@uc.edu
|
| Abstract:
Decisions about the adoption of curriculum materials in schools have significant consequences. For teachers and children, these decisions dictate the basic materials teachers use to teach our children. These adoptions also represent huge financial transactions as districts, or states, invest millions of dollars in materials. Recently, decision makers in education have become more research driven in their decisions about the adoption of curriculum materials. Text book publishers have responded to this environment by sponsoring formal evaluations of the materials they promote to schools. These evaluations have substantial social and economic implications. This panel explores the issues that emerge when evaluators assess the effectiveness of educational materials in collaboration with the publishers who create and market those materials. Panelists will explicate the specifics of this context and highlight how evaluators can manage the unique challenges and opportunities inherent in conducting evaluations in the high stakes context of the "for-profit" publishing world.
|
|
The Bottom Line in the For-Profit World: How the Publishing Industry Shapes the Context of Curriculum Evaluations
|
| Stephanie Baird Wilkerson, Magnolia Consulting LLC, stephanie@magnoliaconsulting.org
|
|
This presentation will focus on how the education publishing industry shapes the context of curriculum efficacy evaluations. The presentation will examine the context of evaluation across various dimensions including purpose, focus, paradigm, organizational evaluation capacity, funding, communication, timeframe, organizational culture, political factors, stakes, and "the bottom line." Attendees will gain a better understanding of how the context of textbook evaluations is both unique and similar to general evaluation contexts. The presentation gives a global overview of the context of conducting textbook evaluations and sets the stage for following presentations to go into more depth in many of the highlighted areas. For over eight years, Stephanie Baird Wilkerson has been evaluating supplemental and basal curriculum materials for education publishers including Pearson Education, Houghton Mifflin Harcourt, National Geographic School Publishers, Northpoint Horizons, Study Island, Curriculum Associates, and EducationCity.com.
|
|
|
Different Points of View: The Fundamental Differences in How Publishers and Researchers See Evaluation Research
|
| Martin Saperstein, Saperstein Associates, msaperstein@sapersteinassociates.com
|
|
To some extent, the assessment of curriculum materials represents the collision of two different worlds: business and academics.-á The primary agents within each area often bring with them sharply constrasting sets of assumptions and goals. This presentation will focus on the most significant areas of potential misunderstanding and confusion. Differences range from very simple issues of vocabulary (e.g. the meaning of "signficant" findings) to deep-rooted differences in philosophies and ethics. Martin Saperstein has been president of Saperstein Associates for nearly 30 years and conducted dozens of studies for major book publishers. In this presentation he will share his experience regarding the major differences that affect collaborations between academic researchers and large publishing companies.
| |
|
Who Said it was Easy? Design Challenges and Considerations Associated With Evaluating the Effects of Educational Curricula
|
| Miriam Resendez, PRES Associates Inc, mresendez@presassociates.com
|
| Mariam Azin, PRES Associates Inc, mazin@presassociates.com
|
|
Presenters, having conducted numerous large-scale experimental and quasi-experimental evaluations examining the effects of various educational curricula on student learning, will discuss the unique design challenges and considerations associated with conducting these types of studies. Design topics that may be addressed include, but are not limited to: a) site recruitment; b) unique issues associated with conducting evaluations of core curricula (rather than supplemental programs); c) implementation monitoring; d) measurement of extraneous variables that can influence study findings (such as teacher quality); e) selection of outcome measures; and f) strategies for enhancing the power and sensitivity of evaluations in detecting any treatment effects.
| |
|
Good News and Bad News: Representing and Reporting Mixed Evaluation Results in the High-stakes World Of Textbook Publishing
|
| Rebecca Eddy, Claremont Graduate University, rebecca.eddy@cgu.edu
|
|
Preskill (2005) and others have provided guidelines about how evaluators should report evaluation results to make them most useful to stakeholders. However, evaluators have to balance competing interests when reporting on evaluations of educational curricula. For example, evaluators must work to meet rigorous design standards set forth by the What Works Clearinghouse (WWC), and simultaneously make the evaluation findings interpretable enough for the client (textbook publisher) and the client's stakeholders (states, districts, and schools). In addition, evaluators often have the responsibility of making the results of advanced statistical analyses interpretable to consumers that accurately conveys implications of these findings. Textbook publishing is not unlike most other areas, where clients are biased to report only positive results and evaluators must resist this temptation. Evaluators should take advantage of their unique role in the publishing world and advocate a strategy where evaluation findings influence product development.
| |
|
Empowering the Consumers of Curriculum Efficacy Studies: How Can We Help the Schools Interpret Our Research?
|
| Jerry Jordan, University of Cincinnati, jordanjm@uc.edu
|
|
The ultimate users of assessments of curriculum materials, teachers and administrators, often are not experienced consumers of research. Anecdotal evidence implies that some decision-makers respond to superficial characteristics of reports and may misunderstand primary claims. In this presentation, three direct actions are suggested to help with the decisions made at the grass roots level: 1) Research should be conducted from the perspective of the teachers to discern which dimensions of the materials are most consequential to them. 2) School PD activities should include education on how to interpret rigorous evaluations of curriculum materials. 3) Consultants with evaluation experience should reach out to schools and offer their expertise in interpreting the implications of the research presented to the schools. Jerry Jordan is a research associate in the Evaluation Services Center at the University of Cincinnati. Most recently, he has conducted several field tests of curriculum materials for Macmillan/McGraw-Hill and Glencoe Publishers.
| |
|
Session Title: Issues in Measuring Students' Interest in Science, Technology, Engineering, Mathematics and Related Careers: Using Multiple Data Sources in Evaluations of National Science Foundation Innovative Technology Experiences for Students and Teachers (ITEST) Programs
|
|
Multipaper Session 641 to be held in Suwannee 15 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Torie Gorges, SRI International, torie.gorges@sri.com
|
| Abstract:
This session offers perspectives and strategies used in evaluations of five National Science Foundation Innovative Experiences for Students and Teachers (ITEST) programs. ITEST programs seek to engage students from diverse backgrounds in science, technology, engineering, and mathematics (STEM) fields with the goal of increasing the numbers of youth who seek out careers in these areas. In each of these programs, researchers have faced the challenge of evaluating students' changing interest in STEM and STEM careers. We have found that instruments such as surveys do not always capture subtle shifts. In this session, we will discuss the methods we have used to capture small changes and growing interests, including modifications to survey items and the addition of instruments such as parent focus groups, interviews with program staff, and observations. The session offers practical advice from the field to evaluators of educational programs, particularly those focused on STEM and STEM careers.
|
|
Measuring Subtle Shifts: Career Interests of Middle School Girls in a Technology Fluency Informal Education Program
|
| Torie Gorges, SRI International, torie.gorges@sri.com
|
| Melissa Koch, SRI International, melissa.koch@sri.com
|
| Reina Fujii, SRI International, reina.fujii@sri.com
|
|
Build IT is a technology fluency program for middle school girls developed by SRI International and Girls Inc. of Alameda County. One of the program's goals is to encourage participants to explore and pursue careers in STEM fields. To do this, girls create technologies and meet women working in STEM fields. One of the challenges for evaluators has been measuring subtle shifts in girls' interest in STEM and related careers. We found that staff who work with the girls daily saw more interest in STEM than we picked up via surveys and interviews. For example, students loved creating video games stuck to answers like "teacher" and "pediatrician" when asked by evaluators what careers interested them. In this paper, we discuss modifications to interview questions and additional data sources, such as staff feedback and observations, which enable us to better capture small shifts in career interests as well as big changes.
|
|
Tracking Students' Knowledge of and Interest in Science, Technology, Engineering, and Mathematics Careers
|
| Ann Howe, Independent Consultant, achowe@earthlink.net
|
|
The SUCCEED Apprenticeship Program enables middle school students of diverse racial and ethnic backgrounds and both genders to become proficient in the use of information technologies within the context of science, technology, engineering, and mathematics (STEM). One of the stated goals of the two-year program is to build and maintain excitement for STEM with the expectation that this will eventually lead to careers in one of these fields. We have tracked students' career knowledge and interests over the two-year span through surveys and repeated interviews of students and mentors. In addition to providing information about career interests these methods have also allowed us to form tentative conclusions about the elements of the project that tend to reinforce students' interest in STEM careers. Modification of our methods as three cohorts have moved through the program will be discussed.
|
|
Possible Selves and Career Goals: Multiple Ways of Exploring the Question "What Do You Want To Be When You Grow Up?"
|
| Karen L Yanowitz, Arkansas State University, kyanowitz@astate.edu
|
| Tanja McKay, Arkansas State University, tmckay@astate.edu
|
| C Ann Ross, Arkansas State University, cross@astate.edu
|
| Staria S Vanderpool, Arkansas State University, svand@astate.edu
|
|
CSI: Creating Student Investigators uses forensic science to engage middle and high school students in a week-long intensive STEM enrichment experience. This paper presents different ways of looking at STEM interests, career aspirations, and connections between these methods. Students generated a list of careers goals, and completed general survey regarding STEM interest and activities. Self-efficacy, or the belief that one's actions will produce desired outcomes, may influence aspirations and thus, students completed a science self-efficacy scale. Finally, students completed a possible self measure by describing (open-ended) what they thought would be most likely true about themselves in the future. Possible selves, part of general identity development, may encompass a variety of life goals. Information gained from the possible self measure thus can place career and STEM goals in context with other goals. Discussion will focus upon triangulating these measures and the information that each provides when considering student career development.
|
|
Examining Career Interest for Youth in the Detroit Area Pre-college Engineering Program
|
| Shannan McNair, Oakland University, mcnair@oakland.edu
|
| Margaret Tucker, Detroit Area Precollege Engineering Program, mtucker@dapcep.org
|
| Jason Lee, Detroit Area Precollege Engineering Program, jdlee@dapcep.org
|
|
Documenting the career interests of youth is important to precollege program goals but not as simple as it looks. Surprisingly, our challenges begin at the beginning, deciding how to categorize responses to an open ended-question about "what do you want to do as an adult?" or deciding what to include in a list of possible careers. Comprehensive lists do not always take into account all levels of STEM careers. For example, is designing computer software, the same as a job repairing personal computers? Also, newer career categories may not be included in existing lists. In addition, respondents may have the same career in mind, but call it by one name on a pre-program survey and another on a post-program survey. Challenges related to asking the question, documenting the response, categorizing the response and measuring change are discussed. Panel presenter will promote interaction by eliciting ideas for solutions to the issues.
|
|
Measuring Students' Science, Technology, Engineering and Math (STEM) Interest and Learning through Data Triangulation
|
| Karen Manuel, Puget Sound Center for Teaching, Learning and Technology, kmanuel@psctlt.org
|
|
There are many challenges in evaluating the impact of an after school or summer program on students' science, technology, engineering and math (STEM) learning and interest, as well as their future course and career choices. This paper will describe how TechREACH measures middle school students' changing career interests and interest in STEM in general, and triangulates data sources as a means of assessing program impact. The TechREACH evaluation employs a mixed methods approach, including pre- and post- student surveys, focus groups, teacher surveys and interviews, parent surveys, math/science teacher surveys, observation, grade collection, and information about students' subsequent involvement in STEM activities in high school. The evaluation examines interest and achievement in STEM coursework, proficiency in computer technology, knowledge of and interest in STEM degrees and careers, and likelihood to pursue future STEM courses and careers.
|
|
Session Title: Institutionalizing Evaluation and Grantmaking
|
|
Multipaper Session 642 to be held in Suwannee 16 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Chair(s): |
| Stephanie Evergreen,
Western Michigan University, stephanie.evergreen@wmich.edu
|
|
Trying to Do Evaluation 'Right', Right From the Start: The National Fish and Wildlife Foundation's Effort to Institutionalize Evaluation Into the Entire Grantmaking Lifecycle
|
| Presenter(s):
|
| Christina Kakoyannis, National Fish and Wildlife Foundation, christina.kakoyannis@nfwf.org
|
| Matthew Birnbaum, National Fish and Wildlife Foundation, matthew.birnbaum@nfwf.org
|
| Abstract:
To better demonstrate the environmental or social impacts of the organization's grantmaking investments, the National Fish and Wildlife Foundation provided additional staff and leadership support to embed a more comprehensive system of monitoring and evaluation into the entire lifecycle of the Foundation's new long-term initiatives. Now two years into the process, we reflect back on our effort to translate key principles from evaluation theory into practice within a conservation foundation. In this presentation, we provide insights regarding what worked and what did not in establishing an evaluation system to continuously learn from our grantmaking. We discuss key lessons from our experience, such as accounting for the disciplinary context, striking a balance between funder and grantee responsibilities, and understanding the trade-offs between the flexibility that accommodates specific grants and the standardization that allows for comparisons across multiple initiatives.
|
|
Planning for Evaluation of New Initiatives: Determining Evaluation Needs for the Tribal Tobacco Education and Policy Initiative
|
| Presenter(s):
|
| Linda Bosma, Bosma Consulting LLC, linda@bosmaconsulting.com
|
| Chris Matter, ClearWay Minnesota, cmatter@clearwaymn.org
|
| Jaime Martinez, ClearWay Minnesota, jmartinez@clearwaymn.org
|
| Nicole Toves, ClearWay Minnesota, ntoves@clearwaymn.org
|
| Joanne D'Silva, ClearWay Minnesota, jdsilva@clearwaymn.org
|
| Abstract:
Planning for evaluation is as essential as planning for program implementation, but funding agencies do not always have sufficient in house expertise to determine evaluation needs.
Recently, ClearWay MinnesotaSM began funding the Tribal Tobacco Education and Policy initiative (TTEP), and wanted an evaluation tailored to this unique policy initiative. Rather than issue a general request for evaluation proposals, staff enlisted an evaluator familiar with initiatives similar to TTEP, to help them determine the project's evaluation needs. The evaluator assisted staff in articulating the TTEP's theory of change and recommended an evaluation framework, then assisted in drafting the formal call for proposals and review and hiring of the evaluation team. The authors will present an overview of the TTEP initiative, the decision to seek evaluation consultation, and the process that foundation staff and the evaluator undertook that defined their evaluation needs, leading to successfully securing an evaluation firm.
|
|
Assessment of Shifting Grantmaking Strategies: Can Increased Standardization Work?
|
| Presenter(s):
|
| Anita Baker, OMG Center for Collaborative Learning, Anita@omgcenter.org
|
| Abstract:
Between July and September 2008, OMG Center for Collaborative Learning provided short-term grantmaking evaluation and technical assistance to a public Grantmaker committed to supporting nonprofit housing development organizations. The purpose of this evaluation was to help the Grantmaker obtain feedback and insights from key stakeholders regarding their recently modified grantmaking process, and to review all grantmaking decisions for consistent application of the new procedures. The modified grantmaking process reflected a major change in context for both the Grantmaker and grantees as they moved from a proposal-based strategy to a more uniform organizational underwriting approach. The evaluation results showed that pursuant to the goal of standardizing grantmaking, there was both perception of increased standardization and evidence of it in grant award distributions. The paper presents findings from the review as well as a discussion of methods and strategies used to collect and analyze data from multiple stakeholders and from grant proposals.
|
| | |
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Divisive Contexts and Their Impact on Evaluation |
|
Roundtable Presentation 643 to be held in Suwannee 17 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Human Services Evaluation TIG
|
| Presenter(s):
|
| Joanne Farley, Human Development Institute, joanne.farley@uky.edu
|
| Chithra Perumal, Human Development Institute, chithra.perumal@uky.edu
|
| Abstract:
Questions for roundtable discussion will be raised focusing on a two year evaluation of a community collaborative intended to enhance the academic performance and well-being of students in a particular school. The collaborative-school relationship experienced a number of issues and problems which impacted the ability of evaluators to conduct high quality evaluation. These issues generated problems for the evaluation in areas such as following the goals of the evaluation, collecting evaluation data, obtaining evaluating findings, developing valid interpretations of evaluation findings and reporting out evaluation results and implications. Since multi-group collaboratives can often have issues of leadership, representation, and agenda, focusing on this particular collaborative should initiate rich dialogue among roundtable participants. The purpose of this roundtable will be to encourage group discussion of how to resolve and deal with challenges to conducting a high quality evaluation. Participant dialogue will be informed by being given a history of the focal collaborative and discussion of the issues that operated in areas such as collaborative leadership; racial/ethnic divisions among constituents of the collaborative, e.g., students, teachers, and parents; and community commitment to the collaborative mission and goals.
|
| Roundtable Rotation II:
Methodological Successes and Challenges in Conducting Comprehensive Organizational Health Assessment in Child Welfare |
|
Roundtable Presentation 643 to be held in Suwannee 17 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Human Services Evaluation TIG
|
| Presenter(s):
|
| Sheridan Green, University of Denver, sheridan.green
|
| Robin Leake, University of Denver, robin.leake@du.edu
|
| Julie Morales, University of Denver, julie.morales@du.edu
|
| Cathryn Potter, University of Denver, cathryn.potter@du.edu
|
| Abstract:
Comprehensive organizational assessment in child welfare helps guide and effectively target intervention efforts. This proposed roundtable will begin by the University of Denver, Butler Institute for Families presenting their work in the development of a comprehensive organizational health assessment for child welfare; the selection of domains, methods and measures; the on-site assessment process; and the follow-up used by a regional, federally-funded child welfare workforce initiative. The evaluative use of the assessment to gauge the intervention and initiative-level effectiveness will also be presented. The encouraged discussion will focus on the successes and challenges of this approach and other evaluators' experiences, specifically in the context of human services and child welfare. Recommendations for modifications, refinement and further work will be solicited from the roundtable participants.
|
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
County Level Benchmarking: Helping Local Groups to Make the Most of Public Data |
|
Roundtable Presentation 644 to be held in Suwannee 18 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Organizational Learning and Evaluation Capacity Building TIG
|
| Presenter(s):
|
| Virginia Dick, University of Georgia, vdick@cviog.uga.edu
|
| Melinda Moore, University of Georgia, moore@cviog.uga.edu
|
| Abstract:
More local groups are becoming interested in utilizing public data for determining community issues and establishing and tracking community benchmarks. Public data (i.e. census data, educational data, public health data, etc.) can provide an important resource for community assessments and benchmarking. As groups address particular issues in individual counties, they often become interested in learning how to track progress. Many times these efforts do not include additional dollars to support the hiring of evaluation support and the community members are interested in learning how to utilize public data and track information on their own. With the increasing availability of data, open source software, and other resources, community groups can manage these tasks on their own with support and guidance. This roundtable will explore various sources of public data that can be accessed, methods for accessing and utilizing data, and some real life examples from communities.
|
| Roundtable Rotation II:
Dealing With Existing Data: The Benefits and Pitfalls |
|
Roundtable Presentation 644 to be held in Suwannee 18 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Organizational Learning and Evaluation Capacity Building TIG
|
| Presenter(s):
|
| Katie Daniels, Western Michigan University, knelsondaniels@gmail.com
|
| Kristin Everett, Western Michigan University, kristin.everett@wmich.edu
|
| Nakia James, Western Michigan University, nakia.s.james@wmich.edu
|
| Abstract:
Evaluators are often faced with questioning the extent to which it is appropriate to use existing data in the context of an evaluation. This Round Table discussion will address both the advantages and disadvantages of dealing with existing data, highlighting experiences from working with actual clients. This session will address how to make choices about existing data given the following broad considerations; 1) the alignment between the purpose, objectives, and intended use for the evaluation, 2) the structure of the storage and analysis of the existing data, and 3) the ramifications and implications of instrumentation modification. A preliminary checklist evaluators can use with their clients will be shared. The check list will enable evaluators to ask the right questions in order to make good decisions that ensure valid and reliable findings and lead to the appropriate use of resources in the context of evaluation.
|
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
To PhD or not to PhD? That is the Question |
|
Roundtable Presentation 645 to be held in Suwannee 19 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Graduate Student and New Evaluator TIG
|
| Presenter(s):
|
| Samuel Held, Oak Ridge Institute for Science and Education, sam.held@orau.org
|
| Pamela Bishop, University of Tennessee at Knoxville, pbaird@utk.edu
|
| Abstract:
The decision process for two PhD students at the University of Tennessee Program Evaluation PhD program is discussed. Both presenters are working parents juggling work, school, and home life. One presenter has one young child, is a wife, works as the full-time evaluator for the NSF National Institute for Mathematical and Biological Synthesis (NIMBioS), and is a part-time PhD student in her third year of classes. The other presenter is a father to four kids, a husband, a full-time evaluator at the Oak Ridge Institute for Science and Education, and a doctoral candidate. Both presenters will discuss why they decided to continue their education to the doctoral level in evaluation, the pros and cons of getting the advanced degree how they juggle multiple priorities as adult students, and how they balance the competing priorities.
|
| Roundtable Rotation II:
When Your Evaluation Goes Kaput: Lessons Learned From First Time Evaluators |
|
Roundtable Presentation 645 to be held in Suwannee 19 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Graduate Student and New Evaluator TIG
|
| Presenter(s):
|
| Adrienne Cadle, University of South Florida, adriennewoodleycadle@msn.com
|
| Sandra Naoom, University of South Florida, sandranaoom@gmail.com
|
| Abstract:
You have met with your client, proposed an evaluation, and the final contract has been signed, when all of a sudden your evaluation comes to a screeching halt. What do you do? For first time evaluators, this scenario can be very scary. For two first time evaluators, however, this scenario was a wonderful learning experience. These two evaluators learned how to explain the difference between research and evaluation, how to work with clients who think they know everything, how to choose the most effective evaluation team, and what to do when your evaluation timeline drastically changes. As two first time evaluators have learned, conducting an evaluation in the real world can be very different from what has been taught in an evaluation program.
|
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Best Practices in Professional Development: Understanding the Impacts of Teacher Investment and Implementation |
|
Roundtable Presentation 646 to be held in Suwannee 20 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Presenter(s):
|
| Leigh D'Amico, University of South Carolina, kale_leigh@yahoo.com
|
| Grant Morgan, University of South Carolina, praxisgm@aol.com
|
| Ching Ching Yap, Savannah College of Art and Design, ccyap@mailbox.sc.edu
|
| Abstract:
The concepts of teacher investment (i.e., buy-in) and classroom implementation are becoming critical factors in understanding the impact of professional development on school reform and student achievement. To examine the link between teacher investment, classroom implementation, and student achievement, a survey was developed and administered to a cohort of South Carolina middle school teachers who participated in a nine-month, intensive professional development program. In addition, student standardized achievement scores related to those teachers were obtained for a three-year period to gauge student achievement the year prior to, the year of, and the year after each teacher's participation in the professional development program. Results demonstrated that there is a moderately strong positive relationship between teacher investment and classroom implementation, and teachers who perceived themselves to be highly invested in the professional development program and strong implementers of the strategies highlighted had more consistent and often greater gains in student achievement.
|
| Roundtable Rotation II:
Enhancing Teaching Quality in High Need Schools: Evaluating the Implementation of a Standards-Based Professional Development Tool |
|
Roundtable Presentation 646 to be held in Suwannee 20 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Presenter(s):
|
| Jonathan Margolin, Learning Point Associates, jonathan.margolin@learningpt.org
|
| Jane Coggshall, Learning Point Associates, jane.coggshall@learningpt.org
|
| Shannon Fox, National Board of Professional Teaching Standards, sfox@nbpts.org
|
| Abstract:
Take One! is a professional development program offered by the National Board for Professional Teaching Standards (NBPTS) that requires participants to compile a portfolio entry containing evidence of teaching quality. In 2007-08, NBPTS made available the program to over 1300 teachers in 355 high-need schools. A formative evaluation examined the variation in support for program implementation, level of teacher implementation, and perceived change in teaching practice. Surveys were administered to 594 teachers, 58 school principals, and 40 program coordinators; psychometric analysis supported the validity and reliability of survey scales relating to implementation and outcomes. Descriptive analysis of surveys and case studies of five sites indicated that the program influenced teaching practice and enhanced teacher collaboration in the expected direction. Regression analyses and t-tests confirmed that high levels of principal support, mentor support, and technical support predicted greater implementation and perceived change in teaching practice.
|
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Developing Data Profiles to Support Community Health Improvement Efforts |
|
Roundtable Presentation 647 to be held in Suwannee 21 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Presenter(s):
|
| Elizabeth Serow, Florida Department of Health, betty_serow@doh.state.fl.us
|
| Abstract:
Six agencies collaborated to develop internet-based, county-level data profiles of School-aged Children and of Pregnancy and Young Children. Others are in development. The data cover risky behaviors, injuries and violence, social-emotional development, school and social environment and access to care.
Making data related to health and well-being readily available to community leaders and the general public is our way of assisting communities to take a more holistic look at subpopulations and the context in which they live.
These profiles include point-in-time and longitudinal data and are accessed through FloridaCHARTS.com, a site already being used by county/community based coalitions for targeting resources and applying for public and private funds to address relevant local issues.
Issues for discussion:
-Dealing with small numbers, both from the point of view of
confidentiality and rate stability
-Helping lay audiences interpret trend data
-Use of quartiles to help lay audiences identify salient issues
|
| Roundtable Rotation II:
Using the Model for Collaborative Evaluations (MCE) to Address Health Disparities: Evaluations in Communities of Faith |
|
Roundtable Presentation 647 to be held in Suwannee 21 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Presenter(s):
|
| LaShonda Coulbertson, University of South Florida, ltcoulbertson@msn.com
|
| Debra Thrower, University of South Florida, dthrower@ibl.usf.edu
|
| Abstract:
Communities of faith typically have limited resources necessary to evaluate programs their ministries have undertaken. This is more apparent among Communities of Faith within minority communities, which experience the devastating impact of health disparities in diseases such as cancer, diabetes, health disease, stroke, and obesity. Some faith-based organizations have created health and wellness ministries within their congregations to address these disparities, but have scant information detailing the impact of these ministries on the health of the congregation. This session will address the use of the Model for Collaborative Evaluations (MCE) (Campos, 2005) to evaluate a health ministry in an African American church, including the outcomes, lessons learned and potential application of MCE in the future.
Rodráguez-Campos, L. (2005). Collaborative evaluations: A step-by-step model for the evaluator. Tamarac, FL: Llumina Press.
|
|
Session Title: Evaluation for All: Strategies to Increase Racial/Ethnic Diversity in the Field of Program Evaluation
|
|
Multipaper Session 648 to be held in Wekiwa 3 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Multiethnic Issues in Evaluation TIG
|
| Chair(s): |
| Leona Johnson,
Hampton University, leona.johnson@hamptonu.edu
|
| Discussant(s): |
| Tamara Bertrand Jones,
Florida State University, tbertrand@admin.fsu.edu
|
|
Strategies for Increasing the Number of African Americans Entering the Field of Evaluation
|
| Presenter(s):
|
| Charita Ray-Blakely, Texas A&M University, crayblake@neo.tamu.edu
|
| Abstract:
Research suggests that the number of African American workers, in a great majority of professions such as academia, is minimal in comparison to the overall number of workers in that industry. The professional field of evaluation is no exception. Because there are many reasons contributing to this state, it is unlikely that one solution is achievable. Legislation such as Affirmative Action was enacted as a weapon to combat this inconsistency, but is far from an accepted solution. This study will, from an African American perspective, elucidate and explore the issue of awareness, serving as a contributor to the troubling shortage of African American's entering the field of evaluation. Through exploration of this issue, suggestions for increasing the number of African American's entering the field of evaluation will be made.
|
|
Culture Within Context: Innovative Strategies That Will Increase Ethnic Diversity Within the Evaluation Field
|
| Presenter(s):
|
| Jade Caines, Emory University, jcaines@emory.edu
|
| Abstract:
Based on recent data from the American Evaluation Association's internal scan, evaluators from racial/ethnic minority groups have limited representation within AEA membership. This leads to the assumption that evaluators at large are not racially/ethnically diverse. AEA must address these disparities. Evaluation studies must utilize the knowledge, skills, and abilities of ethnically diverse teams. Context, however, matters and calls for a multi-tiered approach aimed at increasing ethnic diversity within the field. Therefore, I propose a three-strand educational approach: First, AEA should create mentorship/shadow programs targeting secondary students. Second, an aggressive marketing campaign can help expose undergraduate/graduate students to the field. Third, workshops aimed at career-changers can highlight the opportunities available in, as well as the benefits of, the evaluation field. With increased federal mandates and a piercing focus on evaluation, there is no time to waste. AEA must step up to the challenges facing this field. The time is now.
|
| |
|
Session Title: Collaborative Evaluations: Successes, Challenges, and Lessons Learned
|
|
Multipaper Session 650 to be held in Wekiwa 5 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Chair(s): |
| Liliana Rodriguez-Campos,
University of South Florida, lrodriguez@coedu.usf.edu
|
|
The Housing Plus Project: Findings From a Multi-agency, Multi-stakeholder Collaborative Evaluation of the Implementation of Supportive Housing
|
| Presenter(s):
|
| John Sylvestre, University of Ottawa, jsylvest@uottawa.ca
|
| Purnima Sundar, Children's Hospital of Eastern Ontario, psundar@cheo.on.ca
|
| J Bradley Cousins, University of Ottawa, bcousins@uottawa.ca
|
| Tim Aubry, University of Ottawa, taubry@uottawa.ca
|
| Jaclynne Smith, University of Ottawa,
|
| Abstract:
HousingPlus is a collaboration among eight supportive housing providers and a university-based evaluation team. The objectives are 1) to develop tools and methods for evaluating program implementation, 2) to evaluate program implementation across the participating agencies, and 3) to use evaluation findings to assist participating agencies to collaborate in finding solutions to shared implementation challenges. This presentation examines findings from the implementation evaluation. It will describe a values-based evaluation tool developed for this evaluation by the evaluation team and stakeholders (including tenants, staff members, and senior managers/executive directors) and the process used to collect and analyze data. The findings will describe perceptions of program implementation from the perspectives of both tenants and service providers (staff members and senior managers/executive directors). These findings will be discussed in terms of their utility for understanding the implementation of supportive housing and for supporting collaborative efforts to improve how these programs are delivered.
|
|
Arts-Based Evaluations as Sites for Participatory and Collaborative Meaning Making
|
| Presenter(s):
|
| Michelle Searle, Queen's University, michellesearle@yahoo.com
|
| Abstract:
One of the core benefits of arts-based inquiry that is echoed evaluation work that is participatory and collaborative - an emphasis on multiplicity: multiple viewpoints, multiple perspectives and multiple voices. Whether arts-based inquiry provides pieces of the process or an encompassing method, it has potential to be attentive to community and environmental context, engage participants, answer new questions, and promote dialogue. Alexander (2005) explains that, 'it does this by encouraging empathetic and embodied engagement of other ways of knowing [that] fosters identification between dissimilar ways of being without reducing the other to bland sameness, a projection of the performing self' (p. 411). Arts processes in participatory and collaborative evaluative contexts have potential to be attentive to context because of their tendency towards experiences that are emotive and embodied (Simmons & McCormack, 2007). These practices can serve as a place for unified experiences, where theoretical and social knowing can merge while creating dialogue that provides a place to engage in critical reflection.
|
|
Evaluation of the Stroke Collaborative Reaching for Excellence
|
| Presenter(s):
|
| Christine Clements, University of Massachusetts Medical School, christine.clements@umassmed.edu
|
| Hilary Wall, Massachusetts Department of Public Health, hilarykwall@gmail.com
|
| June O'Neill, Massachusetts Department of Public Health, june.oneill@state.ma.us
|
| Heather Strother, University of Massachusetts Medical School, heather.strother@umassmed.edu
|
| Jillian Richard-Daniels, MassHealth, jillian.richard@state.ma.us
|
| Abstract:
The Stroke Collaborative Reaching for Excellence (SCORE) is a joint initiative between the Massachusetts Department of Public Health Paul Coverdell National Acute Stroke Registry, funded by the Centers for Disease Control and Prevention (CDC), and the American Stroke Association's Get with the Guidelines-Stroke. The Center for Health Policy and Research, University of Massachusetts Medical School, recently assisted the Massachusetts Department of Public Health with an evaluation to assess the experience of 55 hospitals that participate in SCORE. The purpose was to inform state-level enhancements that could contribute to successful hospital participation and to assist the Massachusetts Department of Public Health in providing CDC with information that could contribute to national program management. This paper presentation will describe the participatory evaluation process, development of evaluation questions, the qualitative and quantitative methods employed for data collection, and key results regarding the benefits and barriers to hospital participation in SCORE.
|
|
Aligning Collaborative and Culturally-Responsive Evaluations: Critical Reflections on Culture and Context in Evaluation
|
| Presenter(s):
|
| Monifa Beverly, University of Central Florida, mbeverly@mail.ucf.edu
|
| Karyl Askew, University of North Carolina at Chapel Hill, karyls@email.unc.edu
|
| Michelle Jay, University of South Carolina, jaym@gwm.sc.edu
|
| Abstract:
Collaborative evaluation is an orientation to evaluation that can promote culturally-responsive practice. Both collaborative evaluation and culturally-responsive evaluation approaches seek to develop and execute evaluations that accurately capture program impact and provide maximum benefit to stakeholders. Both acknowledge that evaluations that do not adequately seek to address the cultural context are likely to distort, minimize, or overlook program effects and undervalue stakeholders' expertise. However, while collaborative evaluation emphasizes evaluators' ability to hear and incorporate stakeholders' views, competencies, and objectives (and thus can be argued to be 'culturally sensitive'), collaborative evaluation lacks the explicit directive to critically examine the multi-faceted nature of culture as does culturally responsive evaluation. The presenters, three evaluators trained in the collaborative evaluation approach, will offer critical reflection on using collaborative evaluation methods to promote culturally responsive practices.
|
| | | |
|
Session Title: Evaluation for Research Centers, Institutes, and Multi-level Multi-context Programs
|
|
Multipaper Session 651 to be held in Wekiwa 6 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Brian Zuckerman,
Science and Technology Policy Institute, bzukerma@ida.org
|
|
A Study of the New Model and Methodology for Institute Evaluation in the Chinese Academy of Sciences (CAS)
|
| Presenter(s):
|
| Xiaoxuan Li, Chinese Academy of Sciences, xiaoxuan@casipm.ac.cn
|
| Bing Shi, Chinese Academy of Sciences, bshi@cashq.ac.cn
|
| Jianzhong Zhou, Chinese Academy of Sciences, jzzhou@casipm.ac.cn
|
| Abstract:
In this study, we presented a new model for institute evaluation based on the case study in Chinese Academy Science (CAS). Firstly, the characteristics of research activity of seven different institutes were analyzed. Secondly, the study summarized the basic rules, output characteristics, the quantitative and qualitative evaluation patterns of output and outcome about different institutes. Lastly, a three-hierarchy evaluation model for different institutes was presented. By this model, the key indicators as well as the benchmark of these indicators were expected to be found out through quantitative and qualitative methods. So that, the performance of the same type institutes could be compared to help the institute make clear their own development objects in the future.
|
|
The Status Analysis of Chinese Academy of Sciences Among National Research Institutes in the World
|
| Presenter(s):
|
| Guoliang Yang, Chinese Academy of Sciences, glyang@casipm.ac.cn
|
| Zhiyuan Liu, Chinese Academy of Sciences, liuzy@casipm.ac.cn
|
| Wenbin Liu, University of Kent, w.b.liu@kent.ac.uk
|
| Abstract:
Chinese Academy of Sciences (CAS) is a leading academic institution in China. In order to identify the status of CAS among national research institutes in the world, it is necessary to put CAS into the background of international S&T development and compare with other national research institutes to discover the advantages / disadvantages so that decision-makers can make suitable development strategy to improve the competitiveness and influence of CAS. This research discussed some key issues, such as the comparability of national research institutes, what to be compared and how to compare. This research tried to build a logical model based on common attributes of national research institutes to analyze the international status of CAS.
|
|
Designing Evaluation for Complex Multiple-component Funded Research Programs
|
| Presenter(s):
|
| Genevieve deAlmeida-Morris, National Institutes of Health, dealmeig@mail.nih.gov
|
| Abstract:
This paper describes the planning and designing of evaluation for a recent set of funded research programs. Research programs are complex, have time-related effects, and in the case of biomedical research, have benefits that can impact health and lives for generations. These complexities make it difficult to count and value the program benefits. Adding to these complexities are contextual dimensions such as scientist-managers administering the programs and participating in the evaluation while the evaluation must be maintained as a distinct function from program management, multiple components in a program some of which provide a service or products for use by other components, and requirements to conduct self-evaluation while also participating in the Agency's overall evaluation of the program. We describe how evaluation planning for these complex research programs was conducted, and how several methodologies were developed and employed to cope with the requirements of multiple contexts.
|
|
Research, Technology, and Development (RT&D) in Context: Evaluating the Indirect Effects of Self-Sustaining Cooperative Research Centers
|
| Presenter(s):
|
| Lindsey McGowen, North Carolina State University, lindseycm@hotmail.com
|
| Denis Gray, North Carolina State University, denis_gray@ncsu.edu
|
| Abstract:
An explicit goal of many federally funded research centers including NSF's Industry/University Cooperative Research Centers (and Engineering Research Centers), is to create self-sustaining centers. Thus, I/UCRCs are expected to have both direct effects while funded by NSF and indirect effects perhaps decades after NSF funding has ceased. The overarching goal of this study was to determine the extent to which I/UCRC program has produced self-sustaining centers and to estimate some of the outputs and impacts produced by these centers. The study examined the status of 73 'graduated' centers. Data collection focused on: funding leverage, human resources impacts, industrial partnerships and technological impacts. Findings suggest that graduated I/UCRCs can be categorized as: not sustained, sustained and transformed. Both sustained and transformed centers continue to demonstrate high levels of indirect effects over time. Implications for sustainability research, outcome evaluation and RT&D policy will be discussed.
|
| | | |
|
Session Title: Prospective Evaluation and Technology: New Developments and 21st Century Challenges
|
|
Panel Session 652 to be held in Wekiwa 7 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Valerie J Caracelli, United States Government Accountability Office, caracelliv@gao.gov
|
| Discussant(s):
|
| Bhavya Lal, Science and Technology Policy Institute, blal@ida.org
|
| George F Grob, Center for Public Program Evaluation, georgefgrob@cs.com
|
| Abstract:
Federal efforts to incentivize the innovation process may help our nation achieve a wide variety of goals, such as stimulating the economy, enhancing energy independence, and improving highway safety. Additionally, because the pace of 21st-century technological change is accelerating at an ever-increasing rate, federal agencies may need to "exercise foresight" by anticipating and keeping pace with fast-changing technology-based trends. Three papers discuss (1) conceptual frameworks for building and evaluating a "science of science policy"; (2) a recent assessment of the National Highway Traffic Safety Administration's strategy for incentivizing new crash-avoidance technologies (based on an evaluation framework that links prospective evaluation to evidence issues); and (3) the need to anticipate or assess the unintended consequences of technology, using the example of fast-changing distractions caused by drivers' use of cell-phones with new and changing capabilities.
|
|
What Might a Theory-Based Roadmap for Developing Innovation Policy Look Like?
|
| Gretchen Jordan, Sandia National Laboratories, gbjorda@sandia.gov
|
|
Prospective evaluation looks forward in time to answer questions such as "where shall we invest to have the most impact?" Given the uncertainty, time lags, and confounding events between a scientific discovery and a "product" in use that provides a benefit or solves a problem, this is a very difficult question. People see different parts of the elephant, and there is no agreement on the big picture. This paper will propose a logic model depiction of a roadmap for developing a Science of Science and Innovation Policy, a new National Science Foundation initiative in response to requests from the White House Office of Science and Technology Policy (OSTP). This logic model is based on a December 2008 OSTP-sponsored workshop. It shows the non-linear science and technology process encompassed by micro and macro level inputs and incentives. Further, the paper will discuss a number of theory-based models that underlie this logic.
|
|
|
Evaluating a Federal Effort to Incentives the Development of Technologies for Helping Drivers Avoid Highway Crashes
|
| Nancy Donovan, United States Government Accountability Office, donovann@gao.gov
|
|
In the area of highway safety, new technology-based trends combine (1) an evolving series of technology developments and new products with (2) consumer adoption and patterns of use. New crash avoidance technologies include electronic systems that assist drivers who may be, for example, drifting out of their lane or risking rear-end collisions. These technologies are projected to continue evolving past 2020 and thus present an opportunity to enhance future safety. However, there are uncertainties about the magnitude of safety improvement and the effectiveness of strategies to incentivize development. (One strategy involves testing new technologies and publicizing results to consumers.) Since no government-wide guidance exists on how federal agencies should address trends across a time horizon more than 5 years forward, GAO developed a framework highlighting activities related to (1) decision-making, (2) evidence development and (3) communication. The paper discusses this framework's evolution and its application to crash avoidance technologies.
| |
|
Evaluating An Agency's Response To A "High Clockspeed" Technology Trend With Unintended Consequences
|
| Judith Droitcour, United States Government Accountability Office, droitcourj@gao.gov
|
|
Although technological innovations improve our lives in various ways, they can have unintended, unforeseen consequences. The side-effects that accompany an evolving "high-clockspeed" technology-based trend may challenge the government's ability to respond in a timely way. One example is drivers' (including teen drivers') use of fast-evolving cell-phones with capabilities such as texting and searching the internet, which are quickly adopted by consumers. This paper discusses a recent evaluation of the National Highway Traffic Safety Administration's response to these new and fast-changing driver distractions. The evaluation, which was conducted by the U.S. Government Accountability Office, found that federal responses were constrained by a combination of (1) a perceived need to await the highest-quality evidence before responding, and (2) difficulties and lags in developing new kinds of quality evidence on an currently emerging and changing phenomenon. Additionally, there was a lack of agency communication with congressional policy-makers about this issue.
| |
|
Session Title: Instrument Development and Evaluability Assessment in Mental Health and Substance Abuse
|
|
Multipaper Session 653 to be held in Wekiwa 8 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG
|
| Chair(s): |
| Roger Boothroyd,
University of South Florida, boothroy@fmhi.usf.edu
|
|
Evaluating and Improving Implementation of Environmental Strategies for Substance Abuse Prevention: Discussion of an Initial Process Measure
|
| Presenter(s):
|
| Crystelle Egan, University of Rhode Island, caegan@mail.uri.edu
|
| Paul Florin, University of Rhode Island, pflorin@mail.uri.edu
|
| Cynthia Roberts, University of Rhode Island, cynthia.roberts70@gmail.com
|
| Abstract:
The evaluation of environmental strategies (ES) for substance abuse prevention is challenged by a lack of standardized measures. Moreover, the capacity to implement ES may be limited among some prevention practitioners. To address these challenges simultaneously, we developed a process measure that also serves as an implementation guide and a prompt for technical assistance. The Monthly Environmental Strategies Tracking Interview is comprised of structured quantitative and qualitative questions regarding the implementation of media, policy, and enforcement strategies by municipal coalitions funded through a Center for Substance Abuse Prevention Strategic Prevention Framework State Incentive Grant (SPF SIG). The objectives of this paper are: 1) to describe the rationale for and development of the measure, 2) to discuss how the measure is used to facilitate strategy implementation, and 3) to explore the strengths and challenges of this approach. Sample collected data will be presented and copies of the measure will be available.
|
|
Evaluation of How Mental Health Staff Promotes Recovery: A Rasch Analysis of the PRO Survey
|
| Presenter(s):
|
| Kathryn DeRoche, Mental Health Center of Denver, kathryn.deroche@mhcd.org
|
| Pablo Olmos, Mental Health Center of Denver, antonio.olmos@mhcd.org
|
| Abstract:
Consumer-centered mental health recovery has emerged as a novel philosophy among adult community-base service providers over the last two decades. Distinctive from the abundance of recovery instruments, the Promoting Recovery in mental health Organizations (PRO) survey measures the consumer's perception of how different types of staff members promote recovery. Six sub-scales representing different types of staff members were developed by the Mental Health Center of Denver's Recovery committee and administered to a stratified random sample of 450 consumers. By using Rasch analysis techniques, similarities and differences were discovered in how therapists, case managers, psychiatrists, other medical staff, front-desk staff, and rehabilitation staff promote consumers' recovery. Furthermore, the relationships among PRO sub-scales and other measures of recovery were examined (consumers' self-report, and clinicians opinions). Finally, the implications for quality improvement based on the results of the PRO analysis are discussed, including training, staff evaluation, and hiring.
|
|
The Youth Evaluation Project at the Mental Health Center of Denver
|
| Presenter(s):
|
| Kathryn DeRoche, Mental Health Center of Denver, kathryn.deroche@mhcd.org
|
| Pablo Olmos, Mental Health Center of Denver, antonio.olmos@mhcd.org
|
| Lydia Prado, Mental Health Center of Denver, lydia.prado@mhcd.org
|
| Abstract:
To date, instruments developed to evaluate progress in youth seeking mental health services contain three inherent limitations, including: (1) lacking input from key stakeholders, (2) focusing on symptom reduction, and (3) the psychometric procedures utilized include assumptions that do not match the theories developed. At the Mental Health Center of Denver, we have started the development of a holistic evaluation system to overcome these limitations. The current presentation will discuss the results of the first stage of research, including analysis of a qualitative constructivism grounded theory containing 33 informal interviews with youth receiving mental health service, parents/guardians, clinicians, teachers, and community members. In the presentation, we will focus on the similarities and differences among the categories of stakeholder, in addition to how the results were used to develop quantitative evaluation surveys. Finally, implications for practice and future directions for additional validation will be discussed.
|
|
Midcourse Evaluability Assessment: Redesigning the Strategic Prevention Framework Cross-Site On the Fly
|
| Presenter(s):
|
| Robert Orwin, Westat, robertorwin@westat.com
|
| Robert Flewelling, Pacific Institute for Research and Evaluation, flewelling@pire.org
|
| Abstract:
This paper analyzes the evaluation circumstances of 26 States participating in the U.S. Center for Substance Abuse Prevention's Strategic Prevention Framework State Incentive Grant, a national public health initiative to prevent substance abuse and its consequences. Each State used a data-driven planning model to allocate resources to 445 communities which in turn launched over 1400 intervention strategies to target prevention priorities in their populations. Using a standardized template, the paper assesses what has happened with each State evaluation's design and implementation, yielding a "grade" reflecting the degree to which credible conclusions about program effectiveness are likely. Inputs include the existence and quality of 1) comparison communities, 2) longitudinal population-based outcomes data, 3) community-level fidelity data, 4) pre- and post-intervention intervening variables data, and 5) geographic concordance between program and data catchment areas. Collectively, the grades and associated narratives will in turn be used as needed to revise the national design.
|
| | | |
|
Session Title: Teaching Evaluation in a Variety of Contexts: Lessons From the Classroom and the Field
|
|
Panel Session 655 to be held in Wekiwa 10 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Teaching of Evaluation TIG
|
| Chair(s): |
| Gary Varrella, Washington State University, gvarrella@wsu.edu
|
| Discussant(s):
|
| Gary Varrella, Washington State University, gvarrella@wsu.edu
|
| Abstract:
Three experienced practitioners will discuss their perspectives on and experiences in developing peers', students', and organizations' abilities to develop and conduct evaluation in a variety of contexts. Panelist will highlight formal and informal educational/teaching approaches referenced to curriculum, relevant pedagogy, and strategies for technical assistance. Interaction among the panelist and between the panel and audience will be a priority, tapping the "collective intellect" of those present as we examine this relevant topic.
|
|
Teaching Evaluation of Development and Cooperation Programs in an International Setting
|
| José Maria Diaz Puente, Polytechnic University of Madrid, jm.diazpuente@upm.es
|
|
This presentation will revolve around experiences teaching a course focused on evaluation of development and cooperation programs in the International Master Program in Rural/Local Development. The Commission of the European Union under the Erasmus Mundus Program funds it. The students of this evaluation course are professionals working in development and cooperation around the world. The course goal is to provide an introduction into evaluation theory and practice, as well as consider practical evaluation tools. In this session I shall encourage discussion regarding the challenge to design the course contents for a heterogeneous population of students; the use of the on-line course elements before the arrival of the students to Madrid; different ways to account for the cultural and ethnical differences; and the use of participatory techniques for engaging the students including skill-building, key points, and grading.
|
|
|
Informal Teaching of and Technical Assistance in Evaluation in Criminal Justice
|
| Jennifer Loeffler-Cobia, Crime and Justice Institute, jcobia@crjustice.org
|
|
Criminal justice organizations focus on two questions, (1) "are the evidence based practices/services we provide to offenders,--juvenile and adult--having an impact on recidivism" and (2) "how can we improve our service to obtain better outcomes." Appropriate applications of program planning and evaluation are critical to answer these questions. The creation and implementation of evaluation and enhancement plans requires effort and attention to detail. Hence, developing, implementing, and maintaining evaluation processes is a project in and of itself within a larger program, requiring a high level of staff commitment. My comments will highlight experiences in providing technical assistance (i.e., through informal teaching/consultation) on evaluation and enhancement plans to juvenile and adult justice organizations related to: (1) developing SMART objectives, (2) developing data collection and utilization plans, (3) analyzing data, and (4) enhancement planning related to improving offender outcomes.
| |
|
Peer, Collaborator, and Teacher: Roles in Expanding Evaluation Capacity Among Extension Faculty
|
| Gary Varrella, Washington State University, gvarrella@wsu.edu
|
|
Extension programs have, quite successfully, adhered effectively used logic models and predominantly quantitative approaches to evaluation. However, as is the case more often than not, in all organizations, evaluation is the last or one of the last things considered in program planning. As with the other populations featured in this session, there is a high expectation for evaluation product, but a relatively low commitment to investment of time in evaluation and complicated by uncertainty about how to approach evaluation. The panelist will discuss strategies and approaches to teaching evaluation in formal and informal settings considering barriers, successes, and strategies that broaden peers' perspective on and ability to conduct straightforward, targeted value-laden evaluation, including that of mixed design (qualitative & quantitative). The nuances and affects of attention to relevant learning theory and adult motivation will be considered as well.
| |
