|
Session Title: Exploring Evaluation and Learning: Winners of the Student Competition
|
|
Panel Session 407 to be held in International Ballroom A on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Presidential Strand
|
| Chair(s): |
| Jennifer Martineau,
Center for Creative Leadership,
martineauj@leaders.ccl.org
|
| Discussant(s):
|
| Hallie Preskill,
Claremont Graduate University,
hallie.preskill@cgu.edu
|
| Michelle Searle,
Queen's University,
michellesearle@yahoo.com
|
| Tysza Gandha,
University of Illinois at Urbana-Champaign,
tgandha2@uiuc.edu
|
| Marisa Allen,
Case Western Reserve University,
marisa.allen@case.edu
|
| Mijung Yoon,
University of Illinois at Urbana-Champaign,
myoon1@uiuc.edu
|
| Alice Navarro,
Texas State University-San Marcos,
alice.navarro@yahoo.com
|
|
Session Title: Accountability, Federal Programs, and No Child Left Behind
|
|
Multipaper Session 408 to be held in International Ballroom B on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| James Van Haneghan,
University of South Alabama,
jvanhane@usouthal.edu
|
|
Programmed for Trouble: Evaluation of Federally Funded Education Programs
|
| Presenter(s):
|
| Keith Murray,
M A Henry Consulting LLC,
keithsmurray@mahenryconsulting.com
|
| Martha Henry,
M A Henry Consulting LLC,
mahenry@mahenryconsulting.com
|
| Abstract:
Federally funded education programs often mandate external evaluation as part of their grant projects. U.S. Department of Education and National Science Foundation projects consume considerable resources and reach into thousands of classrooms. The external evaluation requirement may seem to reflect the evaluation field's incorporation as a crucial partner in successful program implementation in this critical area. In practice, however, the challenges of managing evaluation given the programs' structure and constraints are substantial. Ambiguous chains of authority and communication, ambivalent funder support and oversight, inconsistent definitions of evaluators' responsibilities, and grantee control over evaluator contracts are among the challenges experienced. This paper examines the role of evaluation in federal education programs, informed by input from grantor program staff, grantees, and evaluators. The authors conclude with recommendations for a reconsideration of evaluator responsibilities, grantee education, and realignment of the grantor-evaluator relationship to optimize evaluation's service to education projects and their many stakeholders.
|
|
Evaluating No Child Left Behind Test-based Educational Accountability
|
| Presenter(s):
|
| Katherine Ryan,
University of Illinois at Urbana-Champaign,
k-ryan6@uiuc.edu
|
| Abstract:
Audit cultures continue to intensify nationally and globally--reflections of New Public Management which rely on performance-based management (Power, 1997). The No Child Left Behind Act (NCLB) of 2001 displays this performance-based management approach by dramatically increasing the impacts of test-based educational accountability for managing and improving education. This is essentially a performance measurement system (Ryan, 2007).
Importantly, while there are significant impacts (e.g., school closings) associated with NCLB accountability, provisions for evaluating NCLB high stakes assessment and accountability are notably absent. This paper presents a framework for evaluating state- wide NCLB test-based educational accountability to examine whether the performance measurement system (e.g., high stakes assessments) is being implemented as planned. In addition, the framework will incorporate intended and unintended consequences of performance measurement uses in relationship to students, instruction, and educational outcomes. I review several literatures in developing this framework including the learning organization, implementation theory, standards for accountability system and others.
|
|
Assessing the Feasibility of Incorporating Geocoding and Census Data Into the National Assessment of Educational Progress (NAEP) Framework
|
| Presenter(s):
|
| Stacey Merola,
American Institutes for Research,
smerola@air.org
|
| Steve Davis,
American Institutes for Research,
sdavis@air.org
|
| Abstract:
As part of our ongoing work to develop a new composite measure of socioeconomic status (SES) for NAEP, we are investigating using census data in conjunction with NAEP. Incorporating census data into the NAEP framework presents some logistical and methodological challenges that must be addressed. In this paper we will present the results of a geocoding pilot study conducted in March 2007 with 167 schools across the United States. The users were asked to collect student addresses and then geocode those addresses using a geocoding software that has been developed specifically for NCES. The users will be asked via a survey and debriefings about their experiences. The data collection is currently underway, and to date the results are positive.
|
|
Conducting a Robust Performance Needs Assessment: An No Child Left Behind (NCLB) Case Study
|
| Presenter(s):
|
| Willis Thomas,
Western Michigan University,
willis.h.thomas@wmich.edu
|
| Abstract:
This paper will begin by discussing the recently completed year-long, two-part, No Child Left Behind Parental Involvement evaluation, and how that project led to a performance needs assessment of the related Parent Corps Program.
The design of the performance needs assessment was quite unique, integrating good practices from a variety of methodologies and approaches. This performance needs assessment looked at performance needs, treatment needs, and solutions. It investigated the core-common-critical needs as well as developmental needs of program participants. It resulted in an excellent model, and is worth considering when engaging in program analysis or action research.
|
| | | |
|
Session Title: Systems in Evaluation TIG Business Meeting and Think Tank: Early Steps to Next Steps: Where Are We Now? Where Should We Go Next?
|
|
Business Meeting Session 409 to be held in International Ballroom C on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Systems in Evaluation TIG
|
| TIG Leader(s): |
|
Derek A Cabrera,
Cornell University,
dac66@cornell.edu
|
|
Janice Noga,
Pathfinder Evaluation and Consulting,
jan.noga@stanfordalumni.org
|
|
Bob Williams,
Independent Consultant,
bobwill@actrix.co.nz
|
| Chair(s): |
|
Derek A Cabrera,
Cornell University,
dac66@cornell.edu
|
| Discussant(s): |
|
Tessie Catsambas,
EnCompass LLC,
tcatsambas@encompassworld.com
|
| Abstract:
The TIG turns 3 this year! In the past three years, the TIG has engaged in conversation targeted at identifying key issues of interest regarding systems thinking and evaluation. During the past year, TIG leaders conducted a survey of members to elicit their perspectives on systems thinking and how the TIG can support members’ continued learning. Now it’s time pull together the key themes and ideas that have emerged so far, put that in perspective with members’ expectations for the TIG, and develop a strategy for moving forward. This will be a highly interactive session utilizing appreciative inquiry to craft a strategy for taking the TIG forward.
|
|
Session Title: Conceptual and Empirical Advances in Evaluation Capacity Building
|
|
Multipaper Session 410 to be held in International Ballroom D on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Evaluation Use TIG
|
| Chair(s): |
| J Bradley Cousins,
University of Ottawa,
bcousins@uottawa.ca
|
| Discussant(s): |
| Peter Dahler-Larsen,
University of Southern Denmark,
pdl@sam.sdu.dk
|
| Abstract:
Understanding evaluation capacity, and perhaps more importantly, how to go about fostering, developing or otherwise promoting it - that is, evaluation capacity building (ECB) - has captured the attention of many contributors to the field, not the least of whom are those involved in ECB practice. Yet conceptual advances and empirical research on ECB have been comparatively limited to date. Research and theory stand to benefit, for example, from broadening our understanding of evaluation capacity to include consequences of evaluation (Compton, Braizerman & Stockdill, 2002), and moving beyond organizational contexts to consider implications of ECB on a grander scale (Levin-Rozalis, Rosenstein & Cousins, forthcoming). This multiple paper session will showcase conceptual and empirical advances in ECB and contribute to the development of an agenda for ongoing inquiry that might usefully inform ECB practice.
|
|
Integrating Consequences of Evaluation into Evaluation Capacity Building Inquiry
|
| J Bradley Cousins,
University of Ottawa,
bcousins@uottawa.ca
|
| Swee C Goh,
University of Ottawa,
goh@admin.uottawa.ca
|
| Catherine Elliott,
University of Ottawa,
elliott.young@sympatico.ca
|
|
A significant conceptual development in evaluation capacity building inquiry is reflected in a commitment to understand and foster the capacity to use evaluation (Compton, et al., 2002; Cousins et al. 2004). This paper describes the development and evolution of a conceptual framework intended to guide empirical inquiry that captures this perspective. The capacity to do evaluation is situated among conceptual consequences of evaluation and conditions that mediate such influences. Contributions to framework evolution of a concept mapping study and recent developments in the evaluation utilization literature are identified and implications for empirical inquiry are discussed.
|
|
Precarious Balance
|
| Miri Levin-Rozalis,
Ben-Gurion University of Negev,
rozalis@internet-zahav.net
|
|
The rift in educational evaluation between the 'systems' vs. 'human agencies' approaches stemming from rival ontological conceptions reflects on Evaluation Capacity Building (ECB) processes of both evaluators and evaluee. An axiological 'ethical purposive' approach can help us to overcome this rift. The ends of inquiry into human activity are first ethical and political, concerning the worthwhile life and the just society and only secondarily epistemology or methodology. The main question for ECB is not "how to do" but "what for?" and thus "what difference should evaluation make?" The answers should lead our decisions concerning methodologies and processes that will keep the precarious balance between contradicting demands.
|
|
Evaluation Capacity Across Organizational Contexts: What are the Patterns?
|
| Swee C Goh,
University of Ottawa,
goh@admin.uottawa.ca
|
| Robert Lahey,
REL Solutions Inc,
relahey@rogers.com
|
| Catherine Elliott,
University of Ottawa,
elliott.young@sympatico.ca
|
| Jill Chouinard,
University of Ottawa,
jchou042@uottawa.ca
|
| Keiko Kuji-Shikatani,
Independent Consultant,
kujikeiko@aol.com
|
|
This paper reports preliminary within- and cross-case results from a larger multiple case study designed to inquire into the nature and state of evaluation capacity and its antecedents. Of particular interest is the capacity to use evaluation and its relationship to the capacity to perform and/or oversee the function. Organizations included in this paper vary from agencies that fund community social services, to large federal government departments, to a secondary educational institution. Despite enormous variation in organizational context and evaluation capacity, within- and cross-case analyses are guided by a common conceptual framework. Emergent patterns are identified and discussed in terms of implications for ongoing inquiry.
|
|
A Case Study of a Successful School Evaluation Capacity Building Process
|
| Barbara Rosenstein,
Ben-Gurion University of Negev,
rbarbara@netvision.net.il
|
| Pnina Elal-Englert,
Ben-Gurion University of Negev,
rbarbara@netvision.net.il
|
|
This paper presents an evaluation capacity building (ECB) case of a school facing the challenge of mainstreaming children with special needs. The school has developed a successful program of ECB over a period of 10 years. We examine the case according to three separate but complimentary frameworks: first, that set out by Compton, Baizerman, Stockdill (2002) in which they distinguish between program evaluation practitioner and ECB practitioner. Second, we demonstrate King's (2002) prescriptions for successfully operating an ECB system in a school in relation to three stakeholders groups: teachers, administrators and opinion leaders; students and the curriculum. Third, we illustrate how the case fits Cousins' (2005) discussion of the role of the principal in school organizational learning. After thorough and systematic investigation, we believe that ECB developments at this school can serve as a model for other schools with similar contexts.
|
|
Session Title: Tools for Useful Performance Assessment of Science and Technology Programs: An Example
|
|
Panel Session 411 to be held in International Ballroom E on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Jerald Hage,
University of Maryland,
hage@socy.umd.edu
|
| Discussant(s):
|
| Alfred Powell,
National Oceanic and Atmospheric Administration,
al.powell@noaa.gov
|
| Abstract:
In this proposed panel, we plan to describe the ongoing relationship between the Center for Innovation and NOAA's Center for Satellite Applications and Research (STAR). In meeting the needs of STAR, the Center has been able to further the development of our suite of tools for the assessment and evaluation of research organizations. These tools are centered around the balanced scorecard approach, and we have been working to develop appropriate metrics for research organizations at four levels: organizational, scientific, innovation, and economic benefits. We will present work from our STAR research projects and discuss the ways that these projects offered the opportunity to explore further development of our assessment tools. The panel discussant will be Dr. Alfred Powell, Jr., Director of STAR.
|
|
A Strategic Balanced Scorecard for Publicly Funded Science and Technology Programs
|
| Gretchen Jordan,
Sandia National Laboratories,
gbjorda@sandia.gov
|
|
There is increasing interest in stimulating scientific advance and innovation, and at the same time there are increasing requirements to demonstrate value of organizations and portfolios. However, both current knowledge of how to stimulate innovation and current performance measurement systems fall short. In this presentation, we discuss our approach consisting of a balanced scorecard of performance indicators based on the use of strategy maps and which are informed by theories of knowledge production and innovation. A balanced scorecard that is based on a strategy map is similar to a logic model/theory of change approach to describing strategy and choosing indicators. A balanced scorecard based on organizational strategy clarifies and communicates the strategy, technical focus and performance, and focused management initiatives. Balanced scorecards reflect multiple stakeholders' definitions of 'success', and thus provide more comprehensive information for policy makers and managers than ad hoc sets of indicators. In addition, we have been working to developing real-time measures of scientific progress that let managers know if performance is as expected, and if not, other indicators to show where the bottlenecks might exist. In this manner, the linkage of performance to management practice and theory is key. Our work with STAR has been critical in our efforts to combine generalizable, real time measures of S&T progress and organizational and mission goals in the balanced scorecard framework.
|
|
|
Perceptions of the Research Environment: Kinds of Networks, Research, and Projects
|
| Jerald Hage,
University of Maryland,
hage@socy.umd.edu
|
|
Most existing employee attitude surveys do not focus on the specific attributes that scientists and engineers consider to be particularly important for research organizations. The Research Environment Survey (Jordan, 2005) was developed especially for research organizations and covers key attributes of organizational structure and management practices within the research environment which were identified and defined through an extensive literature review and input from fifteen focus groups that included bench scientists, engineers, and technologists, as well as their managers, across various R&D tasks. In this presentation we discuss an application of that survey at STAR and our efforts to tailor the survey. In particular, we will emphasize our analysis of those survey questions that allowed scientists to indicate their preferences for change in selected attributes of the research environment and those that captured internal networks. An important finding of the study was that those scientists that indicated a greater amount of applied work (applied scientists) also expressed the strongest preferences for an increasing amount of time to be allocated to various attributes, particularly those associated with innovative research. We also demonstrated that it is important to look at the structure of the intra-organizational networks, as we found a strong relationship between position in the network and perception of the research environment. For example, do those with high closeness have favorable perceptions of the research environment, particularly on those items that we think are favorable to fostering network connections and activity? All too often, this role of the organizational context in fostering or inhibiting the desired type of networks is overlooked. Other findings and a general discussion of the recommendations to management that are possible based on the research environment survey will be provided.
| |
|
Measuring the Immeasurable: Innovation and the Economics Benefits of Satellite Data
|
| Jonathon Mote,
University of Maryland,
jmote@socy.umd.edu
|
|
In this presentation, we discuss our work to measure two aspects of STAR's work: innovation and the economic benefits of satellite data. As we have argued elsewhere, standard measures of innovation-patents, papers and citations---are limited for a number of reasons, primarily because of the time lag between innovation and publication and they do not account for the application of the innovation. However, these measures are standards for a reason-they are easily understandable and readily accessible. In this project, we wanted to explore innovative ways to utilize these measures, as well as couple these measures of scientific innovation to their applications, that is, product or process innovations. Another project with STAR sought to quantify the economic benefits of STAR's primary work, processing and conducting research on satellite weather data. The challenge is akin to quantifying the economic benefits of any basic research, where there is little direct connection to the market. Both of these projects represent efforts to quantify some of the intangible contributions of research organizations into useable indicators that can be incorporated into performance measurement frameworks.
| |
|
Session Title: Starting a "Cost Study": Measuring and Improving Costs, Cost-effectiveness, and Cost-benefit in Human Services
|
|
Skill-Building Workshop 412 to be held in Liberty Ballroom Section A on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Costs, Effectiveness, Benefits, and Economics TIG
|
| Presenter(s):
|
| Brian Yates,
American University,
brian.yates@mac.com
|
| Abstract:
You can do a "cost study:" you can understand, measure, and often discover ways to improve: a) costs, b) effectiveness, c) benefits, d) cost-effectiveness analysis, and e) cost-benefit analysis in mental health or other human services. This demonstration begins with a plain-English orientation to definitions of costs, effectiveness, benefits, cost-effectiveness, and cost-benefit. Methods of 1) measuring and 2) analyzing each of these are illustrated with examples from the presenter's research in child management training, and substance abuse prevention programs for children in elementary school, and mental health services. A quantitative understanding of what occurs between the "costs in" and "outcomes out" is enhanced by a model that distinguishes between performance of and participation in program procedures, and between desired and actual change in biopsychosocial processes responsible for program outcomes. References and web sites are provided for further learning.
Examples of each step in understanding and improving relationships between resources used, procedures implemented, biopsychosocial processes altered or instilled, and outcomes produced are drawn from the instructor's evaluation research in health, mental health, and substance abuse settings.
In addition, a variety of studies are reanalyzed to illustrate how cost-procedure-process-outcome analysis enhances the applicability of basic and applied research to the systematic understanding and management of human service systems.
|
|
Session Title: Reflecting on Practice: Strategies and Lessons Learned
|
|
Panel Session 413 to be held in Liberty Ballroom Section B on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Theories of Evaluation TIG
|
| Chair(s): |
| Jeremy Lonsdale,
United Kingdom National Audit Office,
jeremy.lonsdale@nao.gsi.gov.uk
|
| Discussant(s):
|
| Nancy Kingsbury,
United States Government Accountability Office,
kingsburyn@gao.gov
|
| Abstract:
Reflecting on practice is a hallmark of Evaluation practice. Evaluations are improved, defended, and examined for strengths and limitations through systematic metaevaluations. Meeting this standard of practice is inextricably bound to the quality of our work and the subsequent use and influence of evaluation findings. It is also linked to our confidence in the lessons we derive and promulgate for the purpose of improving and better managing programs. Constructive lessons learned are intended to build theory and evaluation capacity, moreover, identified good practices are intended to provide policy makers and practitioners with knowledge and insights that can improve program design, implementation, and effectiveness. This panel presents several different perspectives on how we can productively and critically engage in reflective practice. It will raise issues about criteria for assessing quality, the mechanisms and effects of good practice advice, how to leverage evaluation impact, and benefits, concerns, and questions we should raise.
|
|
The Practice of Metaevaluation: Does Evaluation Practice Measure-up?
|
| Leslie Cooksy,
University of California, Davis,
ljcooksy@ucdavis.edu
|
| Valerie J Caracelli,
United States Government Accountability Office,
caracelliv@gao.gov
|
|
Metaevaluation, a systematic review of an evaluation to determine the quality of its processes and findings, is one of the Program Evaluation Standards. Metaevaluation may be applied to a set of evaluations as a precursor to meta-analysis or evaluation synthesis, but how common is its use in the sense intended by the Program Evaluation Standards -- to ensure that 'stakeholders can closely examine the evaluation's strengths.' After a brief background on the theory of metaevaluation, the paper examines the practice of metaevaluation by the evaluation community. Specific questions that will be addressed are: How many metaevaluations can be readily identified? What criteria of quality are used and how are they applied? The paper will provide information about the range of criteria and approaches to metaevaluation used in practice and raise awareness of the use of metaevaluation as a method for assessing evaluation quality.
|
|
|
Simply the Best? Understanding the Market for Good Practice Advice From Government Research and Evaluation
|
| Elena Bechberger,
London School of Economics and Political Science,
e.k.bechberger@lse.ac.uk
|
|
Many evaluations of government programs carried out by state audit bodies or government institutions are now generating what is termed 'good practice' advice. Whether developing new policies or managing existing programs, there is a wide variety of printed and online guides offering suggestions to public officials about how to do things properly. While there has been some recent research on the use of good practice, the foundation question of how good practices are identified and whether they are in fact good practices has been ignored. This paper examines how compilers of such advice in the United Kingdom identify and evaluate practices to be good and develops a typology of different bases on which such evaluations are formed. It goes on to set out how disseminators of such advice conceive of their audiences, and sheds light on the receptivity to, as well as effects of, such advice.
| |
|
Assessing the Utilization and Influence of Evaluations
|
| Michael Bamberger,
Independent Consultant,
jmichaelbamberger@gmail.com
|
|
There is a widespread concern that evaluations are underutilized. This paper reports on a recent World Bank study that identified a sample of evaluations of development programs and policies where there was plausible evidence that the evaluations had been 'influential' and cost-effective. The cases illustrate the different ways that evaluations can be influential, many of them unanticipated and not all welcome; and describe simple methodologies for attribution analysis and for assessing the cost-effectiveness of an evaluation. Evaluations are never the only factor influencing decision-makers, and a framework is proposed for leveraging evaluation impact through strategic interaction with other sources of influence to which decision-makers are exposed. A mapping exercise is proposed through which an agency could review its evaluations to assess their levels of influence and cost-effectiveness, identify factors determining influence and develop guidelines to enhance utilization.
| |
|
What Questions Should we ask About Lessons Learned?
|
| Martin de Alteriis,
United States Government Accountability Office,
dealteriism@gao.gov
|
|
A wide variety of methods are used to derive lessons learned, from practitioners' anecdotal reports of their experiences in a particular activity, to scholars' broad review of the state of knowledge in a particular field. The variety of methods used, the rigor with which they are utilized, may be a function of the knowledge base from which lessons are drawn. Few people dispute the value of well-supported lessons learned. A range of techniques, if implemented rigorously, could provide policy makers and practitioners with knowledge and insights that would improve program design and implementation. Lessons learned can draw on the experience of a large number of practitioners, as well as experts and researchers. They can link cause to effects in ways that should be beneficial for implementation. However, potential problems can arise and it is important to be able to distinguish bona fide lessons learned from ones conducted with inadequate methodologies.
| |
|
Session Title: International Issues in Evaluation
|
|
Multipaper Session 414 to be held in Mencken Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Government Evaluation TIG
|
| Chair(s): |
| Kris Juffer,
Action Research & Associates Inc,
drkjuffer@comcast.net
|
|
Monitoring and Evaluation for Accountability: The Case of Nigeria's Virtual Poverty Fund
|
| Presenter(s):
|
| Lawal Aboki,
Office of the Senior Special Assistant to the President on Millenium Development Goals,
lyaboki@yahoo.co.uk
|
| Abstract:
The granting of debt relief to Nigeria in September 2005 was surrounded by historically-founded skepticism that the savings would be not be utilized properly for the benefit of ordinary Nigerians. This paper describes the development and implementation of a comprehensive tracking system entitled 'Overview of Public Expenditure in NEEDS' (OPEN) that aimed to both effectively track debt relief expenditures, and act as a platform for wider public expenditure reform. A key component of OPEN is a monitoring and evaluation (M&E) mechanism that monitors the quality of debt-relief funded expenditures. The process has brought together government officials, private sector experts and civil society organizations. The paper describes the numerous challenges faced by the Nigerian government in the development of OPEN. The process of implementation has lessons for the entire international community, but most significantly many lessons that are uniquely African.
|
|
Normative Supports, Psychological Resistances, and Procedural Support: Lessons Learned of Performance Management of the Korean Government
|
| Presenter(s):
|
| Wha-Joon Rho,
Seoul National University,
wjrho@snu.ac.kr
|
| Kilkon Ko,
National University of Singapore,
kilkon@gmail.com
|
| YeonBaek Jeong,
Seoul National University,
|
| Abstract:
Despite a strong initiative of performance management by the Korean government over last years, critics argue that it did not change behaviors of public officials. The criticism implicitly assumes that organizational capacity is the result of performance management. However, the reverse causation would be valid: without enough organizational capacity, no desirable outcomes of performance management are expected.
This paper analyzes how members of organizations support performance management normatively, and how they resist to it in actual implementation procedures, and how they comply with the formal requirement. For the analysis, we surveyed 600 public servants in different jurisdictions, and collected actual performance evaluation data of 27 central government department and agencies of the Korean government. The former includes a normative and psychological attitudes of organizational members. The latter covers the formal responses of organizations required by performance management. Based on those data, we will explain why the delusive high normative support disguises the low compliance of members in actual implementation
|
| |
|
Session Title: Environmental Program Evaluation TIG Business Meeting and Preview of the Fieldtrip to the Herring Run Watershed Association Project
|
|
Business Meeting Session 415 to be held in Edgar Allen Poe Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Environmental Program Evaluation TIG
|
| TIG Leader(s): |
|
Katherine Dawes,
United States Environmental Protection Agency,
dawes.katherine@epa.gov
|
|
Annelise Carleton-Hug,
Trillium Associates,
annelise@trilliumassociates.com
|
|
Lisa Flowers,
University of Montana,
flowers@boone-crockett.org
|
| Chair(s): |
|
Katherine Dawes,
United States Environmental Protection Agency,
dawes.katherine@epa.gov
|
| Discussant(s): |
|
Matthew Birnbaum,
National Fish and Wildlife Foundation,
matthew.birnbaum@nfwf.org
|
|
Session Title: Applying Binary Logistic, Ordered and Unordered Multinomial Logistic Regression Models to Illuminate Learning From Evaluation in Practice
|
|
Demonstration Session 416 to be held in Carroll Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Social Work TIG
|
| Presenter(s): |
| Mansoor Kazi,
University at Buffalo,
mkazi@buffalo.edu
|
| Tom Nochajski,
University at Buffalo,
thn@buffalo.edu
|
| Carrie Petrucci,
California State University, Los Angeles,
cpetruc@calstatela.edu
|
| Abstract:
This demonstration will illustrate new data analysis tools drawn from both the efficacy and epidemiology traditions to investigate patterns in relation to outcomes, interventions and the contexts of practice. The demonstration will include three different applied categorical data analyses: when the outcome is dichotomous (logistic regression), when the outcome has three or more unordered categories (nominal multinomial logistic regression), and when the outcome has three or more ordered categories (ordinal multinomial logistic regression) (Hosmer & Lemeshow, 2000; Kazi, 2003; Jaccard & Dodge, 2004). The presenters will use datasets from their completed evaluations from California, New York and United Kingdom, and discuss real-world applications of the analyses and their contribution to learning from evaluation. The didactic approach will be interactive, guiding the workshop participants through the requirements and limitations of each method, and demonstration its use from practice examples, e.g. human service evaluations of driving whilst intoxicated (DWI) programs.
|
|
Session Title: Evaluation: A Tool for Planning, Implementation, and Organizational Development
|
|
Multipaper Session 417 to be held in Pratt Room, Section A on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Chair(s): |
| Nino Saakashvili,
Horizonti Foundation,
nino.adm@horizonti.org
|
|
How can International Non-governmental Organizations (INGOs) use Metaevaluations for Accountability and Learning?
|
| Presenter(s):
|
| Jim Rugh,
CARE International,
jimrugh@mindspring.com
|
| Abstract:
InterAction is calling for greater accountability for effectiveness by its member INGOs through better evaluation policies and practices. Among the recommended practices is that agencies should conduct credible evaluations of their projects and programs, systematically collect them and make them accessible to interested stakeholders, and to periodically synthesize the findings in order to report on effectiveness on a global level. But this is a huge task! This paper will present the experience by the author as he conducted the most recent meta-evaluation/synthesis of CARE evaluation reports.
It will include practical descriptions of the process of collecting, summarizing, and examining individual evaluation reports, then techniques for aggregating findings both for quantitative statistics and qualitative analysis, and composing a report that is helpful to senior staff and other interested stakeholders.
The main audience for this paper will be others with INGO headquarters M&E responsibilities, or their consultants.
|
|
An Endeavor to Improve Organizational Learning and Quality Within an International Development Organization
|
| Presenter(s):
|
| Mary Picard,
CARE International,
picardm2002@yahoo.com
|
| Abstract:
This paper relates the experience of one of the largest international non-governmental organizations operating in Bangladesh with organizational learning and institutional mechanisms to improve the practice and quality of its development programs. CARE Bangladesh began this recent endeavor by creating a unit tasked with developing a strategy to foster a learning environment with the requisite organizational changes that would lead to better practice and higher quality information. This experience illuminates the challenge of transforming a large-scale, hierarchical organization with multiple field offices and a project-based operating frame to one that uses forms of reflective practice and team-based work to generate and share knowledge from its field experience effectively with a range of actors. The strategy departs from the view that change processes addressing organizational culture, staff incentives, accountability and shared responsibility for the changes, multiple forms of group learning, and the innovations in its core business need to occur in tandem.
|
|
Catalyzing Design, Monitoring and Evaluation (DM&E) for Strategic Program Direction in the Transition From Relief to Development: Practical Lessons for Non-governmental Organizations (NGO)
|
| Presenter(s):
|
| Nicole Demestihas,
Mercy Corps,
ndemestihas@lk.mercycorps.org
|
| Scott Chaplowe,
American Red Cross,
schaplowe@amcrossasia.org
|
| Abstract:
With the magnitude and high profile of recent large-scale disasters, monitoring and evaluation (M&E) of resultant relief and recovery programs has received increasing attention. In post-tsunami Sri Lanka, the demand for accountability increased as the emergency phase transitioned into recovery and development. However, during the initial set-up and scale-up period, there is often little emphasis on M&E systems and processes.
This paper will examine key lessons from Mercy Corps' experience creating a comprehensive design, monitoring, and evaluation (DM&E) system in post-tsunami Sri Lanka. DM&E played a critical role in strategic planning, streamlining indices and indicators to monitor and measure results, and engineering a purpose-built data management IT system. Mercy Corps learned that the space between relief and development provides a valuable opportunity to institutionalize M&E principals and practices and set a clear direction for appropriate and accountable results.
|
|
Evaluating Cooperatives Using Organizational Capacity Building Index
|
| Presenter(s):
|
| Frank Valdivia,
Land O'Lakes International Development Division,
fvaldivia@landolakes.com
|
| Chung Lai,
Land O'Lakes International Development Division,
ctlai@landolakes.com
|
| Peter Ngoma,
Land O'Lakes International Development Division,
pngoma@landolakes.co.mw
|
| Michael Odumbe,
Land O'Lakes International Development Division,
michael@landolakes.co.ke
|
| Abstract:
In international development, organizing people into cooperatives is a common initial activity, with a viable operating cooperative being an expected result of this process. To monitor cooperative management and functions, Land O'Lakes International Development Division has adapted a Capacity Building Tool in two of our programs. Our programs have customized it to their situation and needs. The end product is an organizational capacity building index that is a comprehensive tool used to monitor and measure the progress of cooperatives. This tool has been used to monitor cooperative development in our program in Kenya for the past two years, and is being implemented in our program in Malawi. The main lessons learned were to: tailor cooperative development efforts to specific cooperatives, produce and share detailed information at the cooperative level, and promote the participation of cooperative members in the development process.
|
| | | |
|
Session Title: Current Approaches to Evaluating Programs for Children, Youth and Parents
|
|
Multipaper Session 418 to be held in Pratt Room, Section B on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Human Services Evaluation TIG
|
| Chair(s): |
| Mandi Larsen,
Safe Horizon,
mlarsen@safehorizon.org
|
| Discussant(s): |
| Vajeera Dorabawila,
New York State Office of Children and Family Services,
vajeera.dorabawila@ocfs.state.ny.us
|
|
Assessing the Effectiveness of the Baby&Me Tobacco Free Program
|
| Presenter(s):
|
| Anne Gadomski,
Bassett Research Institute,
anne.gadomski@bassett.org
|
| Nancy Tallman,
Bassett Research Institute,
nancy.tallman@bassett.org
|
| Laurie Adams,
Chautauqua County Health Department,
lauriea@alltel.net
|
| Abstract:
This three year study is designed to evaluate the effectiveness of the Baby&Me Tobacco Free Program, a diaper incentive program, in facilitating post-partum smoking cessation. The settings include 22 sites (WIC offices, obstetric and other clinics) in 9 counties in New York State. The goal of the program is to help pregnant and post-partum smokers to quit smoking. The intervention includes 4 face-to-face contacts prenatally with a cessation specialist who does counseling and carbon monoxide testing. After the baby is born, mothers are tested monthly for carbon monoxide and, if negative, receive vouchers for diapers every 3 to 4 weeks post-partum for one year. As a validation measure, saliva cotinine is measured among a random sample of women who test carbon monoxide negative. Primary outcome measures include biochemically validated monthly abstinence rates ascertained by voucher receipt and a self reported quit rate at 12 months post-partum ascertained by telephone interview.
|
|
Development of an Observational Measure to Assess Staff Implementation of a Residential Program for Youth With Emotional and Behavioral Problems
|
| Presenter(s):
|
| Kristin Duppong Hurley,
University of Nebraska, Lincoln,
kdupponghurley2@unl.edu
|
| Tanya Shaw,
Girls and Boys Town,
shawt@boystown.org
|
| Annette Griffith,
University of Nebraska, Lincoln,
annettekgriffith@hotmail.com
|
| Ron Thompson,
Girls and Boys Town,
thompsonr@girlsandboystown.org
|
| Abstract:
This presentation will focus on the development and preliminary psychometrics of an implementation assessment tool for a multi-faceted human services intervention with at-risk youth. This paper will provide a potential model for developing implementation assessment instruments, as well as the preliminary psychometrics and perceived utility of the instrument to program staff.
|
|
Impacts, Implementation, and Processes: Lessons Learned From the Evaluation of the Girls Inc Preventing Adolescent Pregnancy Program
|
| Presenter(s):
|
| Anu Rangarajan,
Mathematica Policy Research Inc,
arangarajan@mathematica-mpr.com
|
| Susan Zief,
Mathematica Policy Research Inc,
szief@mathematica-mpr.com
|
| Heather Johnston Nicholson,
Girls Incorporated,
hjnicholson@girls-inc.org
|
| Kristin Adams,
Girls Incorporated,
kadams@girls-inc.org
|
| Abstract:
This presentation will contribute to the accumulating evidence on the effectiveness of teen pregnancy prevention strategies by sharing the findings from the rigorous Mathematica Policy Research Girls Shape the Future Study of the Girls Inc. Preventing Adolescent Pregnancy program. For this experimental design study, 835 girls were randomly assigned to the program or a control group at five sites. The presentation will discuss program impacts on exposure to topics in health and sexual education, sexual attitudes and norms, and other risk behaviors that may be antecedents of early sexual activity. The findings will be contextualized using data from the qualitative process evaluation of the five sites of study that include an investigation of program attendance. Researchers will also discuss the implementation of this experimental design study at five sites working within varying constraints on program implementation, and will share lessons learned for future similar evaluations.
|
|
Using Outcomes Measurement and Case Studies to Evaluate an After School Children's Health Initiative: Procedures and Lessons From the Los Angeles' Better Educated Students for Tomorrow (LA's BEST) Internal Evaluation of BEST Fit
|
| Presenter(s):
|
| James Sass,
LA's BEST After School Enrichment Program,
jim.sass@lausd.net
|
| Sandra Kennedy,
LA's BEST After School Enrichment Program,
sandrak@lasbest.lausd.net
|
| Edith Ballesteros,
LA's BEST After School Enrichment Program,
edithb@lasbest.lausd.net
|
| Abstract:
The LA's BEST After School Enrichment Program introduced the BEST Fit children's health initiative in 2003 and began wide-scale implementation in 2005. As of 2007, BEST Fit has provided nutrition education and physical activity to approximately 10,000 children in more than 100 elementary schools in Los Angeles. Initial challenges in developing a monitoring and evaluating plan included identifying common outcomes across independently developed program components, determining outcomes and indicators appropriate for an eight-week program, being responsive to the needs of different funders, minimizing evaluation burdens on children and the organization, and providing information useful for program improvement. The design has involved basic program monitoring, outcomes measurement for child and adult participants, and case studies. This paper will address lessons learned in developing evaluation and reporting procedures for a new children's health awareness initiative. Specific topics will include inter-departmental collaboration and application of the Success Case Method to promote program improvement.
|
| | | |
| In a 90 minute Roundtable session, the first
rotation uses the first 45 minutes and the second rotation uses the last 45 minutes. |
| Roundtable Rotation I:
Conducting Independent Evaluations: Taking a Proactive Step From Student to Professional |
|
Roundtable Presentation 419 to be held in Douglas Boardroom on Thursday, November 8, 3:35 PM to 5:05 PM
|
| Presenter(s):
|
| Thomas Sawyer,
University of Rhode Island,
tsaw5413@postoffice.uri.edu
|
| Crystelle Egan,
University of Rhode Island,
crystelleann@yahoo.com
|
| Jessica Nargiso,
University of Rhode Island,
jnargiso@mail.uri.edu
|
| Abstract:
Graduate students training for a career in evaluation can enhance their skills and build their credentials by conducting independent evaluations. Three psychology graduate student researchers with 3 to 5 years experience in evaluation lead an in-depth discussion with other graduate students about the benefits and challenges of conducting independent evaluations. We have each worked on state-level evaluations of substance abuse prevention programs and conducted our own independent evaluations. We begin by briefly discussing our experience evaluating small-scale youth development and substance abuse prevention programs before facilitating discussion among the roundtable attendees. The issues we have encountered include: charging what we're worth verses working pro bono to gain experience, applying for grants to supplement small budgets, developing evaluation plans, time management, and working with evaluation naïve organizations. We see this round table as an opportunity to share what we've learned and to discuss pertinent issues with other novice evaluators.
|
| Roundtable Rotation II:
Ethical Dilemmas: A Case Study of One Graduate Student |
|
Roundtable Presentation 419 to be held in Douglas Boardroom on Thursday, November 8, 3:35 PM to 5:05 PM
|
| Presenter(s):
|
| Stephen Hulme,
Brigham Young University,
byusnowboarder@yahoo.com
|
| Cary Johnson,
Brigham Young University,
cary_johnson@byu.edu
|
| David Williams,
Brigham Young University,
david_williams@byu.edu
|
| Abstract:
The Guiding Principles (AEA, 2004) of the American Evaluation Association clearly define ethical conditions that all evaluators should adhere to. However, ethical situations are tricky, and there frequently is no “right or wrong”, “black or white” answer. Often, evaluation decisions involving ethics operate in “grey areas” and are difficult to make. The purpose of this roundtable is to explore an ethical dilemma which arose during an internal evaluation to better understand what evaluators must deal with in similar situations and to collaborate with participants in learning how to deal better with situations like this.
|
|
Session Title: Capturing Theoretical Underpinnings of Programs Through Qualitative Inquiry
|
|
Multipaper Session 420 to be held in Hopkins Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Qualitative Methods TIG
|
| Chair(s): |
| Janet Usinger,
University of Nevada, Reno,
usingerj@unr.edu
|
| Discussant(s): |
| Janet Usinger,
University of Nevada, Reno,
usingerj@unr.edu
|
|
Using a Qualitative Approach to Map Beliefs in Education Policy
|
| Presenter(s):
|
| Steve Canavero,
University of Nevada, Reno,
scanavero@gmail.com
|
| Janet Usinger,
University of Nevada, Reno,
usingerj@unr.edu
|
| Bill Thornton,
University of Nevada, Reno,
thorbill@unr.edu
|
| Abstract:
The initial Lasswellian conception of a policy science represented a multidisciplinary approach which included norms and values. However, over the past 30 years the practice has been narrowly defined as empiricist, technocratic, and rationalistic. While there is no one post empiricist perspective, there is a convergent set of themes and approaches that define interpretive policy analysis. Inherent in this emerging methodological approach is an emphasis on values, beliefs, and ideas which under gird policy solutions. This paper will focus on how evaluators may apply this qualitative methodology to map interpretive communities which will enable a thoughtful evaluation of the policy or program. The presenter will use an example of an interpretive policy analysis of charter school legislation to guide the audience through this methodology.
|
|
Learning from History: A Method of Theory Reconstruction for Approximating Program Theory of Non-living Authors
|
| Presenter(s):
|
| Annalisa Raymer,
Cornell University,
alr26@cornell.edu
|
| Paula Horrigan,
Cornell University,
phh3@cornell.edu
|
| Abstract:
The value of articulating program theory is widely recognized and promoted, even when evaluators must retroactively construct the implicit reasoning, assumptions and mechanisms in cases where the initiative commenced without a complete elaboration. Evaluation literature offers approaches for theory reconstruction including: language/action; argumentation; policy-scientific, strategic assessment, and elicitation. While the last type, elicitation, can include any number of techniques for engaging stakeholders in collectively creating cognitive content of any kind, nearly all of the theory reconstruction methods discussed today require input from living people. Member-checks and participation are irreplaceable; yet how might theory reconstruction be accomplished when the subject of interest is the work of an individual now deceased?
Answers can be found within the toolbox for qualitative analysis of textual data. Employing a network approach utilizing Atlas.ti and coding for elements of theory, researchers were able to approximate the working theory of an American pioneer in community planning. The same method might be applied to seminal works in any discipline.
|
|
Using Interview Excerpts to Illustrate and Personalize a Program's Implementation Theory
|
| Presenter(s):
|
| Jennifer Jewiss,
University of Vermont,
jennifer.jewiss@uvm.edu
|
| Abstract:
“Implementation theory … focuses on the delivery of program services” (Weiss, 1998, p. 58) and intertwines with participants' responses to program activities to form the overarching theory of change. One of the goals of an innovative health and human service program was to develop an inter-organizational outreach model that would increase community members' access to information and services from area providers. An extremely limited evaluation budget posed major challenges. For example, how could the evaluation portray this highly individualized approach and “bring it to life” within such budgetary confines? After facilitating structured group discussions with staff and representatives from partner organizations to outline the implementation theory, interviews were conducted with several program participants to document success stories. Vignettes composed from interview excerpts helped illustrate and personalize the principles embedded in the theory. This paper explores various benefits and drawbacks of using targeted interview excerpts to illustrate and personalize program theory.
|
| | |
|
Session Title: Lessons From the Field: Zambian Case Study
|
|
Panel Session 421 to be held in Peale Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Chair(s): |
| Alice Willard,
Independent Consultant,
willardbaker@verizon.net
|
| Abstract:
Learning is an essential component in the development cycle; the essence of learning implies changing the way one does business, or implements a particular project/program, being adaptive through reflection. This implies that throughout the life cycle of a project or program, the overall orientation of any M&E system should be geared towards learning with an emphasis on continuous improvement, by identifying best practices, highlighting opportunities and enhancing competencies of all implementing partners. To maximize the benefits of organizational learning it is important that all stakeholders, inclusive of beneficiaries, are fully integrated into the process. With a strategic focus on the need for learning and the resources to accomplish it, CRS will be able to share its best practices and successful models more widely while finding innovative ways to keep staff members learning and sharing what they learn. As an agency CRS stands to learn much from the Zambian experience.
|
|
Step by Step: Key Points and Audiences for Learning in a Field Monitoring and Evaluation (M&E) System
|
| Alice Willard,
Independent Consultant,
willardbaker@verizon.net
|
|
The basic cycle of assessment, design, implementation, monitoring, and evaluation (ADIME) has become axiomatic in development projects. International evaluators must be adept at each aspect of the cycle, whether they are part of the institution and can follow a project throughout the cycle, or whether they are more transient consultants in the cycle. No matter the locus of the evaluator, there are often missed opportunities for learning at each moment in the cycle. The type and opportunity of learning vary with the particular audience the evaluator encounters: project staff, clients, local partners, donors. This paper identifies key points for learning, and discusses the types of learning possible for different audiences. The author will present options and opportunities for learning for those different points in, and audiences for, the basic M&E system.
|
|
|
Catholic Relief Service (CRS)/Zambia Impact of Lessons From Rigorous Evaluation
|
| James Campbell,
Catholic Relief Services,
jcampbell@crszam.org.zm
|
| Shannon Senefeld,
Catholic Relief Services,
ssenefeld@crs.org
|
|
Stringent evaluation criteria are necessary in any evaluation that seeks to attribute impact or outcome to a program intervention. These evaluations enable international development organizations to appropriately plan and implement programs with targeted interventions. However, the very process of conducting a stringent evaluation in many rural areas can lead to unexpected outcomes and learning.
In 2005, Catholic Relief Services (CRS) began a six-month evaluation to determine the impact of nutritional supplements on anthropometric status and quality of life of HIV positive clients in rural Zambia, using a three-armed approach (two experimental groups and one control). The evaluation relied on existing local partner networks for data collection and verification. Throughout the evaluation, CRS and partners learned valuable lessons regarding the implementation of such a rigorous exercise in a resource poor rural environment. This presentation highlights the key lessons and opportunities for learning for staff and participants.
| |
|
Learning by Design: SUCCESS II and Partner Participation in the Monitoring and Evaluation Design
|
| Thomas Moyo,
Catholic Relief Services,
tmoyo@crszam.org.zm
|
| Alice Willard,
Independent Consultant,
willardbaker@verizon.net
|
|
In 2007, CRS/Zambia contracted an overhaul of the M&E system for one of their HIV/AIDS projects. The project had been renewed and expanded after an initial implementation cycle. At this time, both the donor and CRS wanted more outcome-level indicators, yet worked with partners quite resistant to the addition or complication of already contentious reporting requirements. As part of identifying the moments of learning in the implementation cycle, the consultant worked closely with staff and partners to identify capacity and training requirements. One key element in the design was to establish particular mechanisms for all of the participants of the 'information food chain' to learn what the data meant, and what the data could mean to their own management decisions. This presentation describes the process of setting up an embedded learning function as a specific part of the M&E system, rather than an afterthought to the reporting cycle.
| |
|
Session Title: Classroom Experiences and the Teaching of Evaluation
|
|
Multipaper Session 422 to be held in Adams Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Teaching of Evaluation TIG
|
| Chair(s): |
| Katye Perry,
Oklahoma State University,
katye.perry@okstate.edu
|
|
Evaluation of an Evaluation Course: Lessons Learned
|
| Presenter(s):
|
| Rama Radhakrishna,
Pennsylvania State University,
brr100@psu.edu
|
| Prosper Doamekpor,
Pennsylvania State University,
pkd117@psu.edu
|
| Abstract:
This paper describes the evaluation of an introductory graduate level course in program development and evaluation taught at a major land-grant university in the northeastern United States. The authors present how the course evolved over a six-year period. Specifically, the authors share the process in the development of course, course objectives, content taught, assignments, tests, presentations from students, and criteria for grading. Three sets of evaluation data were collected over a six-year period. These included: mid-semester feedback, end of course/semester evaluation collected by the university, open-ended comments, and peer evaluations. Overall students and peers had positive opinions about the course and the instructor. Students indicated that the course materials and presentations were clear and well organized contributing to understanding of evaluation concepts. Students also indicated the importance and value of the course in learning about evaluation concepts and the application of those concepts to specific programs and/or real life situations.
|
|
Effective Critical Thinking Teaching Strategies as Perceived by Program Evaluation Faculty
|
| Presenter(s):
|
| Dhaifallah Almatrodi,
Western Michigan University,
matrodi@hotmail.com
|
| Brooks Applegate,
Western Michigan University,
brooks.applegate@wmich.edu
|
| Abstract:
A need for improving student instruction and outcomes in graduate evaluation training programs has been pervasively documented in the scholarly literature. To meet this need, this study investigated teaching strategies and associated instructional techniques teachers use to promote critical thinking. A mixed method research design employing a 33-item survey and follow-up interview that assessed how teachers used different teaching strategies was administered to 37 university faculty members affiliated with two graduate evaluation training programs. The survey quarried faculty in their knowledge and use of 33 different strategies for developing critical thinking. The interview targets how these instructors implement the different strategies, what student outcomes are expected, and how these outcomes are best assessed. Data will be examined from both quantitative and qualitative analytical perspectives. The net result of this study will be a working, hypothetically efficacious model of critical thinking teaching strategies for use in graduate evaluation programs.
|
|
Teaching Evaluation Graduate Courses Online: Challenges and Alternatives
|
| Presenter(s):
|
| Koralalage Jayaratne,
North Carolina State University,
jay_jayaratne@ncsu.edu
|
| Abstract:
There is a growing demand for on-line graduate level evaluation courses from full-time professionals. Universities have started to respond to this demand. Delivering online courses is a challenge for instructors as well as students due to asynchronous communication limitations technological variability. How can we establish synchronous communication between instructors and students? What are the other issues and challenges associated with delivering instruction online? What are alternatives to meeting these challenges? Finding answers to these questions is the focus of this paper presentation. This paper presentation is based on information gathered from faculty members who have experience in delivering online courses and literature. This is a timely topic for educators who face the reality of meeting the demand for online courses. It contributes to the evaluation practice by preparing new evaluation professionals.
|
| | |
| Roundtable:
Applying Guskey's Model to a Reading Professional Development Program for K-3 Teachers |
|
Roundtable Presentation 423 to be held in Jefferson Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
| Presenter(s):
|
| Rolanda Bell,
Arizona Department of Education,
rbell@ade.az.gov
|
| Brian O'Connell,
Arizona Department of Education,
brian.o'connell@azed.gov
|
| Allison Larsen,
Arizona State University,
allisonlarsen22@yahoo.com
|
| Abstract:
In an effort to promote effective research based instructional strategies for reading, Arizona teachers were afforded the opportunity to participate in either an online or an instructor led professional development program for reading. A mixed-methods evaluation investigated the impact of these programs on student learning and teachers' use of instructional strategies in the following reading components: phonic/phonemic awareness, vocabulary, comprehension, and fluency. The results indicated that the mode of delivery did not significantly impact teachers' use of the various instructional strategies; however, the students enrolled in the classrooms of teachers participating in the instructor led professional development program obtained significantly higher scores on the state assessment in reading.
|
|
Session Title: Focusing on Exposure as a Central Construct in Health Campaigns Research: Experiences From Entertainment-Education and Other Interventions in Africa
|
|
Panel Session 424 to be held in Washington Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Health Evaluation TIG
|
| Chair(s): |
| Rajiv Rimal,
Johns Hopkins University,
rrimal@jhsph.edu
|
| Discussant(s):
|
| Tara Sullivan,
Johns Hopkins University,
tsulliva@jhsph.edu
|
| Abstract:
Although exposure to campaign messages is critical for intervention success, developing effective, reliable, and valid instruments for measuring exposure is critical for demonstrating success. This panel is organized around the common theme of innovative approaches to the measurement of exposure in national, mass media based campaigns. Illustrations are drawn from experiences in Egypt, South Africa, and Malawi.
|
|
Character Recognition as a Reliable Measure of Exposure to Alam Simsim Among Preschool Children in Egypt
|
| Maria Elena Figueroa,
Johns Hopkins University,
mefiguer@jhuccp.org
|
|
For the evaluation of Alam Simsim (the Egyptician version of Sesame Street) two measures of exposure were developed. A comprehensive scale measure including six self-reported items such as programs watched and favorite programs was computed and compared to a simpler measure based on the recognition of the main characters of the program. The reliability of the scale comprising the six measures was 0.79 (alpha). For the second measure, children were shown pictures of the three main characters and asked to point to each when their name was provided by the interviewer. The reliability of this measure was 0.87. Both measures were used in ordinary least square regression to assess the effect of Alam Simsim on numeracy and literacy skills and on gender attitudes among children 4 to 6 years old in Egypt. The results were similar and statistically significant for each outcome, thus attesting to the measure's validity.
|
|
|
Development of a Continuous Measure of Exposure to a Television Drama to Help Prevent AIDS in South Africa
|
| Lawrence Kincaid,
Johns Hopkins University,
lkincaid@jhuccp.org
|
|
For the evaluation of the national TV drama, Tsha Tsha, in South Africa, a continuous measure of recall was developed that was independent of the hypothesized social outcomes of the drama. Respondents were asked to look at photos of the four main characters from the drama and 'unaided' give their correct names. They were also given a test of knowledge about features of the drama that were unrelated to the health issue of HIV/AIDS or the expected behavioral outcomes. Then they were given a list of the dominant event featured in each episode and asked if they remembered seeing that particular episode. The reliability of the scale comprised of these three measures was 0.93. Because the final measure was continuous, it was possible to determine if a dose response to the drama was operating versus a threshold effect (some minimal level of recall before any effect is possible).
| |
|
Measuring Types of Exposure and Their Effects as a Function of Health Competence: The Integrated Communication for Healthy Living Project in Egypt
|
| Douglas Storey,
Johns Hopkins University,
dstorey@jhuccp.org
|
|
This presentation describes the challenges of measuring exposure to an integrated multimedia plus community-based healthy lifestyles project in Egypt (Communication for Healthy Living-CHL) and attributing to it changes in nine different areas of health behavior from family planning to passive smoking. Using a health competence framework (based on a synthesis of health literacy and social capital perspectives), the paper examines the relationship between an individual's level of health competence and six types of program exposure measures ranging from general (media habits) to specific (self-reported learning from recalled CHL messages). Data derived from a population-based 2004-2005 panel survey of 2073 married women aged 15-49, 1891 husbands, and 1716 never married youth aged 15-24 show different patterns of exposure by gender, age, marital status and level of health competence as well as interaction effects of health competence and exposure on the number of positive health behaviors practiced.
| |
|
The Radio Diaries Program in Malawi Designed to Reduce HIV Stigma: Evaluating the Different Measures of Exposure
|
| Rajiv Rimal,
Johns Hopkins University,
rrimal@jhsph.edu
|
|
The Radio Diaries program is broadcast nationally on six stations in Malawi. Each week, one of the two HIV-positive diarists in each station discusses his or her experience living as an HIV-positive person. The goal of the program is to reduce stigma toward people living with HIV. This presentation discusses the effects of the Radio Diaries program, almost a year after the broadcasts began. It discusses the challenges in measuring exposure to the program, and it introduces three separate measures of program exposure: (a) number of episodes listened to, (b) number of times tuned in to the program, and (c) the style of listening. Although the inter-correlations among the three items were high (r's ranging from .71 to .77, all ps < .001), their ability to predict stigma varied considerably. The presentation will discuss how the quantity and quality of exposure may differ from each other.
| |
|
Session Title: Learning From Organizational Level Evaluations
|
|
Multipaper Session 425 to be held in D'Alesandro Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Chair(s): |
| Sheridan Green,
JVA Consulting LLC,
sheridan@jvaconsulting.com
|
|
Framing an Evaluation of a Nonprofit Community Based Organization: A Comparison of Two Delphi Studies
|
| Presenter(s):
|
| Monica Geist,
University of Northern Colorado,
monicageist@comcast.net
|
| Abstract:
Of the many challenges of evaluating nonprofit community based organizations, there are two major challenges that can be lessened by using the Delphi method. The first challenge is that of small evaluation budgets. The second is getting stakeholders together to discuss and come to agreement on goals and concerns in the organization. The Delphi method is a set of iterative questionnaires, with controlled feedback, that allow participants to give their opinions and interact with other participants without having to attend a face-to-face meeting. By using the Delphi method, evaluators can frame the evaluation before the first site visit. This paper discusses the comparison of two versions of the Delphi method that were administered to a nonprofit organization that is a two-year self-sufficiency program for teen mothers. One version was the paper-pencil version administered via postal mail. The other version was a real-time computer version administered via the web.
|
|
Resistance to Learning From Evaluation in the Context of Non-profit Organizations
|
| Presenter(s):
|
| Luba Botcheva,
The Children's Health Council,
lbotcheva@chconline.org
|
| Julie Slay,
The Children's Health Council,
jslay@chconline.org
|
| Lynne Huffman,
The Children's Health Council,
lhuffman@chconline.org
|
| Abstract:
The presentation will focus on non-profit organizations and will address resistance to learning from evaluation at three levels: service providers, program, and organization. Learning is conceptualized as a form of change that can be resisted as often as it is embraced. This is especially true in the context of evaluation where “learners,” those who are expected to use evaluation findings and consider changing, may not be motivated to learn and are forced by external factors to change. Learners may regard knowledge as threatening; their own perceived level of competence may be challenged by new information. How can evaluators create an environment that facilitates learning and that provides a safe opportunity to change?
Several case studies of summative and formative evaluation will be presented based on our experience as internal and external evaluations for non-profit organizations. Various techniques that we used to work with resistance to learning will be discussed.
|
|
Development and Use of a Comprehensive Measure of Nonprofit Organizational Capacity
|
| Presenter(s):
|
| Sheridan Green,
JVA Consulting LLC,
sheridan@jvaconsulting.com
|
| Robin Leake,
JVA Consulting LLC,
robin@jvaconsulting.com
|
| Veronica Gardner,
JVA Consulting LLC,
v@veronicagardner.com
|
| Abstract:
JVA Consulting, LLC developed a capacity-building program for community- and faith-based organizations with funding from the U.S. Department of Health and Human Services. The program was designed to increase organizational effectiveness in many areas of capacity including leadership, organizational development, programming, funding and community engagement. The study purposes were to validate an assessment tool and to evaluate organizations' capacity gains. Research questions included: 1) were there significant gains in organizational effectiveness? And, 2) were there differences in gains based on the type or amount of capacity building provided? Data from the first three years indicate that organizations demonstrated significant gains in organizational effectiveness. Organizations receiving individualized technical assistance showed significantly greater gains than those attending workshops only. Results revealed that the amount of technical assistance was a significant predictor of gains achieved by participants. Chronbach's alpha for the instrument was .97. These results and current study analyses will be presented.
|
|
Evaluating Technical Assistance Services Provided to Grantees of Federal Agencies: Approaches of and Lessons From the MayaTech Model
|
| Presenter(s):
|
| Kimberly Jeffries Leonard,
The MayaTech Corporation,
kjleonard@mayatech.com
|
| Mesfin S Mulatu,
The MayaTech Corporation,
mmulatu@mayatech.com
|
| James Bridgers,
The MayaTech Corporation,
jbridgers@mayatech.com
|
| Darren Fulmore,
The MayaTech Corporation,
dfulomre@mayatech.com
|
| Wilhelmena Lee-Ougo,
The MayaTech Corporation,
wlee-ougo@mayatech.com
|
| Abstract:
Technical assistance [TA] contracts are increasingly becoming critical mechanisms through which federal agencies accomplish their objectives efficiently and cost-effectively. Evaluation of these services is important to establish effectiveness and contractual accountability. The Baywatch Corporation had developed a model – Targeted Evaluation Assistance Model (TEAM) – to evaluate its technical assistance services to grantees of federal agencies. The TEAM is a multimethod and multistage evaluation model and approaches TA as a tripartite (federal agency, grantees, and contractors/subcontractors) interactive process. Evaluation takes place in three stages of the TA process: 1) identification and prioritization of TA needs; 2) design and delivery of TA; and 3) output and impact of TA. Multiple electronic tracking systems are used to record interactions between the parties; qualitative and quantitative approaches are used to collect pertinent data from multiple sources appropriate for each stage of TA. Lessons learned from MayaTech's TA contract with SAMHSA's Division of Systems Improvement would be presented.
|
| | | |
|
Session Title: Utilizing Hierarchical Linear Modeling in Evaluation: Concepts and Applications
|
|
Demonstration Session 426 to be held in Calhoun Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Presenter(s): |
| Christopher McKinney,
University of Northern Colorado,
christopher.mckinney@mhcd.org
|
| Antonio Olmos,
Mental Health Center of Denver,
antonio.olmos@mhcd.org
|
| Kate DeRoche,
University of Northern Colorado,
kathryn.deroche@mhcd.org
|
| Abstract:
Hierarchical Liner Modeling (HLM) has become increasing popular in social and behavioral science fields. Evaluators who perform research within an organization and across organizations find this technique especially useful. The presentation will guide the participants through the major concepts of HLM and it's associated strengths, such as unbalanced data and missing values. Utilizing an example from the mental health field the processes of preparing data, running the model, model fit, and interpretation of the output will be discussed. A comparison to more common models, such a multiple linear regression and ANOVA will be utilized to demonstrate the effectiveness of HLM in the estimation of effects. Lastly discussion on other model types such as multinomial and logistic regression will be discussed to provide participants an understanding of HLM's flexibility. Throughout the presentation emphasis will be placed on interpreting and reporting HLM concepts and results to stakeholders and other interested parties.
|
|
Session Title: Simultaneous Learning: Four Perspectives From a University-based Evaluation Center
|
|
Multipaper Session 428 to be held in Preston Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Organizational Learning and Evaluation Capacity Building TIG
|
| Chair(s): |
| Gregory Diggs,
University of Colorado, Denver,
shupediggs@netzero.com
|
| Abstract:
Evaluation centers make significant contributions to the culture of learning. This session describes how a university-based evaluation center is attempting to promote a learning culture that will contribute to the professional skills and insights of its evaluation staff while adding value to the university and the larger community. From four perspectives, participants share their learning about the delivery of quality evaluation services and the impact of their role as evaluators on other dimensions of their professional life. Learning is described from the perspectives of 1) university leaders; 2) new evaluators; 3) teacher educator/evaluator; and 4) policy analyst/educator.
|
|
The University Perspective: What we are Learning About Providing Evaluation Services in the Community
|
| Bonnie Walters,
University of Colorado, Denver,
bonnie.walters@cudenver.edu
|
|
Interviews with key informants in higher education reveal the rationale for a university-sponsored evaluation center created to provide evaluation services to educational institutions and non-profit agencies in the local community. This paper examines how an evaluation center contributes to the learning environment in higher education and in partner organizations. In addition, the author examines logistical, budgetary, and management challenges and the center's subsequent learning from these trials/challenges.
|
|
Cognitive Apprenticeship for New Evaluators
|
| Susan Connors,
University of Colorado, Denver,
susan.connors@cudenver.edu
|
|
Under the auspices of a university-based evaluation center, new evaluators engage in experiential learning while participating in real-life evaluations. This paper describes the processes put into place to scaffold the success of the new evaluators while assuring quality evaluation services to clients. The theoretical model that underlies the processes is based on the principles of cognitive apprenticeship developed by Collins, Brown and Holum (1991). Interviews with new evaluators provide insights into their learning that confirm the value of their apprenticeship and illuminate the challenges/ reinforcement provided by evaluation work in the field.
|
|
Teacher Educator and Evaluator: Reflections on Learning Across Multiple Contexts
|
| Kim White,
Metropolitan State College, Denver,
whiteki@mscd.edu
|
|
Lave and Wenger (1991) position learning as 'an integral and inseparable aspect of social practice' (p.31). Participation in multiple contexts can provide opportunities for learning exchanges as each context informs the other. Situated learning theory provides a framework for investigating the ways in which learners participate in communities of practice and apply gained knowledge and experience to other settings (Brown, Collins, & Duguid, 1989; Choi & Hannafin, 1995). In this paper, the author reflects on her participation in multiple contexts, both as a teacher educator and developing evaluator, and examines the learning that follows. Through the analysis of evaluation notes, teaching materials, and lived experiences, the author found that her teaching practices, curricular decisions, and instructional delivery choices were influenced by what she learned across these contexts.
|
|
Learning From Two Roles: Evaluator and Policy Analyst
|
| Anne Martin,
University of Colorado, Denver,
annelmartin@gmail.com
|
|
In this paper, the author shares her unique perspective from her simultaneous work as a policy analyst and as an evaluator. This juxtaposition of employment provides a lens to learn and experience each organization's goals and potential to affect change. From the perspective of policy analyst, it is crucial that program evaluation informs policy decisions. From the perspective of an evaluator, the potential to influence policy development is compelling. Ideally, the intersection between the policy world and the program evaluation field is a continuous circle of informative growth. As program evaluations uncover results about what practices are most powerful in meeting goals, policy should incorporate these results into resource allocation and program implementation. Subsequently, whatever policies or programs are enacted should be accompanied by evaluation to inform the ongoing process and make summative policy decisions about the future.
|
|
Session Title: Evaluation Research Involving Criminal Justice Interventions
|
|
Panel Session 429 to be held in Schaefer Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Crime and Justice TIG
|
| Chair(s): |
| Winifred L Reed,
United States Department of Justice,
winnie.reed@usdoj.gov
|
| Abstract:
Evaluation research involving criminal justice interventions will be the topic of this session. Recent reports regarding needed improvements in evaluation research will be reviewed in terms of results from the first generation of national evaluations of anti-crime programs. Recent advances in the field of evaluation research will be reviewed as it concerns ongoing and future evaluations at NIJ. A number ongoing evaluation projects will be described including results from recently completed projects. Future plans for evaluation research will be outlined.
|
|
Overview of Evaluation at the National Institute of Justice
|
| Patrick Clark,
National Institutes of Justice,
patrick.clark@usdoj.gov
|
|
Patrick Clark currently serves as Evaluation Advisor in the Office of Research and Evaluation at the National Institute of Justice. He will present an overview of NIJ's Evaluation program including methods being used to improve evaluations at NIJ and current evaluation support at NIJ
|
|
|
Violence and Victimization Evaluations at the National Institute of Justice
|
| Angela Moore Parmley,
National Institutes of Justice,
angela.mooreparmley@usdoj.gov
|
|
Angela Moore Parmley is Chief of the Violence and Victimization Research Division (VVRD) at the National Institute of Justice. She will provide information on the experience of implementing selected evaluations in the violence and victimization areas. In doing so, she will emphasize strategies to aid implementation of evaluations as well as challenges that have occurred in this implementation.
| |
|
Crime Control and Prevention Evaluations at the National Institute of Justice
|
| Winifred Reed,
National Institutes of Justice,
winnie.reed@usdoj.gov
|
|
Winifred Reed is Chief of the Crime Control and Prevention Research Division at the National Institute of Justice. She will provide information on the experience of implementing selected evaluations in the crime control and prevention areas. In doing so, like Dr. Moore Parmley, she will emphasize strategies to aid implementation of evaluations as well as challenges that have occurred in this implementation.
| |
|
Session Title: Continuous Improvement: Examining the Impact of Year-after-Year Evaluations and Improve Learning Products and Evaluation
|
|
Think Tank Session 430 to be held in Calvert Ballroom Salon B on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Business and Industry TIG
|
| Presenter(s):
|
| John Mattox,
KPMG,
jmattox@kpmg.com
|
| Discussant(s):
|
| John Mattox,
KPMG,
jmattox@kpmg.com
|
| Heather Maitre,
KPMG,
hmaitre@kpmg.com
|
| Abstract:
The purpose of this Think Tank is to share a case study about the mutually beneficial long-term relationship between evaluation and instructional design teams and pose a series of questions about the continuous improvement process in other evaluation settings. The end result should be a consensus as to whether a consistent relationship (positive, negative or null) exists between the continuous improvement in the quality of evaluation efforts and the quality of learning products. While learning products (courses) are the focus, the topic is easily expanded to any intervention that is being evaluated.
The case study comes from KPMG's Center for Learning and Development. During each of the past two years, a detailed impact study has been conducted. Improvements were applied during the second year, and the quality of both the evaluation and the course increased. We are interested in quantifying the nature of the improvements through this Think Tank.
|
|
Session Title: Challenges and Issues: What Do We Know About the Role of Evaluation in Organizational Learning? Where Do We Go From Here?
|
|
Think Tank Session 431 to be held in Calvert Ballroom Salon C on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Organizational Learning and Evaluation Capacity Building TIG
|
| Presenter(s):
|
| Rosalie T Torres,
Torres Consulting Group,
rosalie@torresconsultinggroup.com
|
| Abstract:
This think tank will explore issues and challenges attendees have faced in using evaluation to facilitate organizational learning. The chairperson will introduce the topic and describe factors postulated to promote organizational learning. Participants will be invited to add to the list of factors presented (e.g., leadership support, evaluator role in the organization, organizational culture, time for collaboration, evaluator skill/capacity) and will then be invited to share their experiences in small groups, addressing the question: -What experiences have you had (or been exposed to) relating to this factor which created specific challenges for using evaluation to facilitate organizational learning?- Following small group report-outs, participants will form a second round of small groups, each focusing on two questions: -What conclusions can be drawn about these issues and challenges? Where do we go from here?- The think tank will conclude with report-outs and whole-group dialog designed to stimulate further thinking and exchange on the issue.
|
|
Session Title: Professional Communities of Inquiry
|
|
Panel Session 433 to be held in Fairmont Suite on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Helen Apthorp,
Mid-continent Research for Education and Learning,
hapthorp@mcrel.org
|
| Abstract:
This 90-minute panel offers different perspectives on evaluation and research as means of enhancing our individual and collective capacity to create quality learning environments. The three panelists, a researcher, consultant, and writer, work together in educational program development and evaluation. The panelists will present a brief overview of the past and current role of scientific thinking in education, a school-based model for a purposeful community and system of evaluating student work to improve instruction, and tips on how to translate educational research and evaluation results into accessible guidance. The session chair will pose discussion questions and encourage session participants to explore (a) how learning from evaluation and research might be enhanced and sustained for the broadest impact, and (b) how applying models of scientific thinking used in the other professions might both facilitate and hinder learning through evaluation and research in education.
|
|
Learning Through Experiments
|
| Helen Apthorp,
Mid-continent Research for Education and Learning,
hapthorp@mcrel.org
|
|
It is often observed that personal whims, current fads, and advocates greatly influence educational decisions. However, with external pressure from accountability requirements, there appears to be increasing use of evidence and scientific thinking to justify educational decisions. Examples will be drawn on to trace the influence of science in education away from elitist expert opinion to publicly verifiable research conclusions. Current research and evaluation projects will be used to illustrate how rigorous methodological standards can be applied to conducting experimental studies in schools and gaining the knowledge stakeholders want. Dr. Helen Apthorp, Principal Researcher at Mid-continent Research for Education and Learning (McREL), has extensive research and evaluation experience conducting studies on professional development, curriculum, and instruction for school districts, state and federal agencies, and educational publishers.
|
|
|
Learning Through Purposeful Communities
|
| Sammye Wheeler-Clouse,
Mid-continent Research for Education and Learning,
swheeler-clouse@mcrel.org
|
|
It is vital that principals and district administrators see the importance of a consistent process to use across their buildings to evaluate work. This is a three-tiered process: developing instructional leaders, establishing a purposeful community of learners, and systematic analysis of student work to affect instruction. When the vision and expectations are set, and when teachers develop a collaborative working relationship where trust is at the forefront, they begin to 'evaluate' themselves through a reflective process. An administrative model of expectations and evaluation will be presented. Several structures will be offered to organize learning communities within a school in order to openly share student work, measure learning, increase student achievement, and evaluate instruction. As a building and district leader in several Denver School Districts, Dr. Wheeler-Clouse has first-hand experience in developing instructional leaders and implementing protocols for teachers to assess student work. Dr. Wheeler-Clouse is a Principal Consultant at McREL.
| |
|
Learning Through Translation
|
| Kirsten Miller,
Mid-continent Research for Education and Learning,
kmiller@mcrel.org
|
|
Translating research and evaluation results into accessible guidance for practitioners can often seem a hit-or-miss proposition. However, the thoughtful translation of research into print guidance helps ensure that evaluation results lead to the broadest possible impact. This panelist will discuss how to write for a broad knowledge base, from beginning teachers to principals to state administrators and policymakers; how to link research, evaluation, and action; and how best to impart confusing or incomplete results. Kirsten Miller, Sr. Consultant for Publications at McREL, has extensive experience in translating research and evaluation results into prescriptive guidance. Her work has appeared in Principal Leadership Magazine, Phi Delta Kappan, and Principal's Research Review. She received a 2004 Distinguished Achievement Award from the Association of Educational Publishers in the newsletter category, and was a two-time finalist in the journal category. In 2006, she received a MarCom Platinum award for her work on McREL's website.
| |
|
Session Title: Educational Evaluation Internationally
|
|
Multipaper Session 434 to be held in Federal Hill Suite on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Linda Channell,
Jackson State University,
drlinda@bellsouth.net
|
|
Respecting Teacher Autonomy in the Face of Accountability Mandates: Possibilities for Positive Outcomes
|
| Presenter(s):
|
| Sigurlina Davidsdottir,
University of Iceland,
linadav@hi.is
|
| Penelope Lisi,
Central Connecticut State University,
lisip@ccsu.edu
|
| Abstract:
Icelandic schools were mandated in 1996 to evaluate their work. In 2000, two researchers - from Central Connecticut State University and the University of Iceland - initiated a longitudinal study of four Icelandic schools developing self-evaluations. Teachers and leaders formed teams to develop and implement evaluation systems, instead of using summaries of test scores, as is more traditional, based on accountability demands. Results from the first four years of the study (2000-2004) indicated significant respect for teacher autonomy, and teachers perceiving themselves empowered to make critical decisions about teaching and learning in an increasingly collegial and collaborative work culture. In terms of accountability, teachers reported developing greater skill in examining their own teaching practices, and linking better curriculum developments and student achievement. The results of the study appear to indicate that when accountability is addressed with a respect for teacher autonomy, the impact on teaching and learning can be substantial.
|
|
Spreading the Good Word: Strengthening Understanding of Evaluation in New Zealand
|
| Presenter(s):
|
| Graham Stoop,
Education Review Office,
graham.stoop@ero.govt.nz
|
| Abstract:
In the last decade or so, the New Zealand Education Review Office has developed evaluation methodologies for reviewing schools and early childhood services based on a growing understanding of evaluation theory and practice issues. The methodologies use participatory approaches that expect a high level of engagement with stakeholders (students, teachers, principals, parents and communities) while at the same time maintaining objectivity in the processes of qualitative data gathering, analysis and synthesis. Successful implementation requires that all participants – evaluators and stakeholders – have a good understanding of evaluation and how it can help improve the quality of education outcomes and experiences for students.
The Education Review Office has undertaken a programme of engagement with key groups including parents, families and communities, teachers and principals, and government agencies. There has also been a strong emphasis on training educators recruited from schools in the core evaluation skills.
This presentation will explain the activities undertaken – ranging from involving international experts in national training programmes for evaluators to meetings, seminars and workshops conducted with stakeholders on topical issues to improve the understanding and use of evaluation processes and findings.
|
|
The Need for Teacher Evaluation in Early Childhood Education in Korea
|
| Presenter(s):
|
| Kyoung jin Kim,
University of Illinois at Urbana-Champaign,
kkim37@uiuc.edu
|
| Jungmi Kim,
Sahmyook University,
kimjm@syu.ac.kr
|
| Abstract:
There is no nationwide evaluation system for teachers in early childhood education in Korea. To facilitate children's learning and improve teachers' teaching ability, evaluations of teachers are crucial. In order to create effective evaluations, understanding kindergarten teachers' perspectives on evaluations is needed. Therefore, in this research, we seek to understand teachers' perceptions of teacher evaluations, especially the purpose, content, instrument, method, design, role, and use of evaluations. We employed mixed methods in order to extend the scope of our inquiry and to clarify the findings of our study. We surveyed 200 teachers and interviewed 30 teachers to find out their perspectives on teacher evaluation. Major findings suggest that meaningful evaluation tools are needed to encourage teachers to take the initiative to reflect on their teaching. In addition, these tools should become a part of professional development for early childhood educators
|
| | |
|
Session Title: Identifying Challenges to Using Evaluation to Inform Program Management and Public Policy
|
|
Think Tank Session 435 to be held in Royale Board Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Presenter(s):
|
| Connie Chang,
United States Department of Commerce,
cchang@technology.gov
|
| Discussant(s):
|
| Connie Chang,
United States Department of Commerce,
cchang@technology.gov
|
| Rosalie Ruegg,
TIA Consulting Inc,
ruegg@ec.rr.com
|
| Abstract:
This Think Tank is the culmination of a series of five annual think tanks initiated in 2003 that have focused on challenges to evaluation. The initial session identified six categories of challenges: (1) institutional/cultural barriers, (2) methodological problems, (3) data/measurement difficulties, (4) resources issues, (5) communication obstacles, and (6) conflicting stakeholder agendas. Individually, and in combination, these challenges interfere with evaluation effectiveness in measuring and improving program performance, meeting public accountability requirements, and informing program management, public policy strategies, and national resources allocation decisions. The previous sessions focused on barriers to performing evaluation. This final, wrap-up session investigates barriers that impede the use of evaluation to inform program management and public policy. These barriers include resources, communications, institutional, and stakeholder agendas. In a highly interactive session, participants will discuss the barriers and seek solutions.
|
|
Session Title: Substance Abuse Program Evaluation: Productivity Benefits, Peer Mentorship, Drug Courts and Jail Interventions
|
|
Multipaper Session 436 to be held in Royale Conference Foyer on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG
|
| Chair(s): |
| Samantha Williams,
Centers for Disease Control and Prevention,
stw8@cdc.gov
|
|
Going one Step Further: Evaluating the Formation and Impact of a Substance Abuse Peer Mentorship Team
|
| Presenter(s):
|
| Lindsay Gustafson,
Central Alberta AIDS Network Society,
lindsay@turningpointagencies.org
|
| Christopher Cameron,
Calgary Health Region,
christopher.cameron@calgaryhealthregion.ca
|
| Abstract:
Substance abuse is an extremely prevalent social problem in rural communities. The lifestyle that individuals with substance abuse problems lead places them a severe risk for the contraction of various life altering or life threatening diseases, including Hepatitis C and HIV/AIDS. Health professionals are often unable to reach a significant portion of these individuals despite their best efforts. A not-for-profit organization in a small rural town in the province of Alberta recognized the limitations of attempting to serve these individuals on a strictly professional basis. In response to this recognition this organization recruited a small group of individuals who had previously struggled with and overcome substance abuse problems to serve as peer mentors and service providers. An evaluation of the process that this organization engaged in to create and solidify its “peer team” and the outcomes of this process are presented in this paper.
|
|
Evaluating the Productivity Benefits of Substance Abuse Treatment
|
| Presenter(s):
|
| Angela Snyder,
Georgia State University,
angiesnyder@gsu.edu
|
| Abstract:
This evaluation capitalizes on administrative data linkages between the Connecticut Department of Mental Health and Addiction Services and the Connecticut Department of Labor to study employment outcomes after outpatient substance abuse treatment. Results of this study provide the evidence and inputs necessary to perform informed cost benefit analyses of Connecticut's substance abuse treatment system. A quasi-experimental design is used to compare the post-treatment earnings of the substance abuse clients to individuals in Connecticut with similar demographic profiles. Longitudinal methods are utilized to compare earnings trajectories over time for ten quarters post-treatment. Results suggest that while both groups lose earnings in the outcome period, the outpatient treatment clients may make on average $79 dollars per quarter more after treatment than their demographically-matched counterparts.
|
|
Jail Interventions for Individuals With Co-occurring Disorders: A Comparison of Urban and Rural Programs in Pennsylvania
|
| Presenter(s):
|
| Holly Wald,
HPW Associates,
hwald@hpwassoc.com
|
| Cynthia Zubritsky,
University of Pennsylvania,
cdz@mail.med.upenn.edu
|
| Abstract:
Pennsylvania's Co-Occurring Systems Integration Grant (COSIG) focuses on the development of six unique demonstration programs, two of which include forensic programs that have three service delivery systems working together to identify and support individuals with COD (Mental Health, Substance Abuse and Criminal Justice). This presentation will focus on the two forensic programs that screen and assess persons entering the jails; including both an urban and a mixed rural-suburban model.
Individuals in these forensic programs have continually screened positive for COD at a higher rate than in traditional community COD programs. The results indicated that: women were highly over-represented in the treatment population compared males; women screened and assessed at a much higher rate for mental health disorders as their primary disorders; and men screened and assessed at a higher rate for substance abuse disorders. These gender and disorder-based differences will be discussed. Lessons learned regarding use of these findings for program planning will also be described.
|
|
The Rochester Drug Court: A Replicated Evaluation
|
| Presenter(s):
|
| Stephen L Fielding,
University of Rochester,
sfielding@childrensinstitute.net
|
| Rusti Berent,
University of Rochester,
rberent@childrensinstitute.net
|
| Abstract:
While evaluations of drug courts are on the rise, there are few that have been replicated after the passage of time by a second, independent evaluator. This evaluation includes greater social context information and a longer follow-up than the prior. The original evaluation examined participants arrested from 1995 through 1998, the current from participants arrested from 1999 through 2005. Both evaluations are based on a secondary data analysis of New York State DCJS data (sealed and unsealed records) matched with RDTC records, and both utilize propensity score matching to establish the comparison group. We test the hypotheses that drug recidivism rates up to six years post-arrest will be significantly lower among the RDTC participants compared with a similar cohort of criminal defendants, and that this difference will be equal to or greater than the difference between the prior drug court participants and a similar cohort of criminal defendants.
|
| | | |
|
Session Title: Moving Measures: Employing Visual Theater and Movement to Convey and Augment Participatory Learning in Program Evaluation
|
|
Skill-Building Workshop 437 to be held in Hanover Suite B on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Presenter(s):
|
| Geri Peak,
Two Gems Consulting Services,
geri@twogemsconsulting.com
|
| Abstract:
This innovative approach to measurement responds to a client's request to literally convey evaluation results beyond words and diagrams. By creatively expanding participatory action research approaches with techniques from visual theater and dance (inspired by choreographer Vincent Thomas and Quest: Arts for Everyone), stakeholders can combine movements and words to: identify priority program components to evaluate; illustrate program characteristics; or articulate perceived results. Evaluators working with diverse groups can use these techniques as qualitative tools throughout the evaluation cycle. Incorporating movement can prove meaningful in building equity when divergent ages, power levels, language abilities or cultural perspectives dictate stakeholder involvement. Evaluators of all levels will: 1) review the basic premises in relation to the evaluation life cycle, 2) learn basic visual theater exercises, and 3) practice creating and sharing moving measures based on case studies from the presenter's evaluation practice. Wear comfortable clothing; no prior movement experience necessary.
|
|
Session Title: Advancing Valid Measurement in Evaluation
|
|
Multipaper Session 438 to be held in Baltimore Theater on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Karen Given Larwin,
Gannon University,
kgiven@kent.edu
|
|
Scales and Indices: What's the Difference and Does it Matter?
|
| Presenter(s):
|
| Michael Hennessy,
University of Pennsylvania,
mhennessy@asc.upenn.edu
|
| Abstract:
Statisticians and quantitative researchers make a distinction between effect and causal indicators. But does this distinction really matter? To answer this question, we use data from a longitudinal, multi-site intervention study to examine the measurement model, temporal reliability, and associations between two latent constructs: outcome expectancies for condom negotiation and condom use self-efficacy. The former is a latent variable defined by causal indicators whereas the latter is defined by effect indicators. We find that the effect indicators work well while the causal indicator model is problematic. Therefore, we suggest a four step analytic strategy when researchers are confronted with a set of items that may operate as causal indicators.
|
|
Measuring Identity and Identity Change in Evaluations
|
| Presenter(s):
|
| Elaine Hogard,
University of Chester,
e.hogard@chester.ac.uk
|
| Roger Ellis,
University of Chester,
r.ellis@chester.ac.uk
|
| Abstract:
Many programmes to be evaluated have as their objectives changes in the attitudes and behavior of participants. Explicitly or implicitly such objectives bring into play the elusive but significant concepts of self and identity. The evaluation literature has few if any studies that address identity and identity change as a programme outcome. This paper describes a novel identity-measuring instrument known as IDEX and its accompanying software IDIO. IDEX is based on the repertory grid method of George Kelly but focuses on elements related to past, present, future and alternative identities. It is able to detect significant factors regarding an individual's identity and has proved sensitive to change over time thus making it an ideal outcome measure for the evaluation of a programme. The paper gives examples of the use of IDEX in evaluations of a staff development programme in higher education; and a programme to encourage inter-professional collaboration in social and health care.
|
|
Test Evaluation Using an Explanatory Item Response Model
|
| Presenter(s):
|
| Rachael Tan,
University of California, Berkeley,
jinbee@berkeley.edu
|
| Abstract:
Usually we think of assessments as simply tools used in evaluation, but assessments should be the subject of evaluation as well. In the current era of accountability where testing affects every level of the educational system, it is important to evaluate assessments to ensure they are valid and provide an accurate portrayal of student ability. This research uses a hierarchical measurement model to examine whether student and teacher characteristics can help explain test score differences among students. Among other factors, the model includes students' gender, English language proficiency, special education status, and teachers' years of experience teaching the curriculum and assessments. Beginning an evaluation of an assessment at the data level, as in this research, can identify factors that affect student performance, which can help focus a larger evaluation on important elements that contribute to success on assessments, and indicate issues to consider during test design.
|
|
Construction and Interpretation of Composite Variables
|
| Presenter(s):
|
| Katherine McKnight,
Pearson Achievement Solutions,
kathy.mcknight@pearsonachievement.com
|
| Lee Sechrest,
University of Arizona,
sechrest@u.arizona.edu
|
| Abstract:
Composite variables are combinations of variables that are individually meaningful and are thought to be indicators of the same construct. For example, SES is usually a combination of several variables including income, zip code, occupation, educational level, etc. Evaluators frequently use composite variables to measure constructs such as health risks, poverty, ethnicity, quality of life, social support and so on. Although used frequently, composites can be problematic in ways that might not be immediately obvious. Identifying appropriate variables to combine, selecting an appropriate algorithm for combining them and assessing psychometric properties are just some of the problems composites present to evaluators. In this paper, we highlight some problems and offer some alternatives. Because evaluators use composites frequently, it is important to understand the problems associated with the creation and assessment of these variables.
|
| | | |
|
Session Title: Learning From Experience With Random Assignment Experiments: Lessons From Social Security Benefit-Offset Pilot Projects
|
|
Panel Session 439 to be held in International Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| George Julnes,
Utah State University,
gjulnes@cc.usu.edu
|
| Discussant(s):
|
| Mark Green,
United States Social Security Administration,
mark.green@ssa.gov
|
| Joe Entwisle,
National Consortium for Health Systems Development,
jentwisle@hdadvocates.org
|
| Abstract:
Given the recent emphasis on evidence-based practice and the use of rigorous methods to yield the required evidence, it is important for evaluators to appreciate the issues involved in implementing randomized policy experiments. This session reports on the challenges and tentative outcomes of a 4-state pilot demonstration project that used random assignment to study a U.S. Social Security Administration (SSA) policy innovation designed to encourage persons with disabilities receiving Social Security Disability Insurance (SSDI) support to increase their work efforts and move towards financial independence. In each of the four states, individuals receiving SSDI assistance were recruited and randomly assigned to either “experimental rules” (gradual reduction of cash assistance as earnings increase beyond the SSDI threshold) or “current rules” (abrupt end of cash assistance when earnings cross the threshold) conditions.
|
|
The Vermont Social Security Disability Insurance (SSDI) $1 for $2 Benefit Offset Pilot: Experiences From the Field
|
| Alydia Payette,
Vermont Division of Vocational Rehabilitation,
alydia.payette@dail.state.vt.us
|
| James Smith,
Vermont Division of Vocational Rehabilitation,
james.smith@dail.state.vt.us
|
| Alice Porter,
Vermont Division of Vocational Rehabilitation,
alice.porter@dail.state.vt.us
|
| Peter Burt,
Vermont Division of Vocational Rehabilitation,
peter.burt@dail.state.vt.us
|
|
Vermont's participation in the SSDI Benefit Offset Pilot offered unique opportunities for insight into the implementation issues of a random assignment project involving SSA beneficiaries. Vermont enrolled 577 individuals, 285 in the intervention group, 292 in the control group. Vermont's successful recruitment effort and management of benefits for pilot enrollees required intensive effort and close monitoring. Lessons learned from the process evaluation have been instructive for future pilot design and policy development. Vermont's outcomes evaluation relies on administrative data-including Unemployment Insurance quarterly wage reports, public benefits participation and claims records, and disability benefits data supplied by SSA. While these data are not yet reliable, preliminary data suggest that pilot enrollees are working at a higher level than control group members.
|
|
|
Understanding the Challenges and Opportunities of Randomized Policy Experiments: Lessons From the Utah Social Security Administration Benefit Offset Study
|
| Anne Reither,
Utah State University,
areither@gmail.com
|
| Cathy Chambless,
University of Utah,
cathy.chambless@cppa.utah.edu
|
| Sara McCormick,
University of Utah,
sara.mccormick@cppa.utah.edu
|
| George Julnes,
Utah State University,
gjulnes@cc.usu.edu
|
|
In an effort to understand the employment impacts of a proposed policy change, the Social Security Administration contracted with four states, including Utah, to conduct pilot policy experiments that were to inform a subsequent national experimental study. The Utah project enrolled 502 participants from September 2005 to October 2006 and assigned them randomly to either experimental rules (gradual reduction of cash assistance as earnings increase beyond the SSDI threshold) or current rules (abrupt end of cash assistance when earnings cross the threshold) conditions. This presentation will report on the lessons learned in implementing this policy experiment and offer insights into the dynamics related to employment.
| |
|
The Benefits Offset Pilot Project: The Connecticut Experience
|
| Julie Robison,
University of Connecticut,
jrobison@uchc.edu
|
| Cynthia Gruman,
University of Connecticut,
gruman@uchc.edu
|
| Amy Porter,
Connecticut Bureau of Rehabilitation Services,
amy.porter@ct.gov
|
|
Connecticut's Benefits Offset Pilot enrolled 265 participants: 128 test group and 137 control group. Recruitment challenges included a large number of ineligible applicants due to expired Extended Period of Eligibility and delays in applying the Offset. Connecticut used a mixed-methods approach which included telephone interviews with all participants at multiple time points: baseline, 3, 6, 9 and 12 months. Participants are 56% female, 80% white and 96% have at least a high school diploma. Most reported a physical disability (63%) and/or a mental illness (53%). 73% were employed at enrollment, with 20% working full time. At 3 months after enrollment, test participants were significantly more likely than controls to have: increased their income by $100 or more, told their employer they could work more hours or earn more, increased their work hours, started a job (if unemployed at baseline) or added a second job.
| |
|
Challenges of Conducting Randomized Field Trials: The Experience of the Wisconsin Social Security Disability Insurance (SSDI) Employment Pilot
|
| Barry S Delin,
University of Wisconsin, Stout,
delinb@uwstout.edu
|
| Christopher Sell,
University of Wisconsin,
sellcw@dhfs.state.wi.us
|
| Anne Reither,
Utah State University,
areither@gmail.com
|
|
This paper describes challenges faced in conducting a randomized field trial involving decentralized enrollment and data collection processes and a risk adverse population. The project is one of four state based pilots of a benefit offset provision for the Social Security Disability Insurance (SSDI) program. Wisconsin’s pilot is substantially more decentralized than the others with recruitment, enrollment, and service provision accomplished through state contracted agencies. These agencies are responsible for providing encounter data for research purposes. Thus, persons not trained researchers are responsible for implementing tasks critical to the evaluation’s quality. This paper is organized as a case study using multiple sources of qualitative data from process evaluation activities. It also utilizes information on data consistency and accuracy to collaborate findings based on the qualitative data. Finally, the presentation will include some initial outcomes data.
| |
|
Session Title: Learning to Assess Learning: Using the Instructional Assessment Resources (IAR) Web Site to Improve Teaching and Learning
|
|
Demonstration Session 440 to be held in Chesapeake Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Assessment in Higher Education TIG
|
| Presenter(s): |
| Joel Heikes,
University of Texas, Austin,
joel.heikes@austin.utexas.edu
|
| John Kucsera,
University of Texas, Austin,
kucserajohn@hotmail.com
|
| Abstract:
The Instructional Assessment Resources (IAR) Web site is an online assessment and evaluation tool developed for faculty, staff, and students in higher education. The site is designed to teach users how to conduct best practices instructional assessment and evaluation. Specifically, the site helps users learn how to assess students, assess their own teaching, assess instructional technology, or evaluate programs. In addition to demonstrating how to use the site and its tools, the facilitators will also discuss issues related to teaching assessment and evaluation to faculty and user response to the site.
|
|
Session Title: From Identity-based Perspectives to Social Justice in Evaluation
|
|
Think Tank Session 441 to be held in Versailles Room on Thursday, November 8, 3:35 PM to 5:05 PM
|
|
Sponsored by the Feminist Issues in Evaluation TIG
|
| Presenter(s):
|
| Saumitra SenGupta,
APS Healthcare,
ssengupta@apshealthcare.com
|
| Discussant(s):
|
| Saumitra SenGupta,
APS Healthcare,
ssengupta@apshealthcare.com
|
| Sharon Brisolara,
Evaluation Solutions,
evaluationsolutions@hughes.net
|
| Denice Cassaro,
Cornell University,
dac11@cornell.edu
|
| Abstract:
This think tank session will further the dialog started in a presidential strand on social justice in evaluation. The identity-based topical interest groups (TIG) and their perspectives will be presented by Sharon Brisolara, Denice Cassaro and Saumitra SenGupta in a conversation with the participants. In the last ten years, AEA has seen a number of new TIGs that have begun to foster discussions around various identities that are rooted in gender, ethnicity, sexual orientation, and national origin among others. The diversity has added depth to our discussions and some cross-cutting, overarching themes have begun to emerge that must be interwoven in the fabric of our evaluative endeavors and contexts. Addressing the power structure, inequality, and social justice issues top this agenda.
In a true sense of the term think tank, the participants will take an active role in exploring where the evaluation field is today, where it should be tomorrow, and the possible ways it will get their. In this journey, we will also explore what it will mean in each of our sphere of practice and what it will take to sustain this effort. In particular, we will also explore how each of our identities and perspectives shape this dialog. Together, Sharon Brisolara, Denice Cassaro, and Saumitra SenGupta will bring in the Feminist Issues, the LGBT Issues and Multiethnic Issues perspectives and the experiences they have gleaned over the years through their leadership roles in these TIGs. Breakout sessions will discuss specific perspectives that will be brought together for common strategies and linkage.
|
