|
Session Title: Snapshots of Exemplary Evaluations
|
|
Panel Session 611 to be held in Texas F on Friday, Nov 12, 1:40 PM to 3:10 PM
|
|
Sponsored by the Research on Evaluation TIG
|
| Chair(s): |
| Paul Brandon, University of Hawaii, brandon@hawaii.edu
|
| Discussant(s):
|
| Paul Brandon, University of Hawaii, brandon@hawaii.edu
|
| Abstract:
Existing research on the nature of exemplary evaluation practice is limited. In spite of awards being given for outstanding work, there has been little study of what makes an evaluation exemplary and how to produce excellent work consistently. The four presentations in this panel examine national, regional, and local studies for the purposes of (a) describing the characteristics of exemplary work; (b) identifying the factors, conditions, events, and actions that contribute to exemplary work; and (c) considering the impediments to improved practice. Influences such as effective evaluation designs, strong stakeholder relationships, and evaluator flexibility in dealing with changing contextual factors are shown as key in promoting exemplary practice. Implications for an improved theory of evaluation practice, as well as improvements in evaluation practice, are also considered.
|
|
Informing Policy in a Cultural Crossfire: The Title V Abstinence Education Evaluation
|
| Christopher Trenholm, Mathematica Policy Research, ctrenholm@mathematica-mpr.com
|
| Barbara Devaney, Mathematica Policy Research, bdevaney@mathematica-mpr.com
|
|
In 1998, Mathematica Policy Research began a multi-year evaluation of abstinence programs funded as part of the landmark welfare reform legislation. The evaluation used an experimental design to estimate impacts of programs. Based on baseline and follow-up survey data collected from over 2,000 youth in four sites, the analysis revealed no evidence that the programs increased abstinence from sex; however, it also found no evidence that the programs reduced rates of contraception, a common concern of program opponents.
Despite the contentiousness of its findings, the evaluation was widely acknowledged for its rigor and objectivity and, in turn, it had a substantial impact on federal and state policy. Among the keys to this success were the evaluation’s highly credible research design (and effective implementation of this design), a collaborative approach that sought buy-in from the range of study stakeholders, and a concise, consistent, and balanced presentation of the evaluation’s findings.
|
|
|
Exemplariness as Engaging and Value-added Evaluation
|
| Melanie Hwalek, SPEC Associates, mhwalek@specassociates.org
|
|
Size, scope and type of evaluation are less important than other factors in meeting criteria for exemplariness. Hwalek takes the position that an evaluation is exemplary when: (a) people become excited about the evaluation, (b) the right information is provided at the right time, and (c) the evaluation maintains its methodological rigor. Three cases will be presented as illustration: a shoe string evaluation of an HIV medication compliance training program, a multi-faceted evaluation of a capacity building organization, and a complex multi-state policy change evaluation. Six factors are hypothesized to contribute to exemplary work: culture of the user organization as one of learning, organizational involvement in determining what is to be learned, having an evaluation champion within the user organization, the fit of the evaluation feedback mechanism to the needs of the organization, trust between evaluator and key organizational contact, and the evaluator’s skill in making the evaluation come alive.
| |
|
Curbing Home Health Care Costs While Improving Quality of Care
|
| George Grob, Center for Public Program Evaluation, georgefgrob@cs.com
|
|
In the early 1990’s, the Medicare program began implementing a new benefit to provide in-home care to home-bound patients. This program built on a tradition of dedicated visiting nurses, active since the 1900s. However, the benefits soon became industrialized, with new chains of providers organized to exploit the new benefit. Annual Medicare expenditures mushroomed from $3 billion to $18 billion. Auditors found that 40% of Medicare payments were improper. Others raised questions about the quality of care. The Office of Inspector General’s Office of Evaluation and Inspections undertook a series of studies that led to reforms which ultimately saved $50 billion and promoted better care. Follow-up studies concluded that Medicare beneficiaries truly needing home health care were able to get it. It is this series of evaluations, along with auditors and investigators, that yielded these results. This presentation will address the exemplary features of the studies.
| |
|
Exemplary Evaluations: Characteristics, Conditions, and Considerations
|
| Nick Smith, Syracuse University, nlsmith@syr.edu
|
| Leigh M Tolley, Syracuse University, lmtolley@syr.edu
|
|
Although professional societies such as the American Educational Research Association and the American Evaluation Association give awards for outstanding evaluation studies, there has been relatively little research on what exactly makes evaluation work exemplary, the conditions that contribute to outstanding practice, and the barriers or limitations that must be overcome to improve practice. This paper examines 15 studies previously identified as exemplary and investigates the factors, conditions, events, and actions that both contributed to their excellence and impeded their improvement. Preliminary findings suggest that, in addition to employing an effective study design, building strong stakeholder relationships and being responsive to unforeseen contextual factors are key in producing exemplary work. Results of research such as that reported here will not only reshape thinking about the nature of useful evaluation theory, but will provide concrete insights into the improvement of everyday practice in evaluation.
| |