|
Session Title: Lessons Learned From Multi-Site Cost Studies: What to do for, and How to Cope With Resistance to, Cost-Inclusive Evaluation
|
|
Panel Session 388 to be held in Capitol Ballroom Section 3 on Thursday, Nov 6, 4:30 PM to 6:00 PM
|
|
Sponsored by the Costs, Effectiveness, Benefits, and Economics TIG
|
| Chair(s): |
| Brian Yates,
American University,
brian.yates@mac.com
|
| Abstract:
Evaluating the effectiveness of human services is demanding enough when conducted for one program. Evaluating effectiveness of services offered at multiple sites in a country is a formidable challenge. Evaluating costs as well as effectiveness of services at multiple sites can be exceedingly complex. Add cost-effectiveness and cost-benefit analysis, and evaluation can be near-impossible. For three different multi-sites, we describe lessons learned when including costs and monetary outcomes (benefit) in evaluation research. The first panelist details a successful framework for estimating costs of providing behavioral health and family strengthening services to educational and health care organizations. The second panelist elucidates seven steps that proved useful for a cost analysis over 8 implementations of a standardized program for preventing child neglect. The third presentation examines the resistance that was encountered, and (partially) countered, in a cost study of eight consumer-operated mental health services.
|
|
A Completed Cost Study for the National Multi-Site Evaluation of the Starting Early Starting Smart initiative (SESS)
|
| Elizabeth Harris,
Evaluation, Management and Training Associates Inc,
eharris@emt.org
|
|
This paper describes a completed cost study for the national multi-site evaluation of the Starting Early Starting Smart initiative (SESS). SESS programs integrated behavioral health and family strengthening services into early childhood education or primary health care organizations. While these programs implemented a common approach to working with families of children in need, each was adapted to its unique environment with its own mix of specific services and delivery method (e.g., home-based or center-based services). The process recommended here provides a framework adaptable to the specific circumstances of each case. This cost documentation process provides a systematic and comparable estimation of the costs of providing services to participating families included in each of the programs during the core study period. We summarize the purposes and parameters of the framework, outline a step-by-step procedure for arriving at estimated costs, and provide reporting formats.
|
|
|
Lessons Learned from the Multi-Site Cost Analysis of the Family Connections Neglect Prevention Program
|
| Melissa Brodowski,
United States Department of Health and Human Services,
melissa.brodowski@acf.hhs.gov
|
| Jill Filene,
James Bell Associates,
filene@jbassoc.com
|
| Phaedra Corso,
University of Georgia,
pcorso@uga.edu
|
|
Although there has been a growing emphasis on the importance of economic evaluation for the human services, there have been only a limited number of child maltreatment prevention programs that have incorporated cost analysis into their overall evaluation plans. The presentation will highlight the seven essential steps that were used to plan and implement a comprehensive cross-site cost analysis for a set of eight ongoing Federal grants that are replicating and rigorously evaluating Family Connections, a demonstrated effective neglect prevention program. Lessons learned and facilitating factors include the active engagement of site-level program implementers, use of standardized templates, prospective data collection, ongoing technical assistance throughout data collection process, and communication and dialogue about data inconsistencies. Finally, ways in which cost analysis and can be used to facilitate ongoing learning and bridging the knowledge gap between researchers and practitioners will be shared.
| |
|
Lessons learned from a Multi-Site Evaluation of Costs, Benefits, Cost-Effectiveness, and Cost-Benefit of Consumer-Operated Services
|
| Brian Yates,
American University,
brian.yates@mac.com
|
|
Unanticipated consequences of a multi-site evaluation of costs, benefits, cost-effectiveness, and cost-benefit of consumer-operated services (COS) versus traditional mental health services are described. Although the program announcement specified that costs of COS would be measured, asking COSs to report costs was translated into measurement not of the value of resources used, but only of monetary outcomes: benefits. Also, resistance to measuring costs prevented program expenditures from being measured with the specificity that could link use of resources to performance of program activities, with had been key to planned cost-effectiveness and cost-benefit analysis. Even evaluation of COS programs' use of non-monetary resources, such as volunteered time and donated space, was resisted. Research requirements and site characteristics interacted to cause one COS to operate under capacity, increasing cost per consumer so that findings for that program were likely unrepresentative.
| |