| Session Title: Making Lemonade: Taking Advantage of Federal Government Performance Results Act (GPRA) Reporting Requirements in Evaluation |
| Multipaper Session 696 to be held in Mineral Hall Section D on Friday, Nov 7, 4:30 PM to 6:00 PM |
| Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG |
| Chair(s): |
| Donna Atkinson, Westat, donnaatkinson@westat.com |
| Discussant(s): |
| Bill Luckey, Westat, billluckey@westat.com |
| Abstract: The role of performance measurement or accountability data collected under Government Performance Results Act (GPRA) has been debated as both a methodological and an evaluation policy issue since it was introduced. Although the debate continues, the emphasis on accountability at the Federal level continues to expand. The Center for Substance Abuse Treatment (CSAT) requires all grantees funded through their discretionary portfolio to collect standardized data for all clients at admission and six-months post-admission. Although accountability data typically are insufficient for evaluation, these standardized data can serve an important role in evaluations, particularly when complemented with other data sources. This session will provide three examples of how evaluators at the local, cross-site and Federal level have incorporated CSAT GPRA into their efforts. |
| The Center for Substance Abuse Treatment GPRA Data Repository |
| Deepa Avula, Center for Substance Abuse Treatment, deepa.avula@samhsa.hhs.gov |
| This presentation will provide an overview of CSAT GPRA data collection efforts and how the agency uses the data beyond mandatory reporting. CSAT has implemented a standardized data reporting requirement, with some 400 active grantees that provide substance abuse treatment services. The resulting data repository, which includes standard data on measures of importance to the agency from a national perspective, e.g., abstinence from substance abuse, living conditions, employment and education, criminal activity, and social connectedness, currently includes matched intake-follow-up data on over 70,000 clients served by grantees currently funded by the Center. Grantees are able to access a series of reports on their data as well as download an Excel version that can be imported into statistical packages for further analysis. Other uses that CSAT makes of the data beyond fulfilling reporting requirements, both for responding to external requests from oversight agencies such as SAMHSA and Congress as well as internal management within the Center, will also be discussed. |
| Using GPRA Data in Local Evaluation |
| Kristin Stainbrook, Advocates for Human Potential, kstainbrook@ahpnet.com |
| GPRA data, as well as complementary local data, can be used as a tool to improve program services. Advocates for Human Potential, the evaluator for the Sherman Street Program, a CSAT-funded Treatment for the Homeless grantee, has used GPRA data to assist with program management, quality assurance, and sustainability. Through regular presentations to program management, client GPRA information is used to provide an overall picture of program functioning to identify program strengths and weaknesses, as well as areas for staff supervision and training. Data are also presented to program staff on a regular basis to provide feedback on client impact and assist in planning program changes. Finally, data are used in presentations to partnering agencies, community meetings, and local government officials in order to garner program support. This presentation will include examples employing GPRA and local evaluation data to support programs in management, and quality assurance, as well as sustainability. |
| Substance Abuse and Mental Health Administration (SAMHSA) Center for Substance Abuse Treatment (CSAT) Access to Recovery Cross Site Evaluation: A Focus on Client Data |
| Laura Dunlap, RTI International, ljd@rti.org |
| Recognizing the enormous societal costs of failure to get needed treatment services, the President and SAMHSA established Access to Recovery (ATR), a 3-year competitive discretionary grant program to States, Territories, and Tribal Organizations. The primary focus of the ATR program has been to promote a client-centered system by improving treatment access through utilizing treatment payment vouchers, expanding independent client choice of providers, expanding access to services, and increasing substance abuse treatment capacity. As part of this effort, CSAT has funded a cross-site evaluation to provide information on the effectiveness and sustainability of the ATR program being implemented in 24 grantees. This presentation will provide a discussion of the cross-site evaluation design pertaining to the use of client-level data collected by these grantees via CSAT's SAIS-GPRA system focusing on the advantages and limitations of GPRA data as well as evaluation and modeling techniques for using these data in a cross-site evaluation. |
| Winning One for the GPRA? Making Case-Mix Adjusted Performance Monitoring Data Available to Grantees in CSAT-funded Substance Abuse Treatment Programs |
| Joe Sonnefeld, Westat, josefeldsonnefeld@westat.com |
| Zhiqun Tang, Westat, zhiquntang@westat.com |
| Duke Owen, Westat, dukeowen@westat.com |
| CSAT grantees have had access to web-based reports on GPRA data they have submitted. The Center was interested in providing grantees with comparative information, so this year it began to distribute reports that allow grantees to compare their six National Outcome Measures from the GPRA tool to those of other programs funded under the same discretionary grant program. The outcomes are case-mix adjusted using logistic regression and displayed graphically. Issues in the development of the methods and their usefulness of reports to local evaluations will be discussed. The context is the changing climate of evaluation policy in which an emphasis on performance monitoring may reduce the number of randomized and comparison-group evaluations. Large multi-site naturalistic studies of pre-post data can be argued to be better suited to identifying treatment and setting interactions with severity because they can include patients normally excluded from randomized trials for ethical reasons. |