| Session Title: Helping Communities Improve: Evaluation as Feedback for Learning |
| Multipaper Session 798 to be held in Royale Conference Foyer on Saturday, November 10, 12:10 PM to 1:40 PM |
| Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG |
| Chair(s): |
| Carolyn Lichtenstein, Walter R McDonald & Associates Inc, clichtenstein@wrma.com |
| Discussant(s): |
| Brigitte Manteuffel, Macro International Inc, bmanteuffel@macrointernational.com |
| Abstract: This multi-paper session describes how the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program (CMHI) operates as a feedback mechanism for federally-funded grantees funded by the Substance Abuse and Mental Health Services Administration's Center for Mental Health Services. These grant communities are funded to develop, maintain, and evaluate systems of care for children's mental health services. Much of the data compiled and analyzed by the national evaluation are fed back to grant communities using various reporting mechanisms designed to support sites in their efforts to improve. Session papers emphasize different perspectives on this theme, including the Federal impetus for this model, an overview of different components of the national evaluation, and two examples of data reporting from particular national evaluation components. |
| The National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program (CMHI): The Federal Perspective |
| Sylvia Fisher, United States Department of Health and Human Services, sylvia.fisher@samhsa.hhs.gov |
| This presentation describes how CMHI national evaluation reporting activities promote continuous self-improvement activities at the grantee level and facilitate federal-level decision-making and provision of technical assistance. Data collection and reporting mechanisms include the (1) Annual Report to Congress, which describes whether participating children improved on clinical and functional outcomes and whether these improvements endure over time, and (2) System-of-Care Assessments (SOCA), which are administered three times during the 6-year grant cycle, yielding reports that help grantees more effectively allocate personnel and funding and prioritize activities to conform to system of care service delivery principles. Both inform decision-making and continuous program improvement activities by identifying gaps in system development, barriers to collaboration, and effective program components and interventions with participating children and their families. This process is described in greater detail from a federal perspective. |
| National Evaluation Results as a Mechanism for Continuous Quality Improvement in Grantee Communities |
| Carolyn Lichtenstein, Walter R McDonald & Associates Inc, clichtenstein@wrma.com |
| Integral to the national evaluation of the CMHI are local evaluation capacity-building and use of evaluation data for local program development, improvement, and sustainability. Data are collected for this evaluation using several different methods, including descriptive and longitudinal interviews with caregivers of children and youth served by the system of care, site-visit interviews with key informants in each grant community, and Web-based surveys of providers and administrators. Multiple processes are used to feed data and findings back to the grant communities. This paper describes the national evaluation's research questions, the data collected to answer each question, and current feedback mechanisms. |
| The Evidence-based Practices and Cultural and Linguistic Competence Studies: Complex Topics for Complex Systems |
| Kurt Moore, Walter R McDonald & Associates Inc, kmoore@wrma.com |
| John Fluke, Walter R McDonald & Associates Inc, jfluke@wrma.com |
| This paper shares data analyses from the CMHI national evaluation's Evidence-Based Practices (EBP) and Cultural and Linguistic Competence (CLC) Studies focusing on characteristics of providers, administrators, and systems that are implementing new treatments and CLC practices. Initial findings are also presented from the first CLC sub-study on implementation factors based on in-depth interviews and focus groups with four 2005-funded CMHI grant communities. The adoption of evidence-based treatment practices may entail profound and potentially risky changes in system structure and behavior and the availability of rapid, high-quality evaluative feedback may facilitate this organizational learning/change. The use of evidence-based practices raises questions about cultural and linguistic competence, as well. The CLC Study examines grantee self-assessment approaches, implementation of CLC practices, and the intersection of EBP and CLC. |
| Data Reporting: Information to Guide Learning |
| Susan Drilea, Walter R McDonald & Associates Inc, sdrilea@wrma.com |
| John Fluke, Walter R McDonald & Associates Inc, jfluke@wrma.com |
| This paper describes two reporting mechanisms used to feed back descriptive and longitudinal interview data to grant communities to support continuous quality improvement for the national evaluation of the CMHI. Reports produced for the grant communities include: (1) Data Profile Reports (DPRs), and (2) Continuous Quality Improvement (CQI) Reports. DPRs are produced three times a year at the aggregate and community level. The national evaluation statistically summarizes data from the interviews and produces PowerPoint slides of graphs and tables for communities to use in presenting this data to their stakeholders. Grant communities can download their own data file for additional analysis to support local program needs. The national evaluation also produces CQI Reports four times a year at the aggregate and community level. Grant communities use CQI Reports to identify areas needing programmatic technical assistance. |