|
Session Title: Getting Down to Cases: Using Data to Inform Evidence-based Decision-making
|
|
Panel Session 792 to be held in Santa Monica on Friday, Nov 4, 4:30 PM to 6:00 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Sally Olderbak, University of Arizona, sallyo@u.arizona.edu
|
| Abstract:
Ultimately, evidence has to be meaningful and relevant at the level of the individual decision maker. Data are only the raw material for the formulation of evidentiary conclusions. How such conclusions are formulated can be critical to their use. This session describes some settings and empirical data that need to be reflected in decisions made in specific instances. A meta-analytic study of Cognitive Behavioral Therapy provides conclusions that permit a range of decisions at levels represented by clinics, clinicians, and patients. Another example shows how calibration of an index of addiction severity can facilitate decisions at the clinical level. A third illustration, having to do with measures of impairment in clinical settings, shows how decisions may be improved when additional variables are considered. Finally, a large scale analysis of data on value-added assessment of teacher performance will be used to show the problems associated with decisions about individual teachers.
|
|
Meta-analytic Assurance and the Individual Case
|
| Sacha Brown, University of Arizona, sdbrown@u.arizona.edu
|
|
Most treatment evaluations, including evaluations of Cognitive Behavioral Therapy (CBT), focus on aggregate-level outcomes while ignoring cost-benefit perspectives of individuals receiving treatment, therapist/agencies providing treatment, and those paying for treatment. Initial analyses comparing contributions from cognitive and behavioral manipulations demonstrate that cognitive therapy (CT) contributes little to CBT therapeutic outcome (e.g., Longmore & Worrell, 2007). The limited additive effect of CT and lack of analyses from stakeholder perspectives necessitates a more comprehensive examination of CBT efficacy. We evaluated a sample of randomized CBT component-controlled trials treating anxiety disorders. Our statistical analyses occurred in two major phases. One evaluated the additive effects of each unique component against wait-list and placebo controls. The second compared the effects of each component against one another. Stakeholder cost-benefit variables were also incorporated (e.g. treatment duration, treatment mode, specific disorder). Findings suggest that more analyses that include stakeholder perspectives provide new insights into treatment effectiveness
|
|
|
Beyond Evaluation: Individual Differences do Matter
|
| Mende Davis, University of Arizona, mfd@u.arizona.edu
|
|
Latent correlations between Impairment, functioning, and disability in several disease conditions are sometimes very high, e.g.,for cerebral palsy and muscular dystrophy. For example, if patients cannot move their legs, they cannot walk up stairs. Despite the magnitude of the correlations, some of the residual variance can be explained by other variables. A notable instance is the additional explanatory power of intelligence. Smarter people are able to find ways to get around specific disabilities and impairments in order to permit better functioning. The relationship between these variables are quite different from one condition to another. Personal/family characteristics and available resources contribute to use of services and outcomes. States with ample funding for services, families 'in the know' regarding useful services, and individual characteristics (motivation, goals, habilitation vs. rehabilitation) make a big difference in outcomes. Evaluation results in general may not provide a good guide for what should be done in particular
| |
|
Calibrating an Addiction Severity Index for Better Decision Making
|
| Ryan Seltzer, Universiity of Arizona, rseltzer@email.arizona.edu
|
|
Calibration is a method for enhancing the interpretability of changes in arbitrary survey scores often attributed to treatment intervention. The Fagerstrom Test for Nicotine Dependence (FTND) is a six-item survey used by tobacco cessation programs to characterize patients into low, medium, and high levels of nicotine dependence. Calibration of the FTND is essential for quantifying how patients with varying levels of dependence progress through and use cessation programs. The FTND is administered at intake to all patients enrolling in the Arizona Smokers' Helpline. Statistical techniques (e.g., GLM and SEM) were used to link program data such as relapse frequency, duration in program, number of counseling sessions, and quit rate to differences in FTND scores. Calibration procedures revealed, for instance, that for every increase in FTND point, probability of abstinence at seven months, on average, decreased by 13%. Calibration can identify programmatic areas that require flexibility based on tobacco dependence.
| |
|
Value-added Analyses of Teacher Performance: Maybe Good in General, Maybe Troublesome in Practice
|
| Mei-kuang Chen, University of Arizona, kuang@email.arizona.edu
|
|
Value-added performance measures for teachers are rapidly proliferating. Although they have not yet been fully evaluated, it seems likely that, in general, they will prove to have some utility in guiding policies about teacher evaluation. It is not so clear that such measures can be easily and uniformly applied in individual schools and to individual teachers. Analyses of a very large data set representing data on about 3000 teachers and more than 150,000 students suggest that the problems at school and teacher levels will be formidably challenging. Getting from "significant" findings about value-added teacher scores to decisions about training requirements, salary increments, and retention may result in resistance that will defeat the entire enterprise. Evaluators will need to be proactive in helping decision makers (and other stakeholders) to understand value-added measures and how they need to be used as part of the process of improving education.
| |