|
Getting Into the Black Box: Using Factor Analysis to Measure Implementation
|
| Presenter(s):
|
| Xiaodong Zhang,
Westat,
xiaodongzhang@westat.com
|
| Atsushi Miyaoka,
Westat,
atsushimiyaoka@westat.com
|
| Abstract:
Although regarded as a critical aspect in program evaluation, implementation is also known as a “black box” to program evaluators. This situation is compounded by a lack of methodological agreement in how to accurately measure implementation. The paper begins with a brief overview of different approaches to quantifying implementation fidelity. Drawing on our experience with the evaluation of Reading First-Ohio, we present a factor-analysis approach to measure program implementation. We also discuss the use of implementation scales and the theoretical and practical implications of the method.
|
|
Necessary Questions for Evaluators to Consider Before, During, and After Conducting Inter-Rater Reliability Analyses
|
| Presenter(s):
|
| Dane Christian,
Washington State University,
danechristian@mail.wsu.edu
|
| Michael Trevisan,
Washington State University,
trevisan@wsu.edu
|
| Denny Davis,
Washington State University,
davis@wsu.edu
|
| Steve Beyerlein,
University of Idaho,
sbeyer@uidaho.edu
|
| Phillip Thompson,
Seattle University,
thimpson@seattleu.edu
|
| Kunle Harrison,
Tuskegee University,
harrison@tuskegee.edu
|
| Robert Gerlick,
Washington State University,
robertgerlick@wsu.edu
|
| Abstract:
As part of an evaluation for Capstone Engineering Design Course Assessment Development, Inter-Rater (IR) reliability analyses were conducted to ensure consistency in engineering faculty ratings. Most methods for calculating IR reliability analyses are not without problems. The selection criteria of an IR reliability index ought to be based on its properties and assumptions, the level of measurement of the variable for which agreement is to be calculated, and the number of raters in the analysis. Whichever index is used, researchers need to explain why their choice of index is appropriate given the context of the characteristics of the data being evaluated. This paper considers conceptual and methodological issues among some common IR indices, and the choice of percent agreement for the Capstone project is explained in light of the selection criteria. Some questions for evaluators to consider before, during, and after conducting IR reliability analyses are offered.
|
|
The Implicit Association Test (IAT): A Tool for Evaluation?
|
| Presenter(s):
|
| Joel Nadler,
Southern Illinois University at Carbondale,
jnadler@siu.edu
|
| Nicole Cundiff,
Southern Illinois University at Carbondale,
karim@siu.edu
|
| Gargi Bhattacharya,
Southern Illinois University at Carbondale,
gargi@siu.edu
|
| Steven Midleton,
Southern Illinois University at Carbondale,
scmidd@siu.edu
|
| Abstract:
The Implicit Association Test (IAT) is a methodological way to measure implicit bias using two group comparisons. While, general measures of implicit biases (automatic reactions) have been present in the psychological literature for quite some time, IAT is relatively new (Greenwald, McGhee, & Schwartz, 1998). Since its inception, IAT measures have been extensively used in stereotype and prejudice research. IAT results have been shown to be weakly related to explicit measures when social desirability concerning the target stimuli is involved. However, when there is no socially expected “right” answer there is a stronger relationship between the two methodologies. Issues of validity and reliability concerning the IAT have been actively discussed in numerous articles since the inception of the IAT. The unasked question is whether a measure assessing automatic, implicit, or “unconscious” reactions can be of use in consulting and evaluation. Theoretical application, practical concerns, and possible appropriate use are discussed.
|
|
Policy and Evaluation: Developing a Tool to Synthesize Evidence in Education Partnerships
|
| Presenter(s):
|
| Mehmet Ozturk,
Arizona State University,
ozturk@asu.edu
|
| Abstract:
For several years, a substantial evidence-based educational practice has been emerging among researchers and policy-makers. However, much less attention has been given to identifying effective ways to produce the needed evidence. Developing effective techniques and tools that can help education researchers and evaluation experts make sense of the available evidence is critical. In this context, effective synthesis of evaluation evidence has become crucial.
This paper discusses the development of effective tools for synthesizing evidence within education partnerships. The Program Effectiveness Scale/Rating System, which was developed to help policy-makers better understand the impact of programs, strategies, and innovations, is presented as a practical example. By synthesizing and placing evaluation results into a format that is easy to understand, the Program Effectiveness Scale can help policy-makers make objective judgments on the worth of a program or strategy.
|
| | | |