|
Scales and Indices: What's the Difference and Does it Matter?
|
| Presenter(s):
|
| Michael Hennessy,
University of Pennsylvania,
mhennessy@asc.upenn.edu
|
| Abstract:
Statisticians and quantitative researchers make a distinction between effect and causal indicators. But does this distinction really matter? To answer this question, we use data from a longitudinal, multi-site intervention study to examine the measurement model, temporal reliability, and associations between two latent constructs: outcome expectancies for condom negotiation and condom use self-efficacy. The former is a latent variable defined by causal indicators whereas the latter is defined by effect indicators. We find that the effect indicators work well while the causal indicator model is problematic. Therefore, we suggest a four step analytic strategy when researchers are confronted with a set of items that may operate as causal indicators.
|
|
Measuring Identity and Identity Change in Evaluations
|
| Presenter(s):
|
| Elaine Hogard,
University of Chester,
e.hogard@chester.ac.uk
|
| Roger Ellis,
University of Chester,
r.ellis@chester.ac.uk
|
| Abstract:
Many programmes to be evaluated have as their objectives changes in the attitudes and behavior of participants. Explicitly or implicitly such objectives bring into play the elusive but significant concepts of self and identity. The evaluation literature has few if any studies that address identity and identity change as a programme outcome. This paper describes a novel identity-measuring instrument known as IDEX and its accompanying software IDIO. IDEX is based on the repertory grid method of George Kelly but focuses on elements related to past, present, future and alternative identities. It is able to detect significant factors regarding an individual's identity and has proved sensitive to change over time thus making it an ideal outcome measure for the evaluation of a programme. The paper gives examples of the use of IDEX in evaluations of a staff development programme in higher education; and a programme to encourage inter-professional collaboration in social and health care.
|
|
Test Evaluation Using an Explanatory Item Response Model
|
| Presenter(s):
|
| Rachael Tan,
University of California, Berkeley,
jinbee@berkeley.edu
|
| Abstract:
Usually we think of assessments as simply tools used in evaluation, but assessments should be the subject of evaluation as well. In the current era of accountability where testing affects every level of the educational system, it is important to evaluate assessments to ensure they are valid and provide an accurate portrayal of student ability. This research uses a hierarchical measurement model to examine whether student and teacher characteristics can help explain test score differences among students. Among other factors, the model includes students' gender, English language proficiency, special education status, and teachers' years of experience teaching the curriculum and assessments. Beginning an evaluation of an assessment at the data level, as in this research, can identify factors that affect student performance, which can help focus a larger evaluation on important elements that contribute to success on assessments, and indicate issues to consider during test design.
|
|
Construction and Interpretation of Composite Variables
|
| Presenter(s):
|
| Katherine McKnight,
Pearson Achievement Solutions,
kathy.mcknight@pearsonachievement.com
|
| Lee Sechrest,
University of Arizona,
sechrest@u.arizona.edu
|
| Abstract:
Composite variables are combinations of variables that are individually meaningful and are thought to be indicators of the same construct. For example, SES is usually a combination of several variables including income, zip code, occupation, educational level, etc. Evaluators frequently use composite variables to measure constructs such as health risks, poverty, ethnicity, quality of life, social support and so on. Although used frequently, composites can be problematic in ways that might not be immediately obvious. Identifying appropriate variables to combine, selecting an appropriate algorithm for combining them and assessing psychometric properties are just some of the problems composites present to evaluators. In this paper, we highlight some problems and offer some alternatives. Because evaluators use composites frequently, it is important to understand the problems associated with the creation and assessment of these variables.
|
| | | |