|
MOSART may Play out of Tune: A Mixed Method Look at Validity and Reliability Issues of Misconception Oriented Stanards-based Assessment Resource for Teachers (MOSART)
|
| Presenter(s):
|
| James Salzman, Ohio University, salzman@ohio.edu
|
| Ken Gardner, Aurora University, kgardner@aurora.edu
|
| Abstract:
The Misconception-Oriented Standardized Assessment Resource for Teachers (MOSART) has been touted for use in Math-Science Partnership (MSP) grants as a valid instrument for measuring changes in content knowledge of teachers and their students. The multiple-choice items are aligned with K-12 physical and earth science content standards, as identified by the NRC's National Science Education Standards, and represent 'research'-documented misconceptions concerning science concepts. This paper provides a mixed method analysis of validity and reliability of MOSART. The authors first critically analyzed the quality of items when considered under the lens of test development standards (Nitko, 2001; Nunnally & Bernstein, 1994), then empirically tested the items and scales with a MSP sample. These analyses reveal some concerns about validity and reliability of the instrument. While not discounting the potential value of MOSART, the authors recommend evaluators employ these analyses when choosing instruments that claim established validity and reliability and reporting results.
|
|
The Analysis of the Classroom Behavior Observation Tool: Triangulating on Disruptive Classroom Behavior in the Evaluation Process
|
| Presenter(s):
|
| Steven Lee, University of Kansas, swlee@ku.edu
|
| Julia Shaftel, University of Kansas, jshaftel@ku.edu
|
| Jeaveen Neaderhiser, University of Kansas, jneaderh@ku.edu
|
| Jessica Oeth, University of Kansas, joeth@ku.edu
|
| Abstract:
Many school reform initiatives include school-wide discipline or social-behavioral programs to reduce classroom behavioral problems negatively impacting academic achievement (Pelham et al., 2005). Evaluation of these initiatives requires reliable and valid behavioral measures of students at classroom school, district or state levels. Lee and Shaftel (2004, 2006, 2007) developed and validated surveys completed by teachers and students to identify the extent of problem behaviors and assets for students at the classroom level called the Classroom Behavior and Asset Scale Teachers (CBAST) and Students (CBASS). To triangulate on the assessment of classroom behavior problems, the present study developed and analyzed a classroom observation tool called the Classroom Behavior Observation (CBO). A large sample of observations in high school classrooms from a Midwestern state yielded data on the reliability and validity of the CBO. The implications for using the tools in school program evaluation and directions for future research will be discussed.
|
|
Student Grades as an Outcome Measure in Program Evaluation: Issues of Meaning and Validity
|
| Presenter(s):
|
| Kelci Price, Chicago Public Schools, kprice1@cps.k12.il.us
|
| Abstract:
As the demand for impact evaluation in education continues to grow it is important for evaluators to have valid measures for assessing program effectiveness. Although student grades are often used to assess the impact of educational interventions, there exists considerably ambiguity about the validity of this measure. This research explores the validity of grades with reference to three main issues: 1) the relationship of grades to student knowledge, 2) the sensitivity of grades to changes in student knowledge, and 3) whether there are systematic school factors which impact the relationship between grades and student knowledge. This presentation includes both high school and elementary levels, in math and reading. Implications of the findings for evaluators' use of grades as a measure of program impact are discussed.
|
|
Adapting Instruments to Measure Social Emotional Learning Outcomes in Young Children: Lessons Learned From the Evaluation of the Inner Resilience Program
|
| Presenter(s):
|
| Eden Nagler, Metis Associates, enagler@metisassociates.com
|
| Susanne Harnett, Metis Associates, sharnett@metisassociates.com
|
| Abstract:
Having recently completed an evaluation of the impact of the Inner Resilience Program on teachers and their 3rd-5th grade students, we found there is a dearth of instrumentation to measure social and emotional learning (SEL) outcomes in young children. We will present our experiences identifying and adapting existing instruments for children. Decision-making about modifications and design choices will be discussed and final versions of the instruments will be shared during this paper presentation. Our evaluation findings will be presented and potential issues in interpreting results will be explored, including a discussion of the trade-off between richness of results and developmental appropriateness of instrumentation. An overall discussion of the lessons learned and remaining questions also will be facilitated. We hope that this presentation and discussion will help to advance the development of valid instrumentation for use with young children in the fast-growing field of SEL research and evaluation.
|
|
A National Study of Core Exemplary Teaching Characteristics: An Exploratory Construct Validation
|
| Presenter(s):
|
| Sheryl A Hodge, Kansas State University, shodge@ksu.edu
|
| B Jan Middendorf, Kansas State University, jmiddend@ksu.edu
|
| Linda P Thurston, Kansas State University, lpt@ksu.edu
|
| Cindi Dunn, Kansas State University, ckdunn@ksu.edu
|
| Abstract:
Evaluators at the Office of Educational Innovation and Evaluation (OEIE) sought to 1) identify core exemplary teacher characteristics through an exhaustive review of the literature; 2) develop a construct for assessing the importance of these traits as identified by professionals in the field; and 3) explore whether emerging patterns of responses reveal underlying components within the larger construct. The sample (N = 719) included a nationally solicited disproportionately stratified random sampling of current teachers, principals, and superintendents. Equivalent geographical representation provided strong evidence for external validity of the study design, while capture of demographic characteristics enabled group comparisons, diminishing threats to internal validity. Results indicated that even though researchers, accrediting bodies, state and federal legislatures, and professional organizations differentially define characteristics of mastery teaching, current professional practitioners report unique agreement with the most important traits associated with the daily practices of exemplary teachers.
|
| | | | |