Considering Multiple Perspectives and “Missing Voices” in Program Evaluation

Session Number: AHE4
Track: Assessment in Higher Education
Session Type: Multipaper
Session Chair: Sandy Jean Hicks, Ph.D. [University of Rhode Island]
Presenter 1: Karen Hui-Jung Chen [National Taipei University of Education]
Presenter 2: Molly Reas Hall [Virginia Tech]
Presenter 3: Antonio Olmos [University of Denver]
Presenter 4: Eileen Dryden [Institute for Community Health]
Time: Oct 17, 2014 (08:00 AM - 09:30 AM)
Room: Mineral E

Level: None
Abstract 1 Title: The Different Perspectives of Internal and External Evaluation -- An Analysis of the Higher Education Accreditation in Taiwan
Presentation Abstract 1: Quality of higher education is ensured by internal and external evaluation. Although internal and external evaluation is complementary to each other, how to make balance between them is important issue. The purpose of this study is to investigate the gap through content analysis of 642 objections from 65 universities and the replies from HEEACT in Taiwan. Results from analyzing the objections showed institutions disagreed with the on-site-visit report for the reasons: (1) inconsistent with the fact; (2) asking modification of the rhetoric on the reports. On the other hand, the reviewers kept their decisions unchanged in the replies for the concerns: (1) institution providing invalid data; (2) reviewers trusting what they have seen or heard during the on-site visit; (3) institutions' objections not focusing on the same issue as review comments. It is hope this study will make better understanding of the gap and improve evaluation process in the future.
Abstract 2 Title: A Missing Voice in Higher Education Quality Assessment: What Do Front-Line Faculty Really Think About Quality?
Presentation Abstract 2: Although engaging faculty members in quality assessment activities such as institutional accreditation and academic program review is a major issue at many colleges and universities, surprisingly little is known about how front-line faculty view quality in higher education. This session will highlight major themes from a recently conducted literature review on faculty perspectives related to quality and quality assessment in higher education and discuss their implications for practice in the field of higher education assessment. For example, a review of the literature suggests that: (1) there appear to be disconnects between how faculty think about quality and how quality assessment activities are currently structured, (2) there is a gap between how faculty members view quality and how administrators view quality, and (3) there is a gap between the "rhetoric of quality" (i.e., the espoused emphasis on improvement) and the "reality of quality" (i.e., the actual emphasis on accountability).
Presentation 2 Other Authors: Penny Burge, Virginia Tech, burge@vt.edu
, ,
Abstract 3 Title: Differences in Perceptions of Teacher Preparation for Teacher Candidates, Teacher Graduates, and Program Faculty
Presentation Abstract 3: Core competencies essential for effective teaching were identified via a literature review and a review of standards for teacher education, and vetted by state groups with interests in teacher education. Survey items based on these competencies asked teacher candidates, teacher graduates, and teacher education program faculty how well the program prepared teachers. The 41 items common to surveys of the three groups were used to define nine core competencies and means were compared across teacher candidates, graduates, and program faculty. Surveys were distributed electronically and responses were received from 296 teacher candidates, 648 graduates, and 501 program personnel. Statistically significant overall differences were found for eight of the nine competencies, with effect sizes ranging from small to moderate. Means for teacher graduates were systematically lower than means for candidates or program faculty. We discuss some implications of the results for teacher education program
Presentation 3 Other Authors: Kimberly Hartnett-Edwards, University of Denver, Kimberly.Hartnett-Edwards@du.edu
Kathy Green, University of Denver, kgreen@du.edu
Kent Seidel
kent.seidel@du.edu
University of Denver
Abstract 4 Title: Exploring the Use of Longitudinal Triad-Interviews in Medical Residency Program Evaluation
Presentation Abstract 4: This presentation highlights the value of using longitudinal triad-interviews while evaluating medical residency programs in the midst of transformation. There are a number of challenges faced when evaluating programs in dynamic contexts. Drawing from our experience evaluating an Internal Medicine residency training program at a healthcare system the Northeast, we will describe how we used this method to address many of these challenges and the lessons learned while implementing it. By 'longitudinal triad-interviews' we mean interviews with three individuals simultaneously that take place annually over the course of a number of years. This method provided the benefits of rich, focus-group like discussion among participants while allowing individual-level, interview-type data to be collected. The interview findings were used to guide changes to the program, measure short-term outcomes, inform additional data collection tools, and begin to create long-term individual resident case studies.
Presentation 4 Other Authors: Reann Gibson, Institute for Community Health, rgibson@challiance.org
, ,

Session Abstract: 

Considering Multiple Perspectives and “Missing Voices” in Program Evaluation