| Session Title: Using Empowerment Evaluation to Facilitate Organizational Transformation: A Stanford University Medical Center Case Example |
| Multipaper Session 799 to be held in Hanover Suite B on Saturday, November 10, 12:10 PM to 1:40 PM |
| Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG |
| Chair(s): |
| David Fetterman, Stanford University, profdavidf@yahoo.com |
| Discussant(s): |
| Abraham Wandersman, University of South Carolina, wandersman@sc.edu |
| Abstract: Empowerment evaluation is guiding evaluation efforts throughout the Stanford University Medical Center. Empowerment evaluation is a collaborative approach and designed to build evaluative capacity, engaging people in their own self-assessment and learning. The process typically consists of three steps: 1) mission; 2) taking stock; and 3) planning for the future. Strategies are monitored and information is fed back to make mid-course corrections and/or build on successes. The process depends on cycles of reflection and action in an attempt to reduce the gap between theories of action (espoused) and theories of use (observed behavior). The approach relies on critical friends to help facilitate the process. This is an important case example organizationally because the effort represents a rare opportunity to align and build on medical student education, resident training, and the education of fellows. The data generated are used to inform decision making, improve curricular practices, and enhance critical judgment. |
| Using Empowerment Evaluation to Engage Stakeholders and Facilitate Curriculum Reform |
| Jennifer Berry, Stanford University, jenberry@stanford.edu |
| David Fetterman, Stanford University, profdavidf@yahoo.com |
| When Stanford University School of Medicine undertook a major reform of its curriculum, the School adopted an empowerment evaluation approach to help monitor and facilitate implementation of the new curriculum. Empowerment evaluation is collaborative, engaging faculty, students, and administration in the cyclical process of reflection and action. Empowerment evaluation relies on the theory of process use. Empowerment evaluation theories and tools were used to facilitate organizational transformation at the course level. Our process included: using the School's mission as a guide; taking stock by holding focus groups and developing new survey instruments, including learning climate assessments; and planning for the future by facilitating discussions about evaluation findings with key stakeholders and having the faculty and teaching assistants revise specific courses. We also established a feedback loop to measure the success of reforms and revisions from one year to the next. Case examples highlight measurable evidence of curricular improvement. |
| Organizational Learning Through Empowerment Evaluation: Improving Reflection Skills With a 360 Degree Evaluation |
| Kambria Hooper, Stanford University, khooper@stanford.edu |
| This study explores the impact of a 360 degree empowerment evaluation system in one of Stanford School of Medicine's required classes for Stanford medical students. This evaluation system has three levels of reflection and improvement. The first is the individual member's performance. The second level of reflection and improvement is small group performance. The final level is organizational learning; the course directors and staff reflect on data, looking for group variability or patterns, to create new goals for the course structure or curriculum. Organizational learning is dependent on each member's ability to give and receive constructive, formative feedback. In response to resistance and confusion around the new evaluation system, we developed several interventions to improve the ability of students, faculty and simulated patients to give and receive constructive feedback. This evaluation demonstrates how organizational learning is improved when the organization's members have opportunities to reflect on individual and team performance. |
| Overestimation of Skills in Medical School: The Need to Train Students How to Self-assess |
| Andrew Nevins, Stanford University, anevins@stanford.edu |
| Stanford's School of Medicine used standardized patients (SP) to help assess medical students' skills. This study focuses on students at the preclinical or course level. Clinical skills were assessed by checklists compiled from a consensus of faculty experts. Students also rated their perception of patient satisfaction on a 1 (low) to 9 (high) scale. SPs completed a matching questionnaire, rating their satisfaction with the student. Student and SP satisfaction ratings were paired and correlated, consistent with empowerment evaluation practices. Overall, students over-rated their performance by 0.75 points. The lowest quintile overestimated performance by 1.57 points, while the highest quintile underestimated performance by 0.003 points (p<0.01). The most significant finding is that lower-performing medical students consistently overestimate their clinical skills. This study highlights the significance of properly training students to conduct more accurate self-assessments. Curricular efforts to improve student self-reflection may improve both clinical skills and patient interactions. |
| Empowerment Evaluation: The Power of Dialogue |
| David Fetterman, Stanford University, profdavidf@yahoo.com |
| Jennifer Berry, Stanford University, jenberry@stanford.edu |
| Empowerment evaluation has 3 steps including mission, taking stock, and planning for the future. However, the middle stage is not always explored in depth. One of the central features of the taking stock step is dialogue. Program participants rate how well they are doing at this step in the process, using a 1 (low) to 10 (high) rating systems. They are also required to provide evidence to support their ratings. However, it is the generative dialogue that is most characteristic of this part of the process and critical to authentic learning, on the community of learners as well as organizational learning levels. Each participant explains why they gave their rating, using documentation to build a culture of evidence. Three examples of dialogue (and norming) are provided: 1) engaged scholarly concentration directors; 2) faculty, administrators, and students grappling with curricular problems; and 3) committed clerkship directors guiding student learning in hospitals. |
| Using Principles of Empowerment Evaluation to Build Capacity for Institutional Learning: A Pilot Project at Stanford Hospital |
| Heather A Davidson, Stanford University, hads@stanford.edu |
| Residency education is rapidly changing from an apprentice-based to a competency-based model where performance outcomes must guide individual feedback and continuous program improvement to meet new accreditation standards. This change represents a cultural shift for teaching hospitals and a management shift that must support systems of assessment. Many faculty members do not have the tools needed to design and implement these goals. Since institutional accreditation requires that all residency programs undergo a peer-led internal review process, Stanford Hospital has created a new protocol to build evaluation capacity. Utilizing principles of empowerment evaluation, the pilot project formalizes feedback loops needed at both program and institutional levels. By combining performance benchmark and portfolio techniques with a mock accreditation site visit, the new protocol provides a more comprehensive assessment of overall program needs; evidence of program quality across the institution; and supports a learning culture where faculty share educational initiatives. |
| Sixth Presenter Alice Edler 6507236412 edlera@aol.com Stanford University Empowerment Evaluation: A Catalyst for Culture Change in Post Graduate Medical Education. Pediatric anesthesia requires special skills for interacting with small patients, not required in general anesthesia training. Empowerment evaluation was used to assess these behaviors in a Stanford pediatric anesthesia fellowship. Trainees, faculty and aggregate data revealed the need for more clinical decision- making opportunities in the fellowship. Clinical judgment ranked the lowest. The role of administrative chief fellow emerged from the self-assessment. It allowed for more opportunities for decision making in day-to-day schedules, curriculum, and disciplinary decisions. This position was rotated over all the fellows. Individual and group improvements were evidenced. Fellows assumed the responsibility for creating new rotations and revising their schedules based on perceived curriculum needs. Faculty evaluations of clinical judgment significantly increased in the clinical judgment item (see table 1). Information from the EE has allowed fellows to model self determination, form a more cohesive group, and provide opportunities for high stakes clinical decision-making. |