|
Session Title: Quick and Quality Level Three Evaluations for Corporate Staff Learning
|
|
Panel Session 836 to be held in TRAVIS C on Saturday, Nov 13, 1:40 PM to 2:25 PM
|
|
Sponsored by the Organizational Learning and Evaluation Capacity Building TIG
|
| Chair(s): |
| Jaime Quizon, World Bank, jquizon@worldbank.org
|
| Abstract:
In Level 3 assessments of staff learning in large learning organizations, we are usually interested in knowing whether a staff learning program resulted in desired changes in the attitudes, skills, and/or behaviors of participants, and the extent to which these changes have manifested in the relevant, work-related performance levels of learning participants. The typical Level 3 evaluation, however, is a complex, time consuming, and expensive process. Meanwhile, the opportunities for changing the nature and essentials of most staff learning programs are usually time-bound and driven by change management and budget constraints. Thus, delays in obtaining timely and actionable information to correct the ineffective and costly elements of ongoing learning programs may exacerbate costs and delay the program’s contributions to corporate business results. The session will present two studies that will compare and contrast rigorous (first study) vs. quick, but effective (second study) evaluations based on recent Level 3 assessments of leadership development and communications’ skills programs for World Bank staff.
|
|
A Rigorous Evaluation of a Team Leadership Training Program in a Corporate Environment
|
| Jaime Quizon, World Bank, jquizon@worldbank.org
|
|
This study is based on an actual, rigorous L3 evaluation of a leadership training program at the World Bank. It is based on a quasi-experimental design that followed seven cohorts of training program graduates over time over a 12-18 month period. The study used “before’ and “after” surveys of training program participants (and a matched “control” group), including their respective peers and the managers. These surveys were supplemented by interviews and discussions with selected participants (and “control” group members) to elaborate on issues that arose in the course of the surveys and on the responses received from the open-ended questions in the surveys. Interviews with managers, peers, and program administrators allowed the study to understand better the impacts of the leadership training program from different perspectives.
This first presentation will focus mainly on the process of undertaking a rigorous L3 evaluation of a staff learning program, highlighting the challenges and benefits of such rigorous studies in a corporate setting.
|
|
|
A Quick and Quality Level 3 Evaluation of the Language and Culture Program in a Corporate Environment
|
| Valya Nikolova, World Bank, vnikolova@worldbank.org
|
|
The Language and Culture program Level 3 evaluation is a study of the application of skills gained through the courses offered by the program and on-the-job experience. It measures improvements in participants’ communication and interaction with clients and peers, awareness of participants in understanding their clients’ culture, ability to transfer skills on-the-job, and demonstrate efficient business results. Evaluation respondents include participants from six language courses, peers, managers, and colleagues of the participants, which enables a gap analysis of different stakeholders’ perceptions about the effect of the program.
The evaluation draws on existing sources of data (Level 1 participant feedback and Level 2 achievement tests and oral proficiency outcomes) while providing further in-depth assessment of the end results of the program and focuses on examining participant and peer and management perceptions of effective communication and appropriate interaction between Bank staff and clients and partners. The evaluation utilizes a mix of quantitative and qualitative approaches with emphasis placed on qualitative feedback.
| |