| Session Title: An Evaluative Approach to Quality Assurance in Higher Education |
| Multipaper Session 625 to be held in Panzacola Section H1 on Friday, Nov 13, 4:30 PM to 6:00 PM |
| Sponsored by the Presidential Strand and the Assessment in Higher Education TIG |
| Chair(s): |
| William Rickards, Alverno College, william.rickards@alverno.edu |
| Discussant(s): |
| Michael Scriven, Claremont Graduate University, mjscriv@gmail.com |
| Jean A King, University of Minnesota, kingx004@umn.edu |
| Abstract: What happens when an entire sector is required to shift from a compliance/audit model of quality assurance to an evaluative one, one that asks and answers explicitly evaluative questions about the value of outcomes and key contributing processes? Is it possible to build an evaluative culture across an entire sector? Can meaningful self-evaluation thrive alongside an external evaluation function? New Zealand has embarked on a program of reforms to strengthen higher education provision and outcomes. A significant element is a new system for 'evaluative' quality assurance. This session will describe challenges and learnings encountered in: (1) the development of an evaluative approach that meets the needs of diverse providers and other stakeholders; (2) the capacity building efforts to shift from an audit/compliance/monitoring to an evaluative model; and (3) the change management and culture change strategies used to create both mindshift and behaviorshift in participating government agencies and across the sector. |
| The Challenges of 'Evaluative Quality Assurance': Why, How, and Why Now |
| Syd King, New Zealand Qualifications Authority, syd.king@nzqa.govt.nz |
| Why an evaluative approach to quality assurance? This paper outlines the context, the rationale and the challenges associated with the most significant change in quality assurance in higher education in New Zealand in the last 50 years. Topics covered include the drivers for change, what needed to be done and why (what was "the problem"?), the needs addressed and the emerging benefits of the new approach. The sector affected includes New Zealand's eight universities, three wananga (Maori universities), 20 Institutes of Technology and Polytechnics, approximately 750 Private Training Establishments, 39 Industry Training Organizations and a handful of Government Training Establishments. This paper considers the factors underpinning the shift to an 'evaluative approach focusing on outcomes and key contributing processes' - a highly significant shift for providers and the agency staff responsible for evaluating the programs and organizations. |
| Methodological Challenges in Evaluative Quality Assurance: Shining Lights In Dark Corners |
| E Jane Davidson, Davidson Consulting Ltd, jane@davidsonconsulting.co.nz |
| What does an 'evaluative' approach to quality assurance look like? What evaluation theories and methodologies does it draw on, and what does that look like in practice? What was the rationale for the design? There are numerous political and methodological challenges in striking a practical balance among competing stakeholder needs and requirements. For example, diverse organizations need evaluations to be tailored to align with their distinctive contributions, their learners, and their contexts. They also need flexibility in the way 'quality' and 'value' are defined, and in the nature of the data used to answer important evaluation questions. At the same time, education providers find comparable data from within the sector useful for benchmarking, and funding bodies need information that will allow them to compare providers and/or programs. This presentation outlines how evaluation logic and methodology were used to forge a realistic balance among these competing needs and concerns. |
| Applying the Evaluative Approach in Indigenous Contexts, Challenges and Opportunities |
| Nan Wehipeihana, Independent Consultant, nanw@clear.net.nz |
| Kate McKegg, The Knowledge Institute Ltd, kate.mckegg@xtra.co.nz |
| Over the last two decades, a key development in the New Zealand higher education sector has been the growth of Maori providers, delivering education grounded in Maori pedagogy and philosophy. Also, mainstream higher education providers now have an increased focus on supporting the retention and achievement of Maori students. This paper will consider the ways in which the new approach is able to take account of the distinctive contributions of these Maori providers, as well as the needs of Maori students in mainstream situations. Drawing on the experiences of Maori providers and feedback from other providers, the paper will discuss key issues such as the extent to which the approach has credibility and validity in Maori contexts. |
| Evaluation Capability Building for Changes in the Quality Assurance of a Whole Education Sector |
| Sue Walbran, New Zealand Qualifications Authority, sue.walbran@nzqa.govt.nz |
| Phillip Capper, WEB Research, phillip.capper@webresearch.co.nz |
| This paper will outline the early initiatives used by the New Zealand Qualifications Authority to begin developing evaluation capability with over 750 diverse educational organizations. This includes the use of multiple strategies to first develop an understanding and willingness to adopt the change and then to encourage the use of evaluative tools and techniques to strengthen organizational self-evaluation. Initiatives covered in the presentation include the provision of regional workshops to over 1600 participants, the use of a facilitated web-based discussion forum and intensive workshops on the theory and practice of evaluation. Each will be visited in terms of how well each worked in relation to what it set out to achieve as well as some advice for others wanting to use these strategies. The understanding required by government agencies in order to support and progress the change is also explored particularly in terms of how the system must engage with capability building initiatives that will take several years to mature. |
| Manipulations, Manoeuvrings and Machinations: Getting Buy-in, Come Hell or High Water |
| Tim Fowler, New Zealand Qualifications Authority, tim.fowler@nzqa.govt.nz |
| What is required to achieve big shifts in understanding and garner support for an evaluative approach that is not necessarily understood or welcomed by all players? An evaluative approach directly challenges conventional thinking about quality assurance in higher education. It requires all stakeholders - the diverse parts of the sector, the evaluators themselves (both internal and external), the other agencies involved, national politicians, and communities alike - to reconsider what constitutes quality in higher education and how that quality should be recognized, valued and improved. The challenge is to convince stakeholders the change is worth attempting even if not all of their concerns can be allayed in advance, nor success in all circumstances guaranteed at first attempt. To change an entire system with many diverse interests requires courage and commitment - along with the ability to win some battles while keeping a careful eye on the strategy to win the war. |