| Session Title: Systems Oriented Tools and Methods in Public Agency Evaluations: Three Case Studies |
| Multipaper Session 871 to be held in Centennial Section A on Saturday, Nov 8, 1:20 PM to 2:50 PM |
| Sponsored by the Systems in Evaluation TIG and the Human Services Evaluation TIG |
| Chair(s): |
| Beverly Parsons, InSites, beverlyaparsons@aol.com |
| Discussant(s): |
| Teresa R Behrens, WK Kellogg Foundation, tbehrens@wkkf.org |
| Abstract: Application of systems theories to evaluation purposes is gaining increasing interest among the evaluation community. Systems tools and methods, in particular those from Human Systems Dynamics (HSD), are seen as offering potential solutions to some of the challenging evaluation questions faced in complex initiative evaluations. Although awareness of the value of introducing systems thinking into evaluation is growing, many evaluators grapple with how to apply systems methods and tools in evaluations. This session provides three case studies that demonstrate application of various HSD methods and tools in initiatives in health care, in human services and in an education initiative, where interactions are complex and numerous, diversity permeates the system and boundaries are unclear. The presenters will discuss the benefits and new insights gained by integrating HSD approaches into the evaluation as well as the challenges of applying these tools within their specific evaluation context. |
| Using Human Systems Dynamics (HSD) to Evaluate an Initiative to Enhance Interprofessional Collaboration in Health Care |
| Esther Suter, Calgary Health Region, esther.suter@calgaryhealthregion.ca |
| Glenda Eoyang, Human Systems Dynamics Institute, geoyang@hsdinstitute.org |
| Lois Yellowthunder, Hennepin County, lois.yellowthunder@co.hennepin.mn.us |
| Liz Taylor, University of Alberta, liz.taylor@ualberta.ca |
| HSD methods and tools were used to evaluate an intervention designed to improve professional collaboration across health disciplines. Two tools were used, the Container-Difference-Exchange (CDE) model, and the landscape diagram, which shows organized, self-organizing and unorganized dynamics, in order to describe emerging patterns within and across participating sites as well as in the project team. The HSD evaluation complemented evaluation components that focused on processes and outcomes of collaborative practice. The HSD methods and tools highlighted the fluid nature of professional boundaries, focus changes over the duration of the project, changing relationships, ongoing transformations in work processes, and the co-evolutionary relationship upon which collaborative practice depends. Using HSD led to a more complete understanding of the changes occurring as a result of the collaborative practice intervention. |
| Evaluating Change in a Complex System |
| Royce Holladay, Human Systems Dynamics Institute, rholladay@live.com |
| A large county agency with a 5-year plan for significant change, shifting expectations, and looming changes in state and federal funding desired an evaluation of their progress in implementing the change plan accompanied by recommendations for moving the change process forward. This evaluation client wanted to know 1) how 'deeply' change had penetrated the organization, 2) how coherent the perception of the change was across the organization, and 3) what individual changes had been attempted and fully implemented. Tools from human systems dynamics provided for the use of in-depth questions and analysis that allowed evaluators to articulate a clear accurate 'picture' of the progress toward overall change goals, reinforce work that was being done well, and make solid, actionable recommendations about next steps for moving forward. This presentation describes the underlying theories and assumptions, shares the tools and the rationale for their use, and outlines recommendations for moving forward. |
| Use of Systems Oriented Tool in Evaluation of an After-School Science Project |
| Patricia Jessup, InSites, pat@pjessup.com |
| This presentation describes the use of a systems-oriented evaluation that focuses on boundaries, diversity, relationships, and perspectives. The evaluand is an after-school and summer program for students traditionally underrepresented in the fields of science, technology, engineering, and mathematics (STEM). Project goals relate to youth participants' content knowledge, attitudes, interest in and pursuit of STEM subjects and careers, and workplace skills. To investigate the role of broad system factors in shaping the outcomes and processes of the project, InSites developed an adaptation of the CDE (containers, differences, and exchange) model from the Human Systems Dynamics Institute to reveal emerging and unexpected patterns and relationships, changing perspectives, and shifting boundaries. The evaluation results are expected to help leaders of the education system effectively position informal education along side formal education offerings within the education system. |