| Session Title: A Multi-Case Study of Organizational Capacity to Do and Use Evaluation |
| Multipaper Session 541 to be held in Panzacola Section G1 on Friday, Nov 13, 1:40 PM to 3:10 PM |
| Sponsored by the Evaluation Use TIG and the Organizational Learning and Evaluation Capacity Building TIG |
| Chair(s): |
| J Bradley Cousins, University of Ottawa, bcousins@uottawa.ca |
| Discussant(s): |
| J Bradley Cousins, University of Ottawa, bcousins@uottawa.ca |
| Abstract: We know more and more evaluation capacity building (ECB) but our knowledge base is limited by a paucity of empirical research and a failure to integrate the capacity to do evaluation and the capacity to use evaluation into a coherent framework for understanding. In this session original findings will be presented from diverse case organizations participating in a multi-case study. Research at each case organization was guided by a common conceptual framework and sought to: 1.Clarify the nature of the capacity to do and use evaluation. 2.Understand the factors and conditions influencing the integration of evaluation into the organizational culture. The mutli-case study will be introduced by the Cousins followed by highlights of each of the six case studies presented by respective researchers. The session will conclude with cross-case analyses presented by Cousins. Implications for ongoing research and practice will be discussed. |
| Evaluation Capacity at Human Resources and Social Development Canada |
| Robert Lahey, REL Solutions, relahey@rogers.com |
| Isabelle Bourgeois, National Research Council Canada, isabelle@storm.ca |
| J Bradley Cousins, University of Ottawa, bcousins@uottawa.ca |
| The organization studied in this case is one of the largest federal departments in Canada, responsible for a variety of social programs aimed at a variety of clients and stakeholders (special needs, children, women, aboriginals, etc.). Many of its programs are grants and contributions programs that are managed by the department but are delivered by third party organizations or that are interdepartmental in nature. The organization was of high interest to us because it is probably the federal department with the longest history of formalized evaluation (some 30+ years), and has the largest internal evaluation unit in the federal government. |
| Evaluation Capacity at the Canadian Cancer Society |
| Steve Montague, Performance Management Network, steve.montague@pmn.net |
| Anna Engman, , afengman@gmail.com |
| The Canadian Cancer Society (CCS), founded in 1938, is the largest health charity in Canada. It raises in excess of $180 million per year through various fundraising activities and special events. The funds are spent on research, information, prevention, advocacy and support services targeted to the reduction of incidence and mortality rates due to cancer and the enhancement of the quality of life of those living with cancer. The CCS does not have an internal team dedicated to evaluation but is unique in among our cases for two reasons. First, it is a national not-for-profit organization, and second, it relies extensively on partnerships as an approach to developing evaluation capacity. |
| Evaluation Capacity at the Ontario Trillium Foundation |
| Keiko Kuji-Shikatani, Ontario Ministry of Education and Training, kujikeiko@aol.com |
| Catherine Elliott, University of Ottawa, elliott.young@sympatico.ca |
| The Ontario Trillium Foundation (OTF), one of Canada's leading grant-making foundations, is an agency of the Ontario Ministry of Culture. OTF's mission is "building healthy and vibrant communities throughout Ontario by strengthening the capacity of the voluntary sector, through investments in community-based initiatives." With grant volume increasing, OTF needs to continue to make judgment calls based on evidence and clear indications of impact. OTF requires evaluation from all agencies and programs it funds and uses evaluation for a variety of internal purposes. OTF is also actively examining the issue of evaluating Foundation performance incorporating the Centre of Effective Philanthropy approach to examine issues such as board governance, defining outcomes strategically, and importance of evaluation as an organizational learning tool. |
| Evaluation Capacity at the International Centre for Development Research |
| Courtney Amo, Social Sciences and Humanities Research Council of Canada, courtneyamo@hotmail.com |
| J Bradley Cousins, University of Ottawa, bcousins@uottawa.ca |
| IDRC is a large Canadian research funding agency in international development. It is a Crown corporation created 1970 to help developing countries use science and technology to find practical, long-term solutions to social, economic, and environmental problems. We recruited IDRC because we knew it to be highly developed in terms of its capacity to do and use evaluation. IDRC recognizes the essential role that evaluation plays in managing research projects effectively and producing relevant results from the research process. The Centre's overall approach to evaluation prioritizes equally the use and adoption of evaluation findings obtained through the application of rigorous methods, and the development of evaluative thinking capacities through evaluation processes. |
| Evaluation Capacity at the Canadian Mental Health Association, Ottawa Branch |
| Tim Aubry, University of Ottawa, taubry@uottawa.ca |
| The Ottawa Branch of the Canadian Mental Health Association is a non-profit organization that has been in operation since 1953. It began with a group of Ottawa volunteers concerned about the mental health needs of the citizens of Ottawa. Today the agency offers employment to approximately 100 case workers and other mental health professionals, and serves upwards of 700 clients on any given day. The organization has had a longstanding commitment to evaluation as evidence by an integration between services and evaluation; the development and use of program logic models and the development of partnerships with universities and community of practice stakeholders. CMHC is committed to evaluating all new funded initiatives, and in the absence of the availability of such funding, will often seek it elsewhere. |
| Evaluation Capacity at the Canada Revenue Agency |
| Swee Goh, University of Ottawa, goh@telfer.uottawa.ca |
| Catherine Elliott, University of Ottawa, elliott.young@sympatico.ca |
| The organization studied in this case is a large federal government agency in Canada, responsible for the administration of the Canadian Tax Act which involves the collection of tax revenues and the enforcement of the act. Under the Director-General (DG) of Audit and Evaluation is a group of auditors and evaluators separated into two groups each under a Director. The case was of high interest to us because of its strong interest in ECB, in the context of a single office of Audit and Evaluation. The evaluation office is comparatively small and many of the evaluators are reasonably new and lack strong experience in either evaluation or in program management. Although there has been a concerted effort at recruiting new evaluators CRA been experiencing difficulty in retaining these new employees. |