|
Session Title: Evaluating Program Sustainability: Definitions, Methods, and Evaluation Dilemmas
|
|
Panel Session 883 to be held in Mineral Hall Section E on Saturday, Nov 8, 1:20 PM to 2:50 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Chair(s): |
| Mary Ann Scheirer,
Scheirer Consulting,
maryann@scheirerconsulting.com
|
| Discussant(s):
|
| Laura Leviton,
Robert Wood Johnson Foundation,
llevito@rwjf.org
|
| Abstract:
An innovative perspective for evaluation policy and practice is to use evaluation across the full range of a program or project's life cycle, from initial needs assessment and planning to evaluating the sustainability of programs after their initial funding has closed. This panel will focus on methods for evaluating a potential end stage of the program life cycle: whether the program's activities, benefits, or other outcomes are sustained beyond its initial funding. We will present evaluations illustrating various aspects of the sustainability of their underlying projects. We will discuss and compare a variety of evaluative methods used to collect data about sustainability, including on-line and other types of surveys, an organizational assessment form, and interviews with project staff. Lessons learned will be discussed about both the methods to evaluate sustainability, and what funders might do to foster greater sustainability of their programs.
|
|
Operationalizing Sustainability as a Project Outcome: Results from an On-line Survey
|
| Mary Ann Scheirer,
Scheirer Consulting,
maryann@scheirerconsulting.com
|
|
Evaluative research for questions of program sustainability has expanded substantially in recent years, but definitional issues remain. This presentation will provide definitions for four different types of sustainability as potential outcomes of health programs. These definitions will be illustrated with descriptive findings from an on-line survey to 'look back' at the extent and types of sustainability that occurred among 48 community-based projects that had received short-term funding from a foundation-funded health program in New Jersey. Large percentages of respondents reported positively to each of four types of sustainability measures - maintaining program activities, continuing to serve substantial numbers of clients, building and sustaining collaborative structures, and maintaining attention to the ideas underlying the projects by disseminating them to others. Strengths and limitations of this methodology for future evaluation will be discussed.
|
|
|
Implications of Organizational Maturation for Evaluating Sustainability
|
| Russell G Schuh,
University of Pittsburgh,
schuh@pitt.edu
|
|
The Staging Organizational Capacity (SOC) is an observational protocol based on a maturity model designed for assessing the developmental maturity of nonprofit service organizations. Just as Tanner Staging led to greater precision in identifying maturation in children than the crude measure of chronological age, the SOC is providing a more refined understanding of organizational maturation that has potential implications for measuring and evaluating sustainability. Developed for the Small Agency Building Initiative (SABI) of the Robert Wood Johnson Foundation, the SOC identifies a standard set of features that change systematically as organizations mature. The potential influence of maturational patterns and developmental dynamics for sustainability of initiatives within and of organizations will be discussed.
| |
|
Where are They Now? Assessing the Sustainability of Foundation Grants
|
| Karen Horsch,
Independent Consultant,
khorsch@comcast.net
|
|
This presentation will focus on methodology and lessons learned from conducting evaluations of the sustainability of grant-funded projects of two different health conversion foundations (Endowment for Health and MetroWest Healthcare Foundation). The presentation will provide:
- A brief overview of the two grantmaking organizations and their grantmaking approaches
- The guiding evaluation questions.
- An overview of methodological challenges to assessing sustainability and how they were addressed including:
o defining sustainability and operationalizing this for evaluation
o assessing sustainability of different types of projects (service delivery projects, planning efforts, systemic change projects, and those focused on organizational capacity building)
o determining the appropriate time to assess sustainability
o creating a culture in grantmaking organizations that supported learning from the experiences of past grants rather than focused on how many 'good bets' had been made
- Description of methodological approaches, including web-based surveys as well as phone interviews.
| |
|
Assessing Sustainability of the Lifestyle Education for Activity Program (LEAP): Methodology and Lessons Learned
|
| Ruth Saunders,
University of South Carolina,
rsaunders@sc.edu
|
|
The Lifestyle Education for Activity Program was a school-based intervention that changed school environment and instructional practices (9th grade PE) to promote physical activity (PA) in girls. In the main study, 45% of girls in intervention versus 36% in control schools reported vigorous PA; schools with higher (n=6) versus lower levels of implementation (n=5) had more active girls (48% versus 40%). The follow up study used three data sources (PE observation, PE teacher interviews, and 9th grade PE student focus groups) to assess instructional practice and three (PE observation, PE teacher and former LEAP team interviews) to assess environmental elements. We triangulated these results to classify schools as 'implementing' or 'non-implementing' at follow up. Schools that were 'high implementers' in the main study and 'implementers' at follow up were defined as 'maintainers'. We discuss our approach to triangulating quantitative and qualitative data sources to assess follow up implementation and maintenance and lessons learned.
| |