| Session Title: Lessons Learned From an Evaluation of a Comprehensive Community Initiative |
| Multipaper Session 860 to be held in Mineral Hall Section E on Saturday, Nov 8, 10:45 AM to 12:15 PM |
| Sponsored by the Non-profit and Foundations Evaluation TIG |
| Chair(s): |
| Pennie Foster-Fishman, Michigan State University, fosterfi@msu.edu |
| Abstract: The complexity of comprehensive community initiatives (CCIs) makes evaluation exceptionally challenging. For example, despite their interest in changing population- and community-level outcomes, CCIs often use interventions that target only a few neighborhoods or that target subgroups of residents. In short, this can produce significant mismatches between the scale and scope of the intervention and expected outcomes, ultimately leading to evaluation results that suggest that CCIs models are ineffective. Evaluating CCIs is also complicated by the fact that some interventions are not evenly dispersed throughout the target community, leading to widely varying levels of program dosage. In this panel presentation we will present the evaluation methods we adopted to address these challenges in our evaluation of one CCI. The lessons learned from our evaluation over the past six years will also be discussed. |
| Who Should I Collect Data From: Simple Questions, Complex Answers When Evaluating A Comprehensive Community Initiative |
| Pennie Foster-Fishman, Michigan State University, fosterfi@msu.edu |
| With the expansive visions that drive most comprehensive community initiatives (CCI) to pursue population- and community-level social change, it is tempting to focus CCI evaluations entirely on large scale outcomes. But, taking such an approach ignores the fact that CCIs are often composed of multiple program components that operate at different scales, have different goals and objectives, target different systems and populations, vary in duration, and are implemented to different degrees. This complexity challenges evaluators at even the most basic of levels: who to collect data from, what outcomes to target, and when and how to best collect this information. This presentation will describe how we tackled these basic evaluation questions in our evaluation of one CCI. Specifically, we will discuss the strategies we used to identify which residents and organizations to sample and how we strived to maximize the possibility of dosage variation in our samples. |
| The Pros and Cons of Using Resident Surveys to Measure Changes in Resident- and Neighborhood-Level CCI Outcomes |
| Jason Forney, Michigan State University, forneyja@msu.edu |
| Steven Pierce, Michigan State University, pierces1@msu.edu |
| Charles Collins, Michigan State University, ccollins1981@gmail.com |
| Soyeon Ahn, Michigan State University, ahnso@msu.edu |
| This presentation will discuss the pros and cons of using resident surveys to evaluate whether a community hosting a comprehensive community initiative (CCI) experienced improvements in resident- and neighborhood level outcomes targeted by the CCI. We will discuss the survey goals and objectives, how the survey was administered, the survey design and sampling challenges we encountered, the most important methodological features of the study we implemented, and data management and analysis issues arising from those methodological choices. We will present key findings related to the value of the door-to-door outreach effort we used to encourage residents to respond, response rates and levels of longitudinal attrition, and findings related to the major evaluation questions addressed by the study. We will also highlight how we used dosage analysis to determine programmatic effects with our survey data. |
| Integrating Secondary Data into the Evaluation of Comprehensive Community Initiatives |
| Melissa Quon Huber, Michigan State University, hubermel@msu.edu |
| Laurie Van Egeren, Michigan State University, vanegere@msu.edu |
| The complexity of comprehensive community initiatives (CCIs) calls for multiple data sources, many of which may already be available as secondary data. Secondary data can be easily accessible and are often used as the 'default' measure of an initiative's impact (e.g., census data). Other secondary data sources, particularly those developed within the local community such as information on neighborhood groups and leaders and community surveys, can provide more detailed snapshots of local impacts but also require creative consideration of the sources available and how they can effectively inform the evaluation. This paper describes lessons learned about the integration of secondary data into a CCI evaluation, including challenges in aligning available secondary data with CCI goals, monitoring shifting definitions of community-defined variables, identifying the full range of relevant secondary data sources available within the community, and developing indicators of the CCI's dosage through the use of these secondary data sources. |
| Using Online surveys to Track Organizational and Service Delivery Network Changes |
| Kristen Law, Michigan State University, lawkrist@msu.edu |
| Collaborative partnerships among service delivery organizations are viewed as an important objective in enhancing CCI success. This presentation will discuss the use of an online longitudinal survey to evaluate and track organizational and service delivery network changes in one CCI. A primary focus of the CCI was to enhance local organizations' capacity to partner both with local organizations and local residents. I will discuss the survey goals and objectives, survey administration, and challenges encountered. I will discuss using measurements of readiness and capacity for change and dosage exposure to predict organizational partnerships and resident involvement within organizations. I will also present key findings related to the outcomes of organization-to-organization and organization-to-resident partnerships and collaboration. |