|
Session Title: The Power of Context and Its Role in Shaping Evaluations
|
|
Panel Session 389 to be held in Sebastian Section K on Thursday, Nov 12, 4:30 PM to 6:00 PM
|
|
Sponsored by the Presidential Strand
|
| Chair(s): |
| Debra Rog, Westat, debrarog@westat.com
|
| Discussant(s):
|
| Debra Rog, Westat, debrarog@westat.com
|
| Abstract:
Just as evaluators have increasingly recognized the need to understand the "black box" of intervention, so have we become more aware of the need to understand the context in which we operate. In some instances, this understanding involves merely acknowledging the context and how it may have had a role in affecting program implementation or outcomes; in other evaluation situations, context is embraced within the study itself. This panel will provide three case examples in which context played a strong role in shaping one or more aspects of the evaluation approach, including the methods, analysis, and dissemination process. The discussant will present a synthesis of what the three papers may offer in helping us understand the ways in which different features of context - especially the nature of the phenomenon itself being studied, and the dynamics in the political context - influence evaluation practice.
|
|
Context Becomes Foreground: Baseline Study of a Program to Improve College-Readiness
|
| Joy Frechtling, Westat, joyfrechtling@westat.com
|
| Joseph Hawkins, Westat, josephhawkins@westat.com
|
| Debra Rog, Westat, debrarog@westat.com
|
|
A topic of increasing interest to foundations, the current administration, and federal policy makers is finding ways of increasing the percentage of students from traditionally underrepresented populations who pursue, and succeed in, post secondary education. This paper will discuss a baseline study of such a program that illustrates how context, in this case the school or educational environment as well as program maturity, significantly impact both the implementation of a college readiness program and the strategies employed for its evaluation. Focusing on 11 high schools -6 traditional public schools and 5 charter schools--in an urban city with a history of educational challenges, the study uses interviews, focus groups, and document review to provide a rich description of early implementation. This description identifies important contextual factors that facilitate or present obstacles to a program's implementation and, likely, ultimate success. Implications for program design, as well as evaluation design, are discussed.
|
|
|
Ignore Context at Your Own Peril: Evaluation of a Community-based Employment Program for Young Adults With Criminal Justice Involvement
|
| Scott Crosse, Westat, scottcrosse@westat.com
|
| Janet Friedman, Westat, janetfriedman@westat.com
|
|
The New York City Justice Corps is an intensive employment-centered program for young adults with criminal justice involvement that is being implemented by two community-based organizations in New York. Context has shaped the design and implementation of the evaluation of this program in several important ways. For example, contextual factors have influenced the: development of working relationships between service provider and evaluation staff (e.g., overcoming wariness of evaluator staff), collection of data from individual participants in the evaluation (e.g., understanding concerns about terms in instruments), and interpretation of findings on program processes and outcomes (e.g., accounting for the effect of changes in local economic conditions on employment-related outcomes). The paper discusses some of the contextual factors in play and challenges posed by them, and efforts by the evaluators to respond to these challenges.
| |
|
Attending to Context: Strategies for Producing Actionable Evidence in Dynamic Policy Issues
|
| Kathryn Henderson, Westat, kathrynhenderson2@westat.com
|
| Debra Rog, Westat, debrarog@westat.com
|
| Linda Weinreb, University of Massachusetts Medical School, weinrebl@ummhc.org
|
|
Social issues are often dynamic and sensitive to changes in the broader economic and political context. Conducting evaluations and related efforts in areas such as homelessness can be challenging, particularly in providing information that can have relevance to the changing needs of decision makers. This paper describes the experience of designing, implementing, and reporting a study in Massachusetts focused on the factors that influence families' shelter stays and subsequent exit locations. With a strong emphasis on informing both local and state policy, the study incorporated stakeholder feedback throughout the study as a way to keep abreast of policy changes and offer data in the most targeted manner possible. This paper will describe the changes in the policy system at the beginning of the study, the subsequent changes sparked by the economic downturn, and the strategies used by the study team to continue to provide information that is relevant and useful.
| |