Return to search form  

Session Title: Evaluating National and State Policy Change Efforts: Campaigner and Funder Perspectives on Evaluation Context, Methods and Lessons

Panel Session 102 to be held in Panzacola Section F1 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Advocacy and Policy Change TIG

Chair(s):

Lester Baxter, Pew Charitable Trusts, lbaxter@pewtrusts.org

Discussant(s):

Jacqueline Williams Kaye, Atlantic Philanthropies, j.williamskaye@atlanticphilanthropies.org

Abstract: A sizable share of the growing discussion of advocacy evaluation practice relates to campaigns that seek to effect policy change at the local level, often involving grassroots efforts led by community-based non-profits. This panel seeks to build on this body of work by focusing on advocacy campaigns that seek to inform or effect policy change at the national or state level-- recognizing and exploring the ways in which they operate in a context that can differ sharply from that of local policy change efforts. Informed by The Pew Charitable Trusts' Planning and Evaluation group's experiences over a twelve-year period, the panel will first address methods, incorporating real world examples and sharing lessons panelists have learned about how to design and conduct successful evaluations of advocacy efforts that aim for state or national policy change. We will complement this focus on methods and practice with panel discussions designed to incorporate the pivotally important context of a key audience for advocacy evaluations, the senior strategists who design and implement the campaigns, and the types of evaluative information and lessons campaign designers value most.

Let's Talk Methods! A Systematic Approach to Evaluating State and National Policy Change Efforts, From Evaluation Design to Use Glee Holton, Pew Charitable Trusts, gholton@pewtrusts.org Nicole Trentacoste, Pew Charitable Trusts, ntrentacoste@pewtrusts.org Informed by Pew's own practice in designing evaluations of advocacy strategy, the panel's opening discussion of evaluation methods will incorporate real world examples and focus on lessons that panelists have learned about how to best develop and conduct successful evaluations of national and state policy change efforts. A sample of topics to be discussed includes: understanding key differences with and similarities to traditional program evaluations; identifying clear and measurable policy goals and designing evaluable campaigns; demonstrating practical means of avoiding conflicts of interest and maintaining evaluation integrity; tracking campaign outcomes and choosing appropriate evaluation models; identifying and managing independent evaluators with appropriate evaluation and policy expertise; addressing issues of contribution and causality where multiple actors are working towards similar policy outcomes; timing the delivery of evaluation findings to meet program information needs; and reporting findings in ways that increase utility.

So, How Can We Help? The Campaigner's Context and a Selection of Lessons From State and National Advocacy Efforts Lester Baxter, Pew Charitable Trusts, lbaxter@pewtrusts.org Looking across the body of state and policy change evaluations that Pew has conducted in recent years, this session will incorporate the perspective of Pew's evaluation staff and senior campaigners and share their views of what evaluators need to know about campaigns and campaign strategies. Discussion will focus on elements of advocacy evaluations that are of greatest use to the campaigners' efforts to better understand, design and manage state and national advocacy campaigns. Panelists will also highlight common themes and key evaluation lessons from this sample of state and national advocacy campaigns.

How Did it Work and What Did We Learn? A Case Study of a National Policy Campaign From the Perspective of an Evaluation Consultant and Program Staff Scott Scrivner, Pew Charitable Trusts, sscrivner@pewtrusts.org The panel's final session will focus on lessons learned from a single national policy campaign, both from the standpoint of evaluation practice as well as campaign design and implementation. Hosted by a senior policy and campaign specialist from the project's evaluation team, we will reflect on the challenges of conducting advocacy evaluations, share practical lessons from working with the campaign's partners and funders, and discuss the importance of understanding the campaign's context and key audiences when designing evaluations and sharing lessons. The panel will also share feedback from the senior program and staff who designed and implemented the campaign on the evaluation's key contributions and challenges, as well as the campaign's intended and unintended consequences.

Session Title: Multiple Dimensions in Needs Assessment Application

Multipaper Session 103 to be held in Panzacola Section F2 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Needs Assessment TIG

Chair(s):

Janet Matulis,  University of Cincinnati, janet.matulis@uc.edu

Discussant(s):

Ann Del Vecchio,  Alpha Assessment Associates, delvecchio.nm@comcast.net

Ohio Head Start Needs Assessments: Results and Implications Presenter(s): Hsin-Ling Hung, University of Cincinnati, hunghg@ucmail.uc.edu Mary Marx, University of Cincinnati, mary.marx@uc.edu James Scott, Ohio Department of Education, james.scott@ode.state.oh.us Deborah Smith, University of Cincinnati, smithdeb@ucmail.uc.edu Imelda Castaņeda-Emenaker, University of Cincinnati, castania@ucmail.uc.edu Abstract: In response to the Head Start Reauthorization Act (amended December 2007), a needs assessment of Ohio's Head Start grantees was implemented. The project employed Witkin and Altschuld's (1995) three-phase model as a framework for assessing grantees' collaboration needs in providing services to children and families specific to federal priority areas: Health Care, Services for Children Experiencing Homelessness, Family/Child Assistance, Child Care, Family Literacy Services, Children with Disabilities and Their Families, Community Services, Partnerships with Local Education Agencies, and Professional Development. Data were collected via online survey. A mixed-method data analysis approach was used to assess priority area needs. Discrepancy analysis was performed to identify gaps in achieving expected collaboration in grantees' practices. Project background, findings, and implications will be presented. Utilization of needs assessment findings in the work at local, state, and federal levels will be discussed as well as applications for projects of a similar nature.

Assessing Needs Assessment Instruments: An Example From a Study on Retention Service Needs for Science, Technology, Engineering and Mathematics (STEM) Students Presenter(s): Yi-Fang Lee, National Chi Nan University, ivanalee@ncnu.edu.tw James Altschuld, The Ohio State University, altschuld.1@osu.edu Jeffry White, Ashland University, jwsrc1997@aol.com Abstract: Classical needs assessments (NA) are based on the discrepancy between the desired and current states; estimation of the two conditions is required. This discrepancy rationale is generally accepted and often seen in surveys utilizing multiple scales. The intent of this presentation is to describe various aspects of the analysis of an NA instrument designed to investigate the importance of, satisfaction with and frequency of use on retention services for minority students in science, technology, engineering, and mathematics. The data were collected through a web-based survey using 5-point Likert scales. Principal Component Analysis was conducted for each scale/dimension to check on the patterns of item clusters. In addition, comparisons between student and faculty/administrator versions of the survey were examined.

Health Plan Needs Assessments: A Framework for Labor-Management Committee Utilization Presenter(s): Gregg Gascon, Ohio Education Association, gascong@ohea.org Abstract: Health insurance coverage for non-elderly Americans is provided primarily through public and private employers; in 2006, more than 158 million Americans received health insurance coverage as a condition of employment. The employment-based health insurance system that has developed over time allows employers to deduct their contributions as a business expense and provides employees with a non-taxable benefit. Given rising health insurance costs, labor and management have come together to look for ways to build their capacity to understand how to create health programs that control costs and increase the health status of employees. This paper proposes a framework for health plan needs assessments that can be conducted in the context of labor-management committees created to address rapidly rising health insurance costs in the workplace. The health plan needs assessment framework is designed to provide actionable data as well as to build the capacity of local stakeholders to address other concerns.

Session Title: Demonstration of Evaluation Frameworks in a Variety of Health Projects

Multipaper Session 104 to be held in Panzacola Section F3 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Health Evaluation TIG

Chair(s):

Stacey Farber,  Cincinnati Children's Hospital Medical Center, stacey.farber@cchmc.org

Evaluating the California Clean Air Project: A Multi-Method Evaluation Framework Presenter(s): Pamela Drake, ETR Associates Inc, pamd@etr.org Seow Ling Ong, ETR Associates Inc, seowlingo@etr.org Abstract: This presentation will focus on the 3-year evaluation of the California Clean Air Project (CCAP). CCAP is a statewide initiative that provides in-person, teleconference and web-based assistance, educational materials, consultations and trainings on secondhand smoke (SHS) issues and policies for communities in California. The goal is to protect individuals from the toxic effects of exposure to SHS in workplaces, multi-unit housing, tribal casinos, and outdoors. A variety of qualitative and quantitative methods were used in the evaluation. Outcomes were measured with online surveys and interviews (including case studies in some communities). Methods for the process evaluation included key informant interviews, a database tracking system, website monitoring, media exposure records, and training evaluations. The presentation will examine the challenges faced in evaluating technical assistance related to policy change. It also will show how each data source contributed to an overall picture of the project.

Evaluating Funded Health Research: Application of a Standardized Evaluation Framework to Optimize Performance Results Presenter(s): Kathryn Graham, Alberta Heritage Foundation for Medical Research, kathryn.graham@ahfmr.ab.ca Amy Wong, Alberta Heritage Foundation for Medical Research, amy.wong@ahfmr.ab.ca Heidi Chorzempa, Alberta Heritage Foundation for Medical Research, heidi.chorzempa@ahfmr.ab.ca Liza Jensen, Alberta Heritage Foundation for Medical Research, liza.jensen@ahfmr.ab.ca Liza Chan, Alberta Heritage Foundation for Medical Research, liza.chan@ahfmr.ab.ca Abstract: The Alberta Heritage Foundation Medical Research (AHFMR) is a not-for- profit provincial health research funding agency. In early 2008, AHFMR developed a common framework for managing performance of various programs. The primary focus is to evaluate the impacts of funded health research and program effectiveness. The framework is innovative in that it incorporates three distinct models from evaluation, health research and performance management disciplines. The models are: (1) the logic model, (2) the 'pay back model' (Buxton and Hanney, 1996), and (3) the balanced scorecard (Kaplan and Norton, 1992). Implementation has begun at the organizational level and is currently being cascaded across all programs. Key drivers for the common framework include: (1) accountability in the use of public funds toward health research, and (2) improving organizational efficiency and effectiveness. Application of the comprehensive framework will help AHFMR better achieve its mandate and meet the needs of stakeholders. Buxton, M., & Hanney, S. (1996). How can payback from health services research be assessed? Journal of Health Services Research & Policy, 1(1), 35-43. Canadian Academy of Health Sciences. (2009). Making an impact: A preferred framework and indicators to measure returns on investment in health research. Ottawa (ON): The Academy. Retrieved from www.cahs-acss.ca/e/assessments/completedprojects.php Kaplan, R. S., & Norton, D. P. (1992). The balanced scorecard: Measures that drive performance. Harvard Business Review, 70(1), 71-79.

Randomized Control Trial of an Agricultural Safety Intervention: Application of Theory of Planned Behavior as a Framework for Behavioral Change Presenter(s): Hamida Jinnah-Ghelani, University of Georgia, hamidajinnah@gmail.com Zolinda Stoneman, University of Georgia, zo@ihdd.uga.edu Abstract: Injury and death rates of youth on farms in the U.S. are particularly high. Most approaches in farm injury prevention have focused on knowledge as the primary outcome. Gains in knowledge are minimally effective for changing behavior. Results will be presented from a three-year randomized control trial evaluating the effectiveness of a family-based farm safety intervention that utilizes the 'Theory of Planned Behavior' as a framework for changing safety behaviors of individuals. Sixty families having children between 10 and 19 years, who are active on the farm, were randomly assigned to one of three groups - parent-led, peer led and control group. Analysis of covariance models on the final wave of data collected, controlling for pre-intervention levels of the outcome variables, will be preformed. We postulate that one group will be more effective in positively changing the knowledge, attitudes, intentions, and behaviors of parents than the other.

What's Our Score? The Development of a Scale for Assessing Differences in Implementation of a Multi-site Initiative Presenter(s): Stephanie Herbers, Washington University in Saint Louis, sherbers@wustl.edu Sarah Shelton, Washington University in Saint Louis, sshelton@wustl.edu Jenine Harris, Saint Louis University, harrisjk@slu.edu Abstract: In 2004, a Missouri health foundation established a nine-year initiative to reduce tobacco use in Missouri. Programs funded through the initiative share common goals, but vary in design and implementation. Now in the initiative's fifth year, outcomes such as prevalence of tobacco use are being assessed. As the initiative evaluators, we will present the process we took to develop a scoring scale to rate Missouri counties on the extent to which the initiative is being implemented. The scoring scale takes into account the range of implementation across counties including amount, comprehensiveness, and effectiveness. As the initiative progresses, we will use the scoring scale to look at changes in tobacco use prevalence at the county-level at various points in time. In addition to our process, we will also present our findings and recommendations for how evaluators can use this type of scoring scale in their work.

Session Title: Complex Systems Evaluation and Dynamic Logic Modeling: Lessons From the Field

Panel Session 105 to be held in Panzacola Section F4 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Systems in Evaluation TIG and the Human Services Evaluation TIG

Chair(s):

Bob Williams, Independent Consultant, bobwill@actrix.co.nz

Discussant(s):

Margaret Hargreaves, Mathematica Policy Research Inc, mhargreaves@mathematica-mpr.com

Bob Williams, Independent Consultant, bobwill@actrix.co.nz

Abstract: Understanding the differences between self-organizing and organized system dynamics can be a key to useful planning and evaluation of initiatives in complex settings. The presenters in this session are exploring ways to visually and conceptually integrate attention to self-organizing system dynamics along side attention to the planned, organized system dynamics typically represented in logic models. The initiatives (funded all or in part by the federal Children's Bureau) are an evidence-based home visitation program, parental involvement in early childhood programs, and a quality improvement center for early childhood maltreatment prevention. The panelists illustrate how they framed their thinking, gathered information, and visually represented findings about both organized and self-organizing system dynamics. They particularly address the nature of logic models in these situations. Their focus is on helping users of their work understand options for influencing the key dynamics of their situation to move in a desired direction.

Planning Infrastructure Changes in Complex Systems Melissa Brodowski, United States Department of Health and Human Services, melissa.brodowski@acf.hhs.gov Melissa Brodowski is managing a number of complex initiatives around the topic of child maltreatment prevention at the federal level. In her role as Federal Project Officer, she has started to encourage grantees to apply new knowledge from the complexity sciences to their work. She addresses how she approaches this task and describes the challenges of bringing a complex systems orientation to the work at the federal level. She discusses the challenges and opportunities with applying complexity to the planning and implementation of a new initiative to support evidence-based home visiting programs with 17 grantees across 15 states and for the national cross-site evaluation. She also describes the benefits that the agency and grantees find in the use of logic models to describe their work while considering how to identify and display the patterns within complex systems that are not expressed in the typical logic model.

The Place of Logic Models When Investigating Self-organizing System Dynamics Jacqueline Counts, University of Kansas, jcounts@ku.edu Governmental programs and systems focused on early childhood emphasize parental involvement. Jackie Counts addresses how she and her colleagues identified and encouraged authentic involvement in agency work through a developmental evaluation. They conducted an agency survey of over 90 agencies to identify systems' components from which they developed a logic model. They conducted focus groups with over 100 parents to understand how parents define involvement and access supports. They used the data from the focus groups to understand self-organizing patterns and how those patterns are congruent and incongruent with the logic model communicating the self-organizing patterns of parental involvement along side the agency logic model to help agencies adjust how their work supports the self-organizing approaches of parents

Balancing Attention to Organized and Self-Organizing System Dynamics Beverly Parsons, InSites, bparsons@insites.org Beverly Parsons is the lead external evaluator for the National Quality Improvement Center (QIC) on Early Childhood. The QIC seeks to improve the well-being of children zero to five years old, and their families, who are at risk of abuse and neglect. The QIC fosters collaborative research and demonstration projects across multiple service systems. The presentation addresses how the evaluation is seeking to understand and visualize self-organizing patterns in this complex array of systems along with the multiple organized system dynamics that are rooted in the hierarchical organizations. She addresses where logic modeling is used to depict organized system dynamics while showing the relationship to self-organizing dynamics. She discusses the challenges of identifying, describing, and using dynamic logic modeling in this evaluation.

Session Title: Interactive Techniques to Facilitate Evaluation Learning

Demonstration Session 106 to be held in Panzacola Section G1 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Extension Education Evaluation TIG

Presenter(s):

Ellen Taylor-Powell, University of Wisconsin, ellen.taylor-powell@ces.uwex.edu

Nancy Brooks, University of Wisconsin, nancy.brooks@ces.uwex.edu

Chris Kniep, University of Wisconsin, christine.kniep@ces.uwex.edu

Abstract: At AEA 2008, we had a room full of engaged participants looking for new ideas they could use to help non-evaluators engage in and learn evaluation. We will add to the techniques we shared last year with some new ideas from our toolbox that bring evaluation to life. This year we will include -- learning peripherals, games, and creative expression -- that facilitate active learning of tough evaluation concepts or tasks. We will demonstrate these interactive techniques and provide clear explanations of how we use each to build evaluation capacity. We will discuss their strengths and weaknesses and applications for other evaluation learning purposes, in different settings with different audiences. Bring your own techniques to share.

Session Title: The Social Context of Water Quality Improvement Evaluation: Issues and Solutions

Panel Session 107 to be held in Panzacola Section G2 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Presidential Strand and the Environmental Program Evaluation TIG

Chair(s):

Linda P Thurston, Kansas State University, lpt@ksu.edu

Abstract: State and Federal agencies support a watershed approach to better address water quality problems in the U.S. and to build capacity of watershed stakeholders to develop and implement effective, comprehensive programs for watershed protection, restoration, and management. This local watershed approach to address water quality problems involves water quality specialists, technical advisors, and local stakeholders such as landowners and county extension agents. Contextual issues are a vital consideration in planning and evaluation for local watershed improvements. Evaluation practitioners have much to offer to this watershed approach. Evaluation is essential to developing, assessing, and improving successful curriculum design, trainings, demonstrations, workshops and conservation practices. Evaluators can address involving stakeholders, developing consumer-friendly evaluation frameworks, building evaluation capacity, and identifying indicators and developing assessment tools. This panel will discuss contextual issues in water quality improvement evaluation and will provide examples of evaluation tools and practices in water quality improvement work.

Critical Issues in Evaluating Water Quality Programs in the Unique Watershed Context Bill Hargrove, University of Texas at El Paso, bhargrov@ksu.edu Linda P Thurston, Kansas State University, lpt@ksu.edu Christa A Smith, Kansas State University, christas@ksu.edu Evaluation is an integral component of water quality improvement. The evaluation of small scale quality improvement initiatives at the watershed level is useful for: identifying and dealing with issues as they arise from the project; monitoring the impact of local projects; comparing local projects to draw lessons; and collecting detailed information as part of larger state- or region-wide water quality evaluations. Evaluations of watershed improvement projects also help those involved in the initiatives to optimize their choice of interventions and use of resources. Many of local, state, and national stakeholders are involved in considering the effectiveness of water quality improvement practices. However, this broad array of stakeholders presents several issues that are critical to evaluating program effectiveness in this unique context. Critical issues include: stakeholder buy-in of the evaluation process; difficulty of collecting relevant data; understanding the complex change processes involved; and building local evaluation capacity.

Developing and Using Social Indicators for Environmental Management Linda Prokopy, Purdue University, lprokopy@purdue.edu Ken Genskow, University of Wisconsin Madison, kgenskow@wisc.edu Many involved with Non Point Source (NPS) pollution projects have expertise and knowledge necessary to plan, implement, and evaluate their projects' physical and environmental components, yet addressing and evaluating the social and human-dimension components presents new challenges. In response to this situation, EPA Region 5, state environmental agencies, and the CSREES Great Lakes Regional Water Quality Program (GLRWQP) have initiated a project to incorporate a social component into NPS project planning and evaluation for the region. The effort involves an inter-organizational team drawing from EPA, state environmental agencies, land grant universities in the CSREES Great Lakes Region, and others. The team has developed a framework for tracking indicators of individual change, such as knowledge, awareness, capacity, constraints and behavior. The team has entered into a three year pilot phase with the six states in the region. This paper will describe the process for developing and using the indicators.

Applying Social Indicators to Water Quality Planning and Evaluation Ken Genskow, University of Wisconsin Madison, kgensskow@wisc.edu Linda Prokopy, Purdue University, lprokopy@purdue.edu Social indicators are statistics and other measures that enable assessment of the social trends and the human dimensions of programs and program impacts. The use of social indicators for natural resource management is gradually increasing. Efforts include applications for ecosystem management, watershed management, aspects of sustainability, and social impact assessment. Social indicators for natural resources management encompass a variety of issues, including community capacity and activism, community interaction and information flow, demographic information, economic conditions and employment, education, and property and land use. However, while providing important trend data for community and regional settings, many commonly used social indicators relate only indirectly to the primary goals of conservation and resource management programs. This presentation discusses the use of relevant primary social data to inform water quality planning and evaluation efforts. The presentation will outline the conceptual framework applied to the multi-state regional water quality initiative presented in paper 2.

Building Evaluation Capacity in Local and State Watershed Improvement Programs Christa A Smith, Kansas State University, christas@ksu.edu Linda P Thurston, Kansas State University, lpt@ksu.edu Bill Hargrove, University of Texas at El Paso, bhargrov@ksu.edu Building capacity at local and state levels, and providing data for decision-making and management practices, is crucial to the sustainability of watershed improvement programs. In particular the biological and social aspects of watershed improvement programs must be understood to replicate successful practices and provide accountability to stakeholders. The need to design and test an evaluation framework that will enable stakeholders to document and report on the contributions of watershed improvement programs is vital to achieving this goal. This paper will explore the development of an evaluation toolkit with comprehensive assessment methods for analyzing the biophysical and social impacts of watershed programs.Evaluation questions used to guide the development of the framework were: - What are the biophysical and social indicators of watershed program activities? - What is the impact of watershed programs on short-term, intermediate, and long-term outcomes? - How can stakeholders assess the value of watershed program investments?

Session Title: Dealing Effectively With Different Views and Perspectives: The Circular Dialogue Technique

Skill-Building Workshop 108 to be held in Panzacola Section H1 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Qualitative Methods TIG

Presenter(s):

Richard Hummelbrunner, OEAR Regional Development Consultants, hummelbrunner@oear.at

Abstract: Circular Dialogues are systemic forms of communication among people who come from or act in different contexts. The aim is to use the different perspectives and views of the participants as a resource e.g. for appraising/validating diverse experiences or identifying joint solutions. Therefore these dialogues can be used very effectively by evaluators dealing with different stakeholder perspectives. After a short introduction of the principles and rules to be followed in a Circular Dialogue, participants will be invited to break into sub-groups and to agree on one issue in each to deal with. In addition, at least three perspectives (or roles) will be identified and a facilitator appointed. Then several dialogue sessions will be run in parallel to give the participants a hands-on opportunity to practice. In the final session the experience of these sessions will be discussed, commented by the presenter who has observed them, as well as participants' questions.

Session Title: Design and Construction of Measures

Demonstration Session 109 to be held in Panzacola Section H2 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Presenter(s):

Lee Sechrest, University of Arizona, sechrest@u.arizona.edu

Katherine McKnight, Pearson Corporation, kathy.mcknight@gmail.com

Mei-kuang Chen, University of Arizona, kuang@u.arizona.edu

Abstract: Measures used for evaluation should be deliberately and carefully conceptualized, designed, and constructed in order to provide the best possible information. That is true even if, eventually, measures are selected from among those already available. Three classes of measures can be identified: common factor measures, emergent variable measures, and quasi-latent variable measures. Each of these classes of measures entails particular assumptions about their structure and require distinctly different approaches to their construction and interpretation. The three classes of measures are defined and differentiated, and the implications of their definitions for reliability and validity are outlined. Procedures for the construction and assessment of each of the three types of measures are presented and illustrated with simulated and real data. Their use in evaluation and their proper interpretations are discussed.

Session Title: Internal Evaluation: Its Unique Contexts Challenges, Opportunities and Uses

Panel Session 110 to be held in Panzacola Section H3 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Evaluation Use TIG

Chair(s):

Wendy DuBow, University of Colorado at Boulder, wendy.dubow@colorado.edu

Discussant(s):

Wendy DuBow, University of Colorado at Boulder, wendy.dubow@colorado.edu

Abstract: Internal evaluators often encounter both challenges and opportunities unique to their status as insiders. The panelists in this session have all worked as both external and internal evaluators and, therefore, bring a perspective on how the issues they currently face as internal evaluators differ from, or are similar to, those they have faced as external evaluators. Many of these issues have a direct impact on the use of the evaluations they have conducted. The three panelists will share the challenges and opportunities they have encountered as internal evaluators and focus on how these issues impacted the evaluation utilization. The discussant will provide an opportunity for audience participation in a discussion of the larger philosophical and practical issues these situations spark. As the panelists and discussant all work in different fields, the inter-disciplinary discussion promises to be stimulating.

MutuallY Reinforcing Uses of Internal Evaluation and Evaluation Capacity Building (ECB) Boris Volkov, University of North Dakota, bvolkov@medicine.nodak.edu Both internal evaluation and evaluation capacity building (ECB) have increasingly become catch-phrases in the program evaluation and organization development field. Although ECB practice is distinguished from program evaluation, the goal of ECB is to reinforce and sustain effective program evaluation practices. One of the means for achieving this goal is increasing an organization's capacity to design, implement, and manage effective evaluation projects. This capacity in many organizations is a function of an internal evaluation unit. Both theory and practice show that, on the one hand, evaluation capacity building is a viable approach for the development of (internal) evaluation systems. On the other hand, internal evaluation practice can--and in many cases does--contribute to developing an organization's capacity to think and act evaluatively. The panelist will discuss points of convergence between ECB and internal evaluation, co-evolving and mutually reinforcing uses of practices, and the systematic process of building evaluation capacity in organizations.

Perception of the Evaluator as External or Internal Affects Stakeholder Evaluation Use Samuel Held, Oak Ridge Institute for Science and Education, sam.held@orau.org This panelist is the lead program evaluator for a not-for-profit organization with multiple funding streams (including Federal grants and contracts), multiple partners covering a range of local, regional, and national programs. His position exists both in the not-for-profit and in a Federal institute that his organization manages, meaning he is an integrated part of a Federal agency that funds programs which he is tasked to evaluate by the agency. In this complex role, he aims to provide neutral, external evaluations of programs that his organization manages and that the Federal government funds. His identity as an internal or external evaluator is not always clear, even to him. Yet, stakeholders' perceptions of his identity (i.e., his connection to the program or organization) affect the evaluations and studies his office conducts. He will share examples of how he has navigated these complexities, while trying to ensure use of evaluation findings.

Session Title: Evaluation During Challenging Economic Times: Strategies for Non-profits and Foundations

Think Tank Session 111 to be held in Panzacola Section H4 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Presenter(s):

Courtney Malloy, Vital Research, courtney@vitalresearch.com

Pat Yee, Vital Research, patyee@vitalresearch.com

Discussant(s):

Courtney Malloy, Vital Research, courtney@vitalresearch.com

Pat Yee, Vital Research, patyee@vitalresearch.com

Harold Urman, Vital Research, hurman@vitalresearch.com

Abstract: This session will examine how non-profit organizations and foundations can continue to support evaluation activities during challenging economic times. Participants will generate strategies that organizations can use to control costs while still collecting and analyzing high quality evaluation data. Participants will choose to participate in two of three breakout groups for 20 minutes each throughout the 90-minute session. Breakout groups will discuss the following topics: 1. Choosing what, when and how to evaluate (e.g., which programs and/or evaluation questions, timing, funding options, staffing, etc.); 2. Designing evaluations (e.g., instrumentation, sampling, data sources, use of findings, etc.); and 3. Leveraging technology: Making the right investment choices. Summary reports from each topic will be provided by breakout leaders followed by comments and questions from participants. Results from the think tank will be documented and made available to AEA as well as posted on an evaluation resources web site hosted by the facilitators.

Session Title: Evaluation in International Development and Oversight

Multipaper Session 112 to be held in Sebastian Section I1 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Government Evaluation TIG and the International and Cross-cultural Evaluation TIG

Chair(s):

Sarah Brewer,  United States Department of State, brewerse@state.gov

Audit Criteria in Performance Audit: How Do They Impact on Design and Results Found? Presenter(s): Kristin Reichborn-Kjennerud, Work Research Institute, rekr@afi-wri.no Abstract: Several authors have described the development towards an audit society. This development is seen as a consequence of New Public Management. Increased delegation has led to increased use of audit and control mechanisms. Performance audit is a specific form of audit that resembles evaluation. A performance audit is a normative study conducted by national, municipal and private audit institutions, normative because the auditors assessments always are based on audit/evaluation criterias. A given context influence the design of an audit/evaluation and thereby the findings and the results. Audit/evaluation criteria that are too limited might prevent the auditor from grasping results that are important for the stakeholders. This paper investigates this context and how the design and the setting of criterias impacted on three studies in the public healthcare-and social security sector. In the end I discuss in what way the practice is in line with standards of the performance audit profession.

Application of New National Evaluation Rules, Regulations and Policies to International Development Programs: Challenges and Opportunities Presenter(s): Satianan Debidin, Canadian International Development Agency, satianan.debidin@acdi-cida.gc.ca Abstract: Over the past few years the Government of Canada (GoC) has taken several reform steps regarding program planning and implementation to support reporting to parliament. A few of the most obvious instruments introduced are the Result Management Accountability Framework, the Accountability Act and other new rules and regulations, all created with the view to improve efficiency, effectiveness, transparency and achieving of and reporting on results across all federal departments and agencies, and therefore, CIDA, an active member of the donor community, has complied.

Missing the Strategic Wood for the Tactical Trees: The Mistakes Made in Evaluating Oversight Institutions and How to Correct Them Presenter(s): Terence Beney, Feedback Research & Analytics, tbeney@feedbackra.co.za Abstract: The logic of the democratic system holds that the executive is held accountable by the legislatures. Legislatures in turn rely on the oversight institutions tasked with evaluation (e.g. Government Accountability Office) for the information with which to assess and address the actions of the executive. In this hypothetical context it is sufficient to evaluate the evaluators by scrutinizing their credibility (independence, productivity, quality of evaluation outputs) and the utility of their evaluations (relevance, dissemination, usefulness of recommendations). Where the evaluators have been evaluated this output-focused approach has become the convention. In a real world context however the burden of oversight cannot be abdicated to imperfectly functioning legislatures. Evaluation offices share that burden, explicitly or implicitly, and should be evaluated accordingly. The evaluation of South Africa's Public Service Commission is used to illustrate the introduction of two assessment dimensions in addition to the evaluation of output against credibility and utility criteria: 1) the evaluation of outcomes derived from the operationalization of their mandate and 2) the evaluation of their strategic advocacy efforts.

Enhancing Democracy Assistance Through Better Evaluation: Measuring the Impact of United States Agency for International Development's (USAID) Political Party Assistance Presenter(s): Altin Ilirjani, United States Agency for International Development, ailirjani@gmail.com Abstract: This paper will discuss recent efforts to evaluate the impact that external aid of US Agency for International Development (USAID) has had on development of democratic political parties. Although political party assistance programs have shown some degree of diversity and variation from country to country, a major focus of party assistance programs has been technical assistance in party-building efforts, often with a focus on party-building through elections. Much of the work of USAID has focused on training party activists and leaderships, strengthening the organizational capacities of parties and improving their abilities to contest elections. Quantitative analysis for this paper are based on results of a new survey of USAID's democracy and governance advisors and activity managers, that aims to identify factors affecting USAID's political party program decisions and design; as well as special challenges to working with political parties as compared to other democracy promotion programs.

Session Title: Strengths and Limitations of Collaborative, Participatory, and Empowerment Evaluation Approaches

Think Tank Session 113 to be held in Sebastian Section I2 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

David Fetterman, Fetterman & Associates, fettermanassociates@gmail.com

Discussant(s):

Rita O'Sullivan, University of North Carolina, ritao@unc.edu

Lyn Shulha, Queens University, lyn.shulha@queensu.ca

Abraham Wandersman, University of South Carolina, wanderah@gwm.sc.edu

Liliana Rodriguez-Campos, University of South Florida, lrodriguez@coedu.usf.edu

Abstract: This will be a participant focused, interactive session. Members of the audience will be asked to work together and list the strengths of collaborative, participatory and empowerment evaluation approaches on poster paper. They will rotate and list the major limitations of each approach. The group will also be asked to rotate one more time and complete a list of recommendations to respond to their list of limitations. Leaders in the group will be asked to report back a summary of their findings and insights. A panel of experts in the field will respond to the lists and add their reflections concerning the lists. Panel experts will include: Collaborative Evaluators Rita O'Sullivan and Liliana Rodriguez-Campos; Participatory Evaluator Lyn Shulha, and Empowerment Evaluators David Fetterman and Abraham Wandersman.

Session Title: Can Whole Systems Be Evaluated? If so, How?

Think Tank Session 114 to be held in Sebastian Section I3 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG and the Systems in Evaluation TIG

Presenter(s):

Molly Engle, Oregon State University, molly.engle@oregonstate.edu

Discussant(s):

Martha Ann Carey, Maverick Solutions, marthaann123@sbcglobal.net

Mary Ann Scheirer, Scheirer Consulting, mascheirer@gmail.com

Andrea M Hegedus, Northrop Grumman Corporation, ahegedus@cdc.gov

Abstract: As evaluators, we are well aware that we work within larger systems that form the context for both the evaluand and the evaluation process. Rarely is the evaluand a complete system itself. Other questions become relevant including what improvements can result if the whole system is evaluated? Can evaluation feasibly address initiatives intended to change the ways a broader system operates? This session will address these and other related questions. The facilitators' work with whole systems has raised these questions, with no easy answers. Evaluators tend to evaluate parts of systems, but have neglected putting the parts together to evaluate the whole. Understanding interrelations among the parts may be a key to improvement in a specific program within its systemic context. The participants will contribute to this thought provoking session by sharing work they have done with whole systems and taking away suggestions for exploring whole systems evaluation further.

Session Title: We See With More Than Our Eyes: Gathering Data in Migrant and Native American Communities

Skill-Building Workshop 115 to be held in Sebastian Section I4 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Multiethnic Issues in Evaluation TIG

Presenter(s):

David Sul, Sul & Associates LLC, dsul@sulandassociates.com

Charlene Sul, Sul & Associates LLC, csul@sulandassociates.com

Abstract: The story telling tradition is strong in Migrant and Native American communities. Expert story tellers use more than their voices to transport the listener to a distant time and place. The techniques presented in this workshop remain true to that tradition. Incorporating modern technologies, the evaluators attempt to share the program participant experience and in so doing, add even more context to the evaluation. The evaluators will present findings from a collection of methods - the family portrait, digital stories, photo mosaic and word clouds. These techniques are used to draw out hidden messages in the lives of program participants. Additionally, they can be used to monitor individual or group progress both instantaneously and over time. Workshop participants will participate in a live deployment of selected techniques. A debriefing of the experience will focus on the impact such techniques may have as critical reports are being created. Further, participants will be left to consider how to substantiate the evidence created in summary reports with prior work and a credible framework for analysis.

Session Title: Incorporating the International Political and Cultural Context in Evaluation

Multipaper Session 116 to be held in Sebastian Section L1 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Gwen Willems,  Willems and Associates, wille002@umn.edu

Discussant(s):

Gwen Willems,  Willems and Associates, wille002@umn.edu

Development Evaluation in Context: Tensions, Opportunities, and the Need for a More Integrated Approach Presenter(s): Lesli Hoey, Cornell University, lmh46@cornell.edu Mark Constas, Cornell University, mac223@cornell.edu Abstract: The notion of development represents varied and often conflicting interests. Intentionally or unintentionally, interventions express a particular set of development priorities. While these priorities may be internally consistent to a specific program, they may not reflect the larger mandate of the implementing agency itself, the goals of parallel development initiatives, the long-term needs of local actors, or concerns that become apparent within larger geographical scales. The present paper explores the factors that determine the extent to which a given evaluation will support or undermine broader development priorities. We first provide a framework to investigate the context of development priorities within which a given program is embedded and argue that such investigations are an essential first phase of work for evaluations in developing contexts. The second part of the paper examines the consequences that such investigations have for the design, implementation and effects of development evaluation.

The Politics and Consequences of Participation in International Development Evaluation Presenter(s): Anne Cullen, Western Michigan University, anne.cullen@wmich.edu Abstract: Despite their wide-spread use, there is a dearth of research on the impact of participatory approaches in international development evaluations. Although proponents of participatory approaches to international development evaluation assert many advantages of their use, the evidence to support these claims is largely anecdotal. Similarly, critics of participatory approaches do not have empirical data on which to base their assertions. Without systematic scientific study, it is difficult to repudiate or substantiate any of these claims. This session presents the findings of an empirical study on participatory approaches to international development evaluation undertaken to (i) better understand current trends and practice; (ii) describe the perceived impacts of participatory evaluation; and (iii) help improve future evaluation practice. Ultimately, this presentation will contribute to empirical knowledge on evaluation practice, particularly as it relates to stakeholder participation in the international development evaluation.

Organization Paradigms and Evaluation Presenter(s): Alexey Kuzmin, Process Consulting Company, alexey@processconsulting.ru Abstract: Organizations are different. They have different 'personalities' or organizational cultures, they are built and operate in different ways. According to Larry Constantine (1993) an organizational paradigm is 'both a standard or model for an organization and a world view, a way to make sense of organizational reality'. Since evaluation should be built in the organizational reality, it's important to identify and consider organization paradigm while choosing the most relevant evaluation approach for a particular program or an organization. In this presentation we shall explore the four organization paradigms described by Constantine (closed, synchronous, random, open) from an evaluator's point of view and suggest how to design useful evaluations with consideration of organizational context.

If, Then, and So What: Theory and the Primacy of Field Experience for International Evaluation Presenter(s): Catherine Elkins, RTI International, celkins@rti.org Abstract: Too often evaluations are driven by external stakeholders, who rarely understand the field context well enough to articulate meaningful queries, or the intervention theory well enough to test its embedded hypotheses. In international evaluation, the full complexity of the field context must inform every aspect of the evaluation: which concepts are central, what can be measured, how it can be measured, who can contribute in which ways, and the extent to or direction in which generalizability can be attempted. Without clearly articulated theory and distinguishable, testable hypotheses, a cross-cultural evaluation study risks producing findings that are little more than anecdotes regardless of its empirical evidence. This paper presents monitoring and evaluation (M&E) and evaluation case studies to integrate theoretical rigor with appreciation of key elements of the local operational context in order to produce study results that are locally and generally useful and useable toward strengthening development theory.

Session Title: Context and Sector-Specific Evaluation: Agriculture, Water and Sanitation, Infrastructure

Multipaper Session 117 to be held in Sebastian Section L2 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Tessie Catsambas,  EnCompass LLC, tcatsambas@encompassworld.com

Discussant(s):

Tessie Catsambas,  EnCompass LLC, tcatsambas@encompassworld.com

The Impact of Water Supply and Sanitation Interventions on Child Health: Evidence From 2001-2006 DHS Surveys on Sub-Saharan Africa and South Asia Presenter(s): Ron Bose, International Initiative for Impact Evaluation, rbose@3ieimpact.org Abstract: I evaluate the impacts on child health, using the incidence of diarrhea as the health outcome, (amongst children living in households) with access to different types of water and sanitation facilities, and from socio-economic and child specific factors. Using cross-sectional health 2006 DHS survey data for Nepal, I employ statistical techniques to match children belonging to different "treatment" groups, defined by water types and sanitation facilities, with children in a "control" group. I also employ a variety of regression techniques to check for the robustness of my results, and demonstrate that it is important to incorporate the survey weights in both linear regression and matching estimation. The paper will provide guidance on ways in which matching can be used to estimate average cost-effectiveness of interventions and discuss some practical issues related to implementing the analysis using popular statistical software including STATA and R.

Rolling Baseline Survey Methodology for Household Income Impact Assessment in Micro-irrigation Induced Agricultural Development Presenter(s): Giel Ton, Wageningen UR, giel.ton@wur.nl Abstract: In countries with a history of multiple and spatial overlapping agricultural development projects, it is difficult, if not impossible, to find proper 'treatment' and 'non-treatment' groups for impact assessment, and derive the 'treatment effect' as a proxy for attributable household income impact. In self-selected customer groups of irrigation technology adopters, the selection bias makes income comparisons even more difficult. We present a methodology for assessing income impact of technology adoption within self selected populations used in Ethiopia, Zambia and Nepal by IDE. Household income is calculated by estimating the gross margins of farm and off-farm activities, before and after a induced change in the agricultural system. Pre-adoption household income in a specific year is compared with post-adoption incomes of another cohort of customers in the same year. The pre-adoption income of each yearly cohort functions as the 'rolling baseline' for assessing post-adoption income impact.

Estimating Impacts of Infrastructure Projects Presenter(s): Duncan Chaplin, Mathematica Policy Research Inc, dchaplin@mathematica-mpr.com Minki Chatterji, Mathematica Policy Research Inc, mchatterji@mathematica-mpr.com Denzel Hankinson, DH Infrastructure, denzelh@gmail.com Anne Rothbaum, Millennium Challenge Corporation, rothbaumae@mcc.gov Abstract: In this paper we investigate options for estimating impacts of electricity infrastructure projects funded by the Millennium Challenge Corporation in Tanzania. We consider a number of issues. These include developing research questions that are of interest to the stakeholders, designing a study that is feasible from an engineering standpoint, determining what services are likely to be received in the absence of the intervention, determining what population to try to generalize to, dealing with mobility of the study participants, dealing with the possibility of other interventions being implemetned simultaneously, figuring out what to do if benefits of the projects are widespread, and trying to determine who is most likely to benefit in a given area. We focus on the feasibility and usefulness of three possible research designs for dealing with these issues, Random Assignment, Regression Discontinuity, and Difference in Difference and find that, as expected, each method has advantages and disadvantages.

Sustainability Evaluation of the Rural Water Supply and Sanitation Schemes Presenter(s): Bimal Sharma, Rural Water Supply and Sanitation Fund Development Board, bimalsharma6@gmail.com Nirmala Upadhaya Rupakheti, Support for Technical and Allied National Development Services, bcsharma2000@yahoo.com Abstract: The sustainability study will be designed to evaluate the overall sustainability of the scheme based on four major dimensions i.e. Institutional, Social/Environment, Financial, and Technical comprising of a number of pre -determined indicators in each category in order to evaluate the strengths and weakness of the program after implementation of each batch and provide recommendations for future improvements. The study intends to assess the overall sustainability of the schemes constructed in order to examine the process, approaches, procedures, guidelines and manner in which RWSSFDB objectives and principles are achieved in delivering sustainable health and hygienic benefits to the rural population through improvements in water supply and sanitation.

Session Title: Symphonic Capability Curve

Demonstration Session 118 to be held in Sebastian Section L3 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

Presenter(s):

Alonford Robinson, Symphonic Strategies, ajrobinson@symphonicstrategies.com

Abstract: The Symphonic Capability Curve (SCC) is an organizational assessment instrument that measures an organization's collective capability in 12 key areas. Using more than 100 separate attributes of high performance, the self-assessment tool helps leaders plot their performance against an ideal, and, more importantly, against a benchmark of peer organizations. The power of the attributes we have chosen rests in their composition, they are all outcomes-based measures. We measure more than just potential to act. We measure the outcomes of organizational action.

Session Title: Statewide Evaluation Studies: Issues in Design, Implementation, Reporting and Policy

Multipaper Session 119 to be held in Sebastian Section L4 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Jordan Horowitz,  Cal-PASS, jhorowitz@calpass.org

Discussant(s):

Andrew Newman,  Mid-continent Research for Education and Learning, anewman@mcrel.org

An Economic Evaluation of Florida Class Size Reduction Policies Presenter(s): Jian Gao, Florida State University, jgao15@yahoo.com Abstract: In November 2002, the Florida Legislature set the maximum class size for core-curricula courses which takes effect in the beginning of the 2010-2011 school year. This policy has been widely debated for its costs compared to other options of school inputs. However, few studies have been done to evaluate the effectiveness and efficiency of this program. This study will use recently released longitudinal school data to examine the impact of the variation in student-teacher ratio in Florida schools has on student achievement. Furthermore, this study will compare the marginal cost of class size reduction with alternative school inputs such as teacher education, teacher experience, per student expenditures, and school size in promoting student achievement. Focusing on the school level variables that can be manipulated by school administrators, this study has important implications for educational policies based on the current economic uncertainty

Informing Multiple Stakeholders: A Participatory, Utilization-Focused Evaluation of State-wide Preschool Initiatives Presenter(s): Katie Dahlke, Learning Point Associates, katie.dahlke@learningpt.org Jason Butler, Learning Point Associates, jason.butler@learningpt.org Nancy Zajano, Learning Point Associates, nancy.zajano@learningpt.org Stephanie Siddens, Ohio Department of Education, stephanie.siddens@ode.state.oh.us Lisa Baker, Ohio Department of Education, lisa.baker@ode.state.oh.us Abstract: This paper presentation will describe the methodology used to implement a participatory, utilization-focused approach for assessing the quality of language and literacy instruction in preschool classrooms funded by three state-wide early childhood education programs. The paper will share lessons learned pertaining to our process of extending the utilization focus to both primary stakeholders (state education agency) and secondary stakeholders (classroom teachers, local administrators, and program administrators). In addition, the paper will describe the nature of the participatory relationship between the primary stakeholders and the evaluators as well as the circumstances and strategies used to cultivate that relationship. While this paper is focused primarily on methodology, it will also present preliminary findings regarding language and literacy instruction in preschool classrooms across the state.

Alignment of Alternate Assessments and State Decisions About System Improvement: A Multi-case Study of Four States Presenter(s): Meagan Karvonen, Western Carolina University, karvonen@email.wcu.edu Abstract: In the years since IDEA 1997 required that students with the most significant cognitive disabilities participate in large-scale assessment systems, states have been challenged to repeatedly revise their alternate assessment (AA) systems to focus on academic subjects and link to grade-level content expectations but with alternate achievement standards. This paper presents a multi-case study of four state departments of education that participated in a project to evaluate and improve alignment of their AA systems. Each state's AA system was subjected to alignment analysis, and priorities for improvement were established based on individual state needs and priorities. The cases comprise the majority part of the project's overall evaluation. Contextual influences within the states played significant roles in their interpretation and use of alignment study results. Findings have implications for understanding how agencies make decisions that influence validity of their large-scale assessment systems within a high-stakes context and with limited resources.

Session Title: Insights From Rapid Evaluations: Improving School Programs for Better Results

Multipaper Session 120 to be held in Suwannee 11 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Health Evaluation TIG

Chair(s):

Doryn Chervin, ICF Macro, doryn.d.chervin@macrointernational.com

Discussant(s):

Leah Robin, Centers for Disease Control and Prevention, ler7@cdc.gov

Abstract: The Division of Adolescent and School Health (DASH) of the Centers for Disease Control and Prevention (CDC) contracted with Macro International Inc. to conduct rapid evaluations of the effectiveness of innovative programs addressing child and adolescent health issues. Rapid evaluations use a mixed methods approach together with rapid, iterative, and team-based methods. DASH's use of the method has been particularly valuable for quickly gauging whether an initiative is effective and where program improvements are worthwhile. This session will describe one such program and the methods and measures used to evaluate it, how unexpected results were understood, and lessons learned for future rapid evaluations. Discussants will reflect on the project to date.

Value of the Rapid Evaluation Method

Leah Robin, Centers for Disease Control and Prevention, ler7@cdc.gov

Marian Huhman, University of Illinois at Urbana-Champaign, mhuhman@illinois.edu

Catherine Rasberry, Centers for Disease Control and Prevention, catherine.rasberry@cdc.hhs.gov

Rapid evaluations are designed to be completed within a relatively short time frame to provide information about the impact of programs, policies, and initiatives. In a rapid evaluation, appropriate stakeholders are engaged in the process to identify salient evaluation questions, develop and implement evaluation plans, and collect and analyze data. This method aims to describe program activities, short-term and intermediate outcomes, and impacts. Further, rapid evaluations can increase accountability by identifying why program components are not being implemented as planned. The value of rapid evaluations is the timely provision of data to encourage action. Results from rapid evaluations can also have implications for other organizations aiming to adopt and implement similar programs and can provide insights for program improvement and quality information for making decisions.

Conducting a Rapid Evaluation in a Local Education Agency in the Southeast

Karen Cheung, ICF Macro, karen.cheung@macrointernational.com

Pamela Lunca, ICF Macro, pamela.j.lunca@macrointernational.com

Sarah Merkle, Centers for Disease Control and Prevention, smerkle@cdc.gov

In 2006, CDC/DASH launched an initiative to provide local education agencies (LEAs) with evaluation technical assistance on asthma management programs using the rapid evaluation method. First, an evaluability assessment was conducted to determine the program's readiness and capacity for evaluation. Then, using participatory, team-based, and iterative processes, the authors assisted a large LEA in the Southeast to develop evaluation questions and an evaluation plan. Individual interviews, focus groups, and questionnaires were used to collect information from various stakeholders, including elementary school students and high school students; principals, nurses, and other program staff; key district staff members; and parents/guardians of elementary school students. The authors will describe how the rapid evaluation method provided the LEA with important feedback to strengthen the overall management of their asthma program by facilitating uniform implementation of program components across all sites and maintaining detailed records of asthma program services to allow for future evaluation activities.

Interpretation and Communication of Unexpected Findings of a School-Based Asthma Program in a Local Education Agency in the Southeast

Dana Keener, ICF Macro, dana.c.keener@macrointernational.com

Karen Cheung, ICF Macro, karen.cheung@macrointernational.com

Catherine Rasberry, Centers for Disease Control and Prevention, catherine.rasberry@cdc.hhs.gov

Just like any other approach to evaluation, rapid evaluations can reveal unexpected findings. This presentation will describe unanticipated results that emerged from a rapid evaluation of a school-based asthma program in a large LEA in the southeast. In addition, the presentation will describe efforts to explore possible confounding variables that might explain the findings; mixed-method techniques used to understand and interpret the findings; and the process for communicating the findings back to the school district. Finally, recommendations for program improvement and future evaluation that stemmed from the unexpected findings will be shared.

Insights From Conducting Rapid Evaluation in School Settings

Doryn Chervin, ICF Macro, doryn.d.chervin@macrointernational.com

Dana Keener, ICF Macro, dana.c.keener@macrointernational.com

Karen Cheung, ICF Macro, karen.cheung@macrointernational.com

Leah Robin, Centers for Disease Control and Prevention, ler7@cdc.gov

This presentation will share key lessons and insights gained from conducting rapid evaluations in school settings. Although some lessons may be specific to school settings, others apply across other settings as well. Some of the lessons that will be discussed include: (1) form strong working relationships with evaluation stakeholders; (2) identify and address gaps in implementation data early in the process; (3) develop formal and ongoing opportunities for sharing evaluation results as they become available; and (4) engage stakeholders in the interpretation of the results. In addition, key questions that emerged from the evaluations will be raised for group discussion. For example, would rapid evaluations benefit from a more robust evaluability assessment prior to the evaluation? Finally, tensions associated with making mid-course corrections and improvements while the evaluation is still ongoing will also be discussed.

Session Title: Assessing Special Need Populations in the Context of High-Stakes Testing

Multipaper Session 121 to be held in Suwannee 12 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Special Needs Populations TIG

Chair(s):

Jane Beese,  University of Akron, jab128@uakron.edu

An Evaluation of High Stakes Tests in Respect to Special Needs Populations Presenter(s): Xin Liang, University of Akron, liang@uakron.edu Jane Beese, University of Akron, jab128@uakron.edu Abstract: Including students with disabilities under the regulations of NCLB has been controversial since the law was approved. According to Darling-Hammond (2007), approximately one-third of public schools have been classified as failing to meet AYP with projections reaching as high as 80% of schools failing to achieve AYP by 2014, primarily due to subgroup targets that must be reached. NWEA MAP is a state-aligned computerized adaptive test that reflect the instructional level of each student that measures growth over time. The purpose of this paper is to present the experience of schools in a Midwestern state using NWEA MAP in comparison with State achievement test assessing academic performance for students with special needs. It is our hope that this paper will generate discussions about using alternative assessment tools and the need for growth information in school accountability related to special needs populations.

Using a Multifaceted Approach to Investigate Innovative Assessments for Students With Special Needs Presenter(s): Vasanthi Rao, University of South Carolina, vasanthiji@yahoo.com Grant Morgan, University of South Carolina, praxisgm@aol.com John Payne, South Carolina Department of Education, jrpayne@ed.sc.gov Ching Ching Yap, Savannah College of Art and Design, ccyap@mailbox.sc.edu Abstract: Recent changes in No Child Left Behind legislation have provided states the flexibility of investigating alternate innovative assessments based on modified achievement standards for certain students with special needs. To gain a comprehensive understanding of the appropriate modification strategies, South Carolina Department of Education, a recipient of two federal (Enhanced Assessment and General Supervision) grants, adopted a multifaceted approach that involve experts such as psychometricians, special education researchers, education and assessment specialists, and education evaluators. These experts collaborated to provide information about the (a) effectiveness of modification strategies, (b) item functioning of students with and without special needs, (c) identification strategies for qualified students for the modified achievement assessment, and (d) current strategies used by educators with qualified students. By shedding light on each facet of the issue, the result is the provision of comprehensive information that can be used to inform alternate assessment stakeholders.

Session Title: Assessing the Impact of Three Dropout Prevention Strategies on Student Academic Achievement in Grades 6-12 in Texas

Multipaper Session 122 to be held in Suwannee 13 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Thomas Horwood, ICF International, thorwood@icfi.com

Discussant(s):

John Kuscera, Texas Education Agency, john.kucsera@tea.state.tx.us

Abstract: The Texas Legislature authorized and funded three pilot programs so that select districts and charter schools receiving grants could develop and implement programs to prevent and reduce dropout, increase high school success, and improve college and work readiness in public schools. In addition, the Texas Legislature authorized and funded the evaluation of these pilot programs and required that the evaluation assess the impact of the programs on student performance, high school completion rates, college readiness of high school students, teacher effectiveness in instruction, as well as the cost-effectiveness of each program. The objectives of the evaluation are to do each of the following for each pilot program: (1) evaluate the implementation of each program, (2) evaluate the impact of each program on student outcomes (e.g., achievement, college readiness, workforce readiness, graduation), and (3) evaluate the impact of each program on other relevant outcomes (e.g., teacher effectiveness).

An Evaluation of the Intensive Summer Program (ISP) in Texas Schools

Rosemarie O'Conner, ICF International, ro'conner@icfi.com

John Kuscera, Texas Education Agency, john.kucsera@tea.state.tx.us

Carol Kozak Hawk, ICF International, carolkozakhawk@gmail.com

The Intensive Summer Program (ISP) provides summer instruction for "at risk" students in Texas with the goals of reducing dropout, increasing school achievement, and promoting college and workforce readiness skills among students. Using a mixed methods approach, this presentation examines statistical analyses of student achievement data and the survey results from school administrators, teachers, and students. Hierarchical linear models (HLM) are used to examine the results of student achievement in standardized tests to determine whether the ISP program positively affected student academic achievement over time. Student surveys are used to further understand the relationships uncovered in statistical analyses, while surveys from the school administrators and teachers shed light on the additional positive benefits from the ISP program. Concluding remarks focus on the evaluation of the ISP program and identify future directions and lesson learned from this evaluation.

An Evaluation of the Collaborative Dropout Reduction Program in Texas Schools

Frances Burden, ICF International, fburden@icfi.com

John Kuscera, Texas Education Agency, john.kuscera@tea.state.tx.us

Sarah Decker, ICF International, sdecjer@icfi.com

The Collaborative Dropout Reduction program is a school-based program aimed at promoting academic achievement and college and workforce readiness in students. Considerable differences exist between the six Collaborative programs in their approaches to developing students' academic and workforce readiness skills. This evaluation examines student achievement across multiple campuses using HLM models, with particular attention to the larger programmatic differences. Additional analyses focus on student self-reported assessments of ethical workplace behaviors and their own college and workforce readiness, which serve to offer greater insight into the statistical results. Finally, interviews with the Collaborative programmatic staff and school administrators uncover additional school-wide positive benefits from the Collaborative program. Concluding remarks will focus on the difficulties encountered in evaluating six diverse Collaborative programs and how commonalities and differences were uncovered and measured.

An Evaluation of the Impact of Teacher Mathematics Instructional Coaches Training on Teachers and Schools

Amy Mack, ICF International, amack@icfi.com

John Kuscera, Texas Education Agency, john.kucsera@tea.state.tx.us

The Mathematics Instructional Coaches (MIC) pilot program provides assistance in developing the content knowledge and instructional expertise of teachers who instruct "at risk" students in mathematics at middle and high schools. The evaluation of the MIC program examines whether strengthening mathematics teachers' knowledge, skills, and abilities led to improvements in teachers' self-efficacy and beliefs about teaching. Surveys and interviews were collected from stakeholders in the MIC program, including administrators and grant coordinators, in order to further understand the results from teacher self-reported measures. Finally, this presentation will conclude with a discussion of the challenges the MIC program faced in developing student-level and school-level findings from programs aimed at training teachers.

Session Title: Measuring the Fidelity of Implementation of Response to Intervention for Behavior (Positive Behavior Support) Across All Three Tiers of Support

Demonstration Session 123 to be held in Suwannee 14 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Karen Childs, University of South Florida, childs@fmhi.usf.edu

Don Kincaid, University of South Florida, kincaid@fmhi.usf.edu

Heather George, University of South Florida, hgeorge@fmhi.usf.edu

Abstract: This workshop will provide information on the development, validation and use of implementation fidelity instruments available for use by schools in evaluating school-wide positive behavior support (otherwise known as response to intervention for behavior). The Benchmarks of Quality (BoQ) is a research-validated self-evaluation tool for evaluating fidelity of school level implementation of Tier 1/Universal level behavior support. Participants will receive information on the theoretical background, development and validation of the BoQ for Tier 1. Participants will learn how to complete the instrument, how the instrument is used by school, district and state teams to monitor implementation fidelity, and how to use results to improve implementation. Participants will also receive information about a new instrument in development; the Benchmarks of Quality for Tier 2/Supplemental and 3/Intensive levels of support. This discussion will include an explanation of instrument development and results of the validation pilot study.

Session Title: Walking the Tightrope: Strategies for Conducting Evaluations Within the Political Contexts of School Districts

Panel Session 124 to be held in Suwannee 15 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Jennifer Coyne Cassata, Prince William County Schools, cassatjc@pwcs.edu

Abstract: Decision-making by local school boards takes place within an intensely political context. Stakeholders, particularly parents, have a direct voice and many groups are mobilizing to actively shape policy decisions. Evaluators working within and for school districts find themselves in a situation where frequently changing political contexts influence the methods used, the extent to which evaluation findings are utilized, and even evaluators' ability to adhere to the Program Evaluation Standards. This panel discussion will include evaluators working within neighboring school districts who will share sample experiences and how they navigated through those experiences to maintain high-quality practice and encourage effective use of the evaluation process and findings.

Do You Hear What I Hear? Attempting to Enhance Utilization of Evaluation Findings Jennifer Coyne Cassata, Prince William County Schools, cassatjc@pwcs.edu Kenneth Hinson, Prince William County Schools, hinsonke@pwcs.edu Two examples will be used to describe the context many evaluators face and strategies for navigating that context to encourage utilization of evaluation findings. The first is one in which evaluators were asked to conduct an evaluation of an elementary mathematics program and wound up in the middle of the math wars. Evaluators have faced issues related to Propriety Standards, in that both sides would like evaluators to take a side, as well as to Accuracy Standards, where evaluators have had to defend the quality and objectivity of the work. In addition, evaluators have had to present findings alongside conflicting anecdotal evidence presented by stakeholder groups. The second is one in which evaluators were asked to determine the interest in a potential program. In this situation, the major issue encountered was the inclination to absorb only those findings that supported previously held positions.

Strategies For Maintaining Objectivity in Political Contexts Anane Olatunji, Fairfax County Public Schools, anane.olatunji@fcps.edu A case-study will be presented and discussed to underscore the challenges evaluators may face when seeking to maintain AEA's guiding principles (i.e., Systematic Inquiry, Competence, Integrity/Honesty, Respect for People, Responsibilities for General and Public Welfare). The presentation will discuss specific strategies evaluators may use to maintain the integrity of the studies they conduct in politically-charged contexts.

Political Contexts, Methodological Constraints, and Program Outcomes Anane Olatunji, Fairfax County Public Schools, anane.olatunji@fcps.edu Kenneth Hinson, Prince William County Schools, hinsonke@pwcs.edu The political contexts of public schools often preclude conducting randomized experiments. Consequently, evaluators must often assess impacts after programs already have been designed and implemented. Confronting this limitation, the evaluators employed a quasi-experimental design using matched subjects in a district-wide academic enrichment program that exhibited intriguing results. This presentation examines matching strategies that investigators may use in order to improve the validity of evaluation results when it is threatened by political circumstances.

Session Title: The Influence of Evaluators' Principles and Clients' Values on Contextually-Bound Evaluation Resource Decisions

Panel Session 125 to be held in Suwannee 16 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Research on Evaluation TIG

Chair(s):

Jennifer Greene, University of Illinois at Urbana-Champaign, jcgreene@uiuc.edu

Discussant(s):

A Rae Clementz, University of Illinois at Urbana-Champaign, clementz@illinois.edu

Abstract: As evaluators, we hold certain principles (enacted values) that guide our practice, as do the clients we serve. This panel will discuss how those principles influence our work, especially in the decisions we must make given resources available to a particular context. The extent to which certain principles and resources are negotiable or non-negotiable across evaluation or evaluation capacity building settings will also be explored. Particular attention will be paid to the enacted values guiding the relationships between the evaluator and the client, who are key resources in determining the direction of the evaluative effort and ensuring its successful implementation and utility.

Theoretical Implications for Research on Evaluation Marvin Alkin, University of California Los Angeles, alkin@gseis.ucla.edu Alkin will present a synopsis of key theoretical approaches relevant to the panel's discussion and discuss the implications of research on evaluator's principles and clients' values for the field of evaluation.

Influence of Evaluators' Principles on Resource Decisions Within and Across Contexts Kara Crohn, University of California Los Angeles, kara.crohn@ucla.edu Crohn will present dissertation research findings from three intensive qualitative case studies of work by Jennifer Greene, Gary Henry and Jean King which highlight ways in which evaluators' principles influence resource decisions. This study expands on "Theorists' Models in Action," New Directions for Evaluation, no. 106 (Alkin & Christie, Eds.). For each participant, relationships among resources and the evaluator's principles were first examined in the case used for "Theorists' Models in Action" and compared with analyses of real evaluations that best exemplified the evaluator's theory in practice. Examples of key relationships and their implications for practice will be described with particular emphasis on principles concerning contextuality, the ways in which resource-principle relationships are enacted in a given context, and similarities and differences across contexts.

Reflections on the Principles That Guide My Work and the Contextuality of Evaluation Resource Decisions Jean A King, University of Minnesota, kingx004@umn.edu As a participant in Crohn's research, King will offer her perspective on the findings presented. She will reflect on the extent to which various factors influence her work, including her principles, her relationships with others on the evaluation team and in the evaluation setting, limitations and opportunities presented in the context, etc.

Session Title: Informal Environments: A Sampler of Audience Research and Evaluation From the Visitor Studies Association

Multipaper Session 126 to be held in Suwannee 17 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Evaluating the Arts and Culture TIG

Chair(s):

Cheryl Kessler, Institute for Learning Innovation, kessler@ilinet.org

Discussant(s):

Joe E Heimlich, Ohio State University, heimlich.1@osu.edu

Abstract: What does evaluation and research in informal learning environments look like? Who does this work? What are the challenges and constraints for evaluating informal learning environments? Expanding the discussion in New Directions for Evaluation's 2005 issue, Evaluating Nonformal Education Programs and Settings, members of the Visitor Studies Association (VSA), an international network of professionals committed to understanding and enhancing visitor experiences in informal settings through research, evaluation and dialogue, will present a showcase of studies in informal environments. The showcase will include evaluation and research studies conducted for a performing arts organization, studies conducted in history, science, natural history, and children's museums; studies aimed at specific and general audience learning; and studies conducted by both internal and external evaluators.

Approaches to Measuring Identity in Informal Learning Environments

Kirsten Ellenbogen, Science Museum of Minnesota, kellenbogen@smm.org

Kirsten Ellenbogen, Director of Evaluation and Research in Learning at the Science Museum of Minnesota will present on recent work in informal learning environments to define identity, measure identity, and the integration of technology into identity-related measurements. Discussion includes criteria for developing an identity as a learner, the interrelationship of identity and participation, and the particular importance of identity in informal learning environments. Approaches include video-based reflective interviews, traditional and technology-supported journaling, and conversation analysis. Ellenbogen is a founding officer of the Informal Learning Environments Research SIG-American Education Research Association, senior chair of the Informal Science Education Strand-National Association for Research in Science Teaching, training coordinator of the Visitor Studies Group (UK), and President of VSA.

Concurrent Evaluations in a Single Institution

Saul Rockman, Rockman et al, saul@rockman.com

Saul Rockman, President of Rockman et al, and VSA Board member, will present on evaluations conducted in Science Centers and Natural History museums. Within a single institution, visitors include school groups, seniors, young adults, and multi-generation families; programs are designed to appeal to any and all of these groups. Concurrent evaluations in one institution include: assessing the perceived value of a multimedia exhibit for adults and families, the educational value of school group visits and programs for teachers, studies of learning from a planetarium show and visitor interest in future programs, how partnerships between public institutions and the private sector yield benefits for both, and the contributions of evening programs and Web 2.0 presence for young adults to increase membership and attendance. Each evaluations engages multiple methods and strategies, varying immediate and longitudinal impacts, and a concern for data interpretation to provide actionable information for institutional and programmatic decision makers.

Evaluating Museum Youth Program for Social Change and Civic Engagement

Mary Ellen Munley, MEM & Associates, maryellen@mem-and-associates.com

Mary Ellen Munley, Principal, MEM & Associates and Past President of VSA, will present on a study with program staff of the US Holocaust Memorial Museum to evaluate Bringing the Lessons Home, the museum's youth program for area teens. The evaluation focused on the ways in which the attitudes and actions of participating youth changed over the course of their involvement in the program, how those changes contributed to personal transformation and increased civic engagement, and how the participants' became catalysts for social change in their communities and in the museum. As they continue to work on moving the program into its next phase of development, program staff are actively using the study to gain a deeper understanding of the relationship between the anatomy of the program design, outcomes for participants, and broader impact.

Evaluation at the Denver Museum of Nature and Science (DMNS)

Kathleen Tinworth, Denver Museum of Nature and Science, kathleen.tinworth@dmns.org

Kathleen Tinworth, Director of Vistor Rsearch & Program Evaluation at the Denver Museum of Nature & Science (DMNS) and member of VSA's Profesional Development Committee, will present on evaluation conducted for a unique and distinctive first-person enactment program in 2007. Two multi-method evaluations (including exit surveys, visitor and enactor interviews and focus groups, observations and tracking and timing studies) have been conducted to assess qualitative and quantitative impacts that the enactor program has on visitor experience at DMNS. The first evaluation focused on the enactors' role within a temporary exhibition (Titanic: The Artifact Exhibition) and the second examined their ongoing work in the Museum's permanent diorama halls.

Evaluating the Long Island Children's Museum's, Be Together, Learn Together Program

Cheryl Kessler, Institute for Learning Innovation, kessler@ilinet.org

Cheryl Kessler, Research Associate with the Institute for Learning Innovation and VSA Board member will present on evaluation conducted for the Long Island Children's Museum (LICM), Be Together, Learn Together program, a partnership with Nassau County, NY Department of Health & Human Services (DHHS) to provide support to children and families receiving social service agencies.

Session Title: Hidden Possibilities: Building Stakeholder Capacity To Utilize Demographic Data

Demonstration Session 127 to be held in Suwannee 18 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

Paul St Roseman, Sakhu and Associates LLC, pstroseman@gmail.com

Abstract: This demonstration presents evaluation service delivery approaches used to develop and support the capacity of organizational stakeholders to compile, manage, and utilize demographic data to: (a) clarify emerging service models or (b) improve understanding of models that already exist but have never been documented. Through the review of three case examples, participants will examine: (1) essential evaluation frameworks/models used to inform service delivery approaches to work with program managers and their staff members to identify demographic data, (2) tools and techniques used to build stakeholder knowledge and capacity to collect and manage demographic data, and (3) approaches used to guide stakeholder effort to use demographic data for fundraising, program planning/development, and accountability reporting. This presentation is most appropriate for evaluation practitioners who collaborate with education or human service program managers and staff members to design, implement, and sustain evaluation processes, as well as utilize evaluation products.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Sharing Lessons Learned In Implementing State Outcomes Systems for Alcohol, Drug Abuse, and Mental Health Services

Roundtable Presentation 128 to be held in Suwannee 19 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG

Presenter(s):

Diana Seybolt, University of Maryland Baltimore, dseybolt@psych.umaryland.edu

Margaret Cawley, National Development and Research Institutes Inc, cawley@ndri-nc.org

Abstract: Each of the presenters has been involved in the development and use of a state outcomes system (Maryland Outcomes Measurement System [OMS] and North Carolina's Treatment Outcomes and Program Performance System [NC-TOPPS] respectively). Each will provide a brief presentation regarding their system and contextual influences which have affected system development and use. Group discussion will provide an opportunity for attendees to discuss their experiences, learn from one another, and brainstorm regarding challenges. Questions to be addressed include 'What contextual influences have affected your work in outcomes measurement?; What roles do government and other stakeholders play? How have you addressed competing priorities?; What has aided or impeded the development or ongoing use of your system?; How are you analyzing and presenting data?; Who has access to data?; What are you doing to promote use of the data for quality assurance purposes?; and 'What are your 'lessons learned'?'

Roundtable Rotation II: Working With Consumers of Mental Health Services in Evaluations: Thinking About the Challenges and Opportunities

Roundtable Presentation 128 to be held in Suwannee 19 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG

Presenter(s):

Amanda Jones, University of Maryland Baltimore, amjones@psych.umaryland.edu

Jennifer Kulik, University of Maryland Baltimore, jkulik@psych.umaryland.edu

Clarissa Netter, Maryland Mental Hygiene Administration, netterc@dhmh.state.md.us

Abstract: Evaluations in a public mental health system are improved when evaluators collaborate with all of the system's stakeholders, including the consumers of the public mental health system's services. The presenters, two University of Maryland evaluators and a consumer liaison from Maryland's Mental Hygiene Administration, have learned that consumer participation in evaluations presents both challenges and opportunities. After presenters speak briefly about their experiences, roundtable participants will discuss a number of topics related to working with consumers in the context of an evaluation, including the roles consumers can take in evaluations (as evaluation partners and/or evaluation participants); the most effective techniques and methods for empowering consumers in their roles during each phase of an evaluation (from evaluation planning to results dissemination); and the contextual elements (e.g., evaluation settings and societal influences) that can affect the evaluation experience for consumers.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Seven Effective Strategies for Assessing and Demonstrating Impact of Faculty Development Programs: From Experience Comes Wisdom

Roundtable Presentation 129 to be held in Suwannee 20 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Assessment in Higher Education TIG

Presenter(s):

James Eison, University of South Florida, jeison@coedu.usf.edu

Yenni Djajalaksana, University of South Florida, ydjajala@mail.usf.edu

Jecky Misieng, University of South Florida, jmisieng@mail.usf.edu

Abstract: Experienced faculty developers who are leaders in the field of faculty development in higher education settings were contacted via an anonymous web-based survey to collect valuable and practical lessons learned from their vast experience. One of the questions was about assessing and demonstrating impact of their development programs. 42 participants responded to the survey and this presentation will highlight insights gained by grouping them into seven broad categories. Among the issues brought up to assess impact was keeping track of how many faculty attend training programs and what they say about the usefulness of the topics. To demonstrate impact, one advice was to document everything in a detailed annual report and 'planting' stories in the local campus publications to maintain visibility.

Roundtable Rotation II: Promoting Change Through Internal and External Evaluations: Academic Center for Excellence

Roundtable Presentation 129 to be held in Suwannee 20 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Assessment in Higher Education TIG

Presenter(s):

Bonnie Smith, Chipola College, smithb@chipola.edu

Dan Kaczynski, Central Michigan University, dan.kaczynski@cmich.edu

Abstract: This presentation will critically discuss internal and external evaluation methods and results covering the first two years of a multi-year federal higher education Title III grant award. Two grant components will be highlighted; the Academic Center for Excellence (ACE) and Supplemental Instruction (SI). Of particular interest in this presentation will be the unique challenges the college has experienced due to the success and growth in student interest ACE has generated. Formative findings indicate success in ACE and SI is due to administrative support and buy in from faculty in promoting a supportive culture for organizational change.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Evaluation as a Management and Learning Tool for Neighborhood Change

Roundtable Presentation 130 to be held in Suwannee 21 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Presenter(s):

Della M Hughes, Brandeis University, dhughes@brandeis.edu

Abstract: This presentation will focus on methods and tools used by The Skillman Foundation evaluation team to assess individual, nonprofit and community capacities during the Readiness Phase of the Detroit Works for Kids Initiative (a ten-year investment in six neighborhoods in Detroit to improve outcomes for young people). Further, we will describe the challenges and opportunities of (1) embedding evaluation in the community for continuous improvement when there are competing voices and interests, and the neighborhoods are in the formative stages of their governance development processes; (2) engaging residents and other stakeholders in defining the capacities and the short and interim indicators for the long-term youth outcomes; and (3) measuring the strength of a system of supports and opportunities for youth when clear organizational, leadership and system baselines were not established at the onset of the initiative. The discussion will also focus on development of effective capacity for learning and data-driven decision making at the neighborhood and cross-neighborhood levels.

Roundtable Rotation II: Assessing Community Capacity to Develop and Implement Powerful Strategies: Tools, Methodology and the Influence of Evaluation on Practice

Roundtable Presentation 130 to be held in Suwannee 21 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Presenter(s):

Mary Achatz, Westat, maryachatz@westat.com

Scott Hebert, Sustained Impact, shebert@sustainedimpact.com

Abstract: A presentation of the tools and methodology that The Annie E. Casey Foundation developed and is using to assess community capacity to achieve tangible results for significant numbers of families and children in 10 communities nation-wide. The tool includes indicators of community capacity along a developmental continuum to mark progress over time, and to identify next steps. The methodology, which begins with a facilitated conversation with key stakeholders, uses a common set of questions to guide conversations across sites and to elicit concrete examples and evidence that support their assessments and that link investments in community capacity to improved outcomes for families and children. Discussion will include key decisions and processes in the development and refinement of the tool and methodology, the contributions of the assessment to continuous local learning and ongoing capacity building, and the challenges of analysis across the sites.

Session Title: Evaluating Technology Training and Development Initiatives

Multipaper Session 131 to be held in Wekiwa 3 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Distance Ed. & Other Educational Technologies TIG

Chair(s):

Jane A Rodd,  University at Albany - State University of New York, eval@csc.albany.edu

The Impact of Multi-Media on Learning and Instruction Serving a Community of Users With Multiple Needs Presenter(s): Jane A Rodd, University at Albany - State University of New York, eval@csc.albany.edu Dianna L Newman, University at Albany - State University of New York, eval@csc.albany.edu Abstract: This paper discussed a four-phase model for evaluating multi-media learning materials that emphasizes the diversity of learners and variations in instructional needs and user characteristics. The authors began with an overview of the evaluation model, supporting evidence for its use, and key characteristics pertaining to use and implemented throughout the phases. The merit and worth of a program using student-centered technology in the domain of engineering was evaluated. Multiple data sources were used across sub-studies conducted to document real-time usability of the technology. Findings of the evaluation supported the use of technology-supported instruction for all types of learners, and identified technology that requires students to interact, develop a cognitive framework and share that knowledge with others, as appearing especially useful. Documentation of the program at each phase of the model is discussed in relation to the innovative approach to meeting the needs of diverse learners and facilitating varied instructional strategies.

Promising Practices in Information Technology Non-formal Education for Girls Presenter(s): Vicky Ragan, Puget Sound Center for Teaching, Learning and Technology, vragan@psctlt.org Carrie Liston, Puget Sound Center for Teaching, Learning and Technology, cliston@psctlt.org James Riedel, Girl Scouts of the USA, jriedel@girlscouts.org Ruthe Farmer, University of Colorado at Boulder, ruthe.farmer@colorado.edu Abstract: Women are underrepresented in Information Technology, and many programs aim to increase girls' interest to address this disparity. Representatives from nearly 200 IT education programs across the USA serving girls in grades 6-12 were surveyed to identify practices effective in engaging girls in IT and to describe their evidence. Practice categories were staff, curriculum, learning experiences, and career information. A second survey queried 937 women working in IT about their education experiences and influences that led to IT careers. Those who participated in an IT-related program during their youth were significantly more likely to have positive attitudes toward the field and feelings of IT competence during middle and high school. Their experiences and beliefs on what is critical to an educational program's success corresponded with the promising practices identified by the program representatives. A publication of promising practices in IT programs for youth will be shared.

Session Title: In Living Color: Black Women in Evaluation, Teaching and Praxis

Panel Session 132 to be held in Wekiwa 4 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Feminist Issues in Evaluation TIG

Chair(s):

Tristi Nichols, Manitou Inc, tnichols@manitouinc.com

Discussant(s):

Anna Madison, University of Massachusetts Boston, anna.madison@umb.edu

Abstract: The panel includes four presentations highlighting the need for understanding the role of context and evaluation practice in academic and other professional settings. Each presentation focuses on expanding evaluation in educational and professional settings. Re-defining evaluative practices within traditional academic settings will be discussed in The Intersection of Race and Gender in the Ivory Tower: Evaluating Academic Life for Women of Color. Incorporating evaluation practices in asynchronous academic settings will be discussed in Evaluation in Asynchronous Learning Environments: Examining the Experiences of African American Women. Evaluation Practice in Schools of Social Work: Is It Deemed a Scholarly Activity provides a discussion on intersectionality of evaluation practice as a scholarly activity and its impact on African American women faculty. Finally, Black Women at the Evaluation Cross provides a discussion regarding the contribution of Black women to the evaluation profession. The influence of race/gender in academic/evaluation settings will also be discussed.

The Intersection of Race and Gender in the Ivory Tower: Evaluating Academic Life for Women of Color Kimberly Farris, United States Army Center for Health Promotion and Preventive Medicine, kimberlydfarris@gmail.com Historically, women have faced multi-systemic challenges, such as discrimination and inequality in academic departments/universities, with regards to the structure of the tenure and promotion process. For women of color in academic settings, the intersection of race and gender compounds the ability to successfully negotiate academic life in the "Ivory Tower". The influence of culture and diversity among faculty are two important concepts within the social work profession, yet it remains a missing piece in the actual presence of the faculty members in numerous schools of social work. The initiation of a dialogue is needed on the following issues: exploring of the need of context-sensitive evaluation as it relates to the tenure and promotion process; understanding how race and gender may positively and/or negatively influence the process; and, successful development of methods that will produce culturally and gender sensitive approaches, in turn, leading to successfully implementation of policy and practices.

Evaluation in Asynchronous Learning Environments: Examining the Experiences of Black Women JaMuir Robinson, Walden University, jamuir.robinson@waldenu.edu Recently, the number of online colleges/universities has increased. This setting, where asynchronous learning occurs, is thought to provide more flexibility for students and instructors. Despite this growth, little literature exists regarding experiences of online faculty members and particularly minority faculty members. Minority professors represent approximately 12% of full professorships, with a smaller number representing Black women. The percentage of Black female faculty members at online institutions is difficult to assess. Little is known about their experiences in online environments where gender/race is not as easily identified, but come into play. Understanding how race/gender play out in the experiences of black female faculty is needed. This presentation will examine and discuss available data on the experiences of this group in online settings, the need for research on experiences with regard to professional growth, departmental support, and performance evaluation given that many online institutions do not utilize traditional tenure and promotion processes.

Evaluation Practice in Schools of Social Work: Is It Deemed a Scholarly Activity Jenny Jones, Virginia Commonwealth University, jljones2@vcu.edu Evaluation practice in the academy has increased significantly in recent years, particularly in schools of social work. This increase is due partly to a renewed focus on community engagement, suggesting an attempt to bridge the gap between the community and academy. Even with this increase, many research-one social work schools have not embraced evaluation as a scholarly activity. Lack of support for evaluation practice oftentimes creates obstacles for Black women faculty committed to improving lives of people of color. Additionally, this group tends to be marginalized by white colleagues as not being "true scholars" because of engagement in evaluation practice as a science. Impact of marginalization potentially affects productivity and erodes professional self-esteem. This presentation explores the intersectionality of evaluation practice as a scholarly activity, the impact of lack of acknowledgement on this group, and how race/gender is factored as it relates to social justice, research support, promotion and tenure.

Black Women at the Evaluation Cross Sarita Davis, Georgia State University, saritadavis@gsu.edu Black women have been highly visible in positions of leadership in AEA (i.e., serving on the board, committees, and Topical Interest Groups (TIG's,). However, according to the 2008 AEA Internal Scan, a majority of the female members are white (67%) and people of color are male (8%). So, if the women are white and the people of color are male, why are Black women so active in AEA? This presentation looks at the visible and hidden places of leadership, research, and praxis of black women in AEA. The goal of this presentation is to understand the contribution of Black women to the profession of evaluation and identify strategies for expanding the pipeline.

Session Title: Engaging Participants in the Evaluation Process: A Participatory Approach

Multipaper Session 133 to be held in Wekiwa 5 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Chair(s):

Corina Owens,  University of South Florida, cowens@coedu.usf.edu

Participatory Evaluation and its Impact on College Course Culture and Student Attitudes Toward Providing Feedback Presenter(s): Nancy Rogers, University of Cincinnati, nancy.rogers@uc.edu Janice Noga, Pathfinder Evaluation and Consulting, jan.noga@stanfordalumni.org Abstract: Institutions of higher education rely heavily on course evaluations to assess the quality of their faculty and the effectiveness of their courses in terms of student satisfaction and learning. Rather than use student feedback to improve courses, course evaluations are a summative assessments used primarily as a tool for considering issues of reappointment, promotion, and tenure. As a result, students are skeptical that course evaluations are valued by instructors for improving course content or teaching quality and consequently have little interest in providing meaningful suggestions for improvement. A formative, participatory evaluation approach to course evaluation provides an opportunity to empower students to be active creators of their academic experiences resulting in meaningful course evaluations for both students and teachers. This presentation will address how participatory evaluation using an action research cycle can contribute to changes in professor-student interactions and increase active engagement of students as co-creators of their college courses.

Assessing Vital Signs: Participatory Evaluation of a Nursing College's Programs and Improvements Presenter(s): Susan Connors, University of Colorado Denver, susan.connors@ucdenver.edu Kathy Magilvy, University of Colorado Denver, kathy.magilvy@ucdenver.edu Abstract: A participatory evaluation jointly conducted by faculty and administrators from the University of Colorado Denver College of Nursing and external evaluators from The Evaluation Center is described. Factors contributing to this successful stakeholder/evaluator partnership are examined including discussion of the higher education context, especially as related to current efforts to track the effectiveness of program improvements in the areas of student advisement, distance learning, and academic support for increasingly diverse student populations. Evaluator behaviors will be examined in relation to the building of trust and delineating shared responsibility with a faculty-governed evaluation committee. Congruence of the ongoing evaluation with a theoretical model of practical participatory evaluation (Smits & Champagne, 2008) will also be presented.

Using Participatory Methods to Develop an Evaluation Framework: State-and Federal-Level Examples Presenter(s): Watson Scott Swail, Educational Policy Institute, wswail@educationalpolicy.org Patricia Moore Shaffer, Educational Policy Institute, pshaffer@educationalpolicy.org Abstract: Participatory evaluation provides for the active involvement of stakeholders in the evaluation process, including program providers, partners, and recipients. During this presentation, we examine how participatory methods were applied to the planning and design of evaluation frameworks for two large-scale educational initiatives at the state and federal levels. Using methods including a stakeholder advisory group, interviews, and collaborative logic model development, the evaluation team facilitated a process through which program stakeholders played an active role in developing these evaluation frameworks.

What About Context and Theory? What's Practical? Lessons Learned From Choosing and Using Participatory-Action-Research and Participatory Evaluation Presenter(s): Becky Melzer, Evaluation Edge LLC, becky@evaluationedge.com Gwen Martin, The Center for Women's Business Research, gwen.martin1@gmail.com Abstract: In 2006, the Center for Women's Business Research's began a national participatory-action-research (PAR) initiative focused on business growth and public policy: Accelerating the Growth of Businesses Owned by Women of Color (WOC). The research addressed the fact that firms owned by WOC are growing in number yet substantial disparities exist in revenues and number of employees. A series of regional research and educational forums were held across the US. As both a research and programmatic endeavor, a participatory evaluation (PE) was conducted. PAR and PE proved an expensive endeavor for this initiative but provided valuable insights. The presenters will discuss the following questions: What dimensions of context would influence your choice to use PAR and PE? What are the theoretical and practical challenges and benefits? Furthermore, the presenters will discuss the challenge of determining what are research findings and what are evaluation findings.

Hearing Our Voices: A Participatory Approach to Engaging Youth in the Development and Evaluation of a Youth Radio Program Presenter(s): Kristi Lekies, The Ohio State University, lekies.1@osu.edu Abstract: This case study focuses on the evaluation of a new youth radio program during its first year of implementation. The program was created by a university professor and produced by high school and university students. Both the professor/evaluator and students needed to learn new roles and approaches. The professor/evaluator, who traditionally viewed youth as evaluation and research subjects, needed to transition to a new perspective of youth as partners. The youth needed to learn new responsibilities and develop greater ownership of the program. Successful strategies included a review of case studies of youth participatory evaluation, consultation with others who involve youth, the use of peer facilitators, field trip evaluation meetings at the university, discussions with media consultants, and engaging youth in a study of youth radio programs across the country. Obstacles and difficulties will be discussed, along with ways case studies can help improve theory and practice on participatory practices.

Session Title: Measuring Interdisciplinary and Mapping Research Emphases From Research Publications

Demonstration Session 134 to be held in Wekiwa 6 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Research, Technology, and Development Evaluation TIG

Presenter(s):

Alan Porter, Georgia Institute of Technology, alan.porter@isye.gatech.edu

Abstract: Research evaluation frequently addresses research outputs - e.g., papers deriving from programmatic funding, or comparison of multiple research centers. Over the past several years we have developed two useful empirical tools: * Integration and Specialization scores to characterize the degree of interdisciplinary * Science overlay maps to depict which research fields are engaged. This demonstration takes you through the process of generating these: 1. Extracting journal titles (e.g., from database searches or proposal references) 2. Associating those with Web of Science Subject Categories 3. Calculating several, comparative interdisciplinarity metrics [especially Integration, Specialization, Shannon and Herfindahl Diversity, coherence, team size] 4. Generating overlays on base science maps (that can incorporate social sciences) 5. Tracking multi-stage research knowledge flows: a) integration of knowledge (reflected by cited references) b) dissemination of findings (via publications) c) diffusion of findings (indicated by citing of those publications).

Session Title: Enhancing Organizational Learning With Technology: Implications of Diversity, Improving Response Rates, and Increasing Evaluation Capacity

Multipaper Session 135 to be held in Wekiwa 7 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Integrating Technology Into Evaluation

Chair(s):

Rebecca Culyba,  Emory University, rculyba@emory.edu

Discussant(s):

Nathan Balasubramanian,  Centennial Board of Cooperative Educational Services, nbala@cboces.org

Improving Response Rates for Multiple Populations: Lessons From the Field! Presenter(s): Barbara Packer-Muti, Nova Southeastern University, packerb@nova.edu Jennifer Reeves, Nova Southeastern University, jennreev@nova.edu Candace Lacey, Nova Southeastern University, lacey@nova.edu Abstract: Participation rate is typically recognized in the literature as one of the factors that can adversely impact the strength of survey research data. This presentation describes the techniques utilized to improve participation rate in a large, university-wide assessment of institutional engagement. The assessment, conducted annually for the past two years, involves web-based questionnaires with invitations sent to 28,000 students, 4,600 employees (including faculty, staff, and administration) and 50,000 alumni. Discussion focuses on the techniques used to increase response rate. Also included is a discussion of the successes and failures, and three years of data on response rates for each set of constituents.

Using Web-based Technologies to Increase Evaluation Capacity in Organizations Providing Child and Youth Mental Health Services in Ontario Presenter(s): Purnima Sundar, Children's Hospital of Eastern Ontario, psundar@cheo.on.ca Susan Kasprzak, Children's Hospital of Eastern Ontario, skasprzak@cheo.on.ca Evangeline Danseco, Children's Hospital of Eastern Ontario, edanseco@cheo.on.ca Tanya Witteveen, Children's Hospital of Eastern Ontario, twitteveen@cheo.on.ca Heather Woltman, Children's Hospital of Eastern Ontario, hwoltman@cheo.on.ca Abstract: Given today's climate of economic uncertainty and fiscal restraint, organizations providing child and youth mental health services are required to do so with limited resources. Within this context, service providers face added pressure to deliver evidenced-based programs and demonstrate program effectiveness. The Provincial Centre of Excellence for Child and Youth Mental Health at CHEO works with organizations to meet these demands by building capacity in program evaluation. While personal instruction and mentoring are an important ways of providing support, face-to-face consultations are not always cost-effective. In this presentation we will describe the use of interactive technology and computer-based learning as an alternative and/or complementary (to face-to-face) means of delivering evaluation information and training. We discuss the process of developing these tools, and share findings from our evaluation of their effectiveness in enhancing the evaluation-related supports we offer to providers of child and youth mental health services.

Finding Common Ground: Implications of Diversity in Data Collection in a Multi-Site, Initiative Evaluation Presenter(s): Virginia Houmes, Washington University in Saint Louis, vhoumes@wustl.edu Nancy Mueller, Washington University in Saint Louis, nmueller@wustl.edu Amy Stringer Hessel, Missouri Foundation for Health, astringerhessel@mffh.org Cheryl Kelly, Saint Louis University, kellycm@slu.edu Jessi Erickson, Saint Louis University, ericksjl@slu.edu Tanya Montgomery, Washington University in Saint Louis, tmontgomery@wustl.edu Doug Luke, Washington University in Saint Louis, dluke@wustl.edu Abstract: In 2007, a Missouri health foundation funded a comprehensive evaluation of a multi-site initiative focused on obesity prevention in Missouri. Projects funded through the initiative share common goals, but vary in focus, setting, and capacity. This can create challenges when evaluating progress, including capturing data on initiative goals; detecting broad, cross-site effects; and ensuring valid results. In conducting the initiative evaluation, we identified a core data set to be collected from each of the projects that captures common indicators while allowing for the diversity of the projects. We developed a web-based data collection system that enables centralized submission and access to the core data by grantees, evaluators, and the foundation. We also provided training on the system that was specifically tailored to each project's diverse needs and capacity. From our process, evaluators will learn the strengths and challenges of implementing a web-based data collection system within a multi-site, initiative-level evaluation.

Session Title: Developing Effective Surveys

Multipaper Session 136 to be held in Wekiwa 8 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the AEA Conference Committee

Chair(s):

Lija Greenseid,  Professional Data Analysts Inc, lija@pdastats.com

Using Context to Evaluate Survey Items Presenter(s): Michael Burke, RTI International, mburke@rti.org Abstract: Question reliability is an often overlooked possibility that may lead to an attenuation of results or, even worse, incorrect assessments of relationships between variables. Pretesting, cognitive testing, and disaster checks are all synonyms of the approach suggested, but the approach is unique in that is specifically calls for participants in the pretesting to draw on their own experiences and apply context to the items or issues at hand. The techniques that will be taught emphasize evaluator interpretation and expertise over subject/participant knowledge and forthrightness. As such, the session teaches evaluators how to collect useful information without having advance or complete knowledge about the way clients interpret or understand survey items or communication messages. Greater use of such methods will, it is hoped, reduce unreliability and increase the quality of evaluations conducted.

Addressing Context by Mixing Methods in Survey Development Presenter(s): Katherine Ryan, University of Illinois at Urbana-Champaign, k-ryan6@illinois.edu Nora Gannon, University of Illinois at Urbana-Champaign, ngannon2@illinois.edu Abstract: Attention to context in warranting inferences from data has gained increasing importance across all domains of evaluation (e.g., education, health care, environment studies) (Julnes & Rog, 2008). One result of this trend is a stronger need for robust methods to produce quality instruments used in evaluation (Desmione & Le Floch, 2008). This paper explores the use of a mixed methods sequential design for questionnaire development in a large-scale evaluation. In preparing for the pilot study, a sequential design was tailored to capture multiple perspectives from the diverse participants intended to complete the questionnaire and from individuals with expert knowledge of the questionnaire aims. Being inclusive of these perspectives permits revisions of the questionnaire that will increases consistency in interpretability across participants (e.g., teachers vs. principals) and between various contexts (e.g., low vs. high achieving schools). By utilizing this design, there is an anticipated increase in the quality of data and strengthening of the inferences warranted from that data to be used by decision makers in developing policies.

Stuck in the Middle: The Use and Interpretation of Mid-Points in Surveys Presenter(s): Joel Nadler, Southern Illinois University at Carbondale, jnadler@siu.edu Rebecca Weston, Southern Illinois University at Carbondale, weston@siu.edu Abstract: Likert type scales are common in survey research. Research suggests Likert type scales should utilize four to seven response options. What seems more a matter of taste is the use of a mid-point in Likert type scales. Researchers can use an even number of response options forcing a choice or use an odd number of responses allowing neutrality. The authors conducted a study comparing different response options on the same set of 28 attitudinal questions. Participants answered questions using one of the following: a 4 point scale (forced choice), a 5 point scale (3 represented 'Neither'), or a 4 point scale with an option of 'No Opinion' after the scale. Results indicated 25% of item means were significantly affected by different response options. Additionally, 'neither' was chosen significantly more often than 'no opinion' on 80% of the items. Implications of this study on response choices in evaluation will be discussed.

The Current Debates About Impact Evaluation Using Randomization: A Political and a Scientific Perspective Presenter(s): Rahel Kahlert, University of Texas at Austin, kahlert@mail.utexas.edu Abstract: The paper presentation analyzes the current debates about impact evaluation using randomization. The controversial issue remains whether the randomized controlled experiment (RCT) represents the 'gold standard' among evaluation approaches, regardless of the context of an evaluation. The author analyzes the randomization debate in U.S. education and in international development since the turn of the twentieth century comparatively. The presenter employs both a political perspective and scientific perspective (cf., Weiss 1972; Vedung 1998) to explain the rationales behind the promotion strategies of the respective sides in the debate. The presentation discusses strengths and limitations of randomized evaluation approaches for social and educational interventions, as put forward by both promoters and skeptics of randomized experiments. The author analyzes the arguments advanced by both sides and proposes ways in which this debate could be mediated.

Session Title: Indigenous Peoples in Evaluation TIG Business Meeting

Business Meeting Session 137 to be held in Wekiwa 9 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Indigenous Peoples in Evaluation TIG

TIG Leader(s):

Katherine Tibbetts, Kamehameha Schools, katibbet@ksbe.edu

Kaylani Rai, University of Wisconsin Milwaukee, kaylyanir@uwm.edu

Joan LaFrance, Mekinak Consulting, lafrancejl@gmail.com

Session Title: Timing Considerations in Evaluating Corrections Programs: How Context, Maturity of Program and Audiences Can Influence the Evaluation Process

Panel Session 138 to be held in Wekiwa 10 on Wednesday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Crime and Justice TIG

Chair(s):

Marian Kimball Eichinger, The Improve Group, mariane@theimprovegroup.com

Abstract: While recidivism rates are often the primary indicator of success in corrections programs, a valuable and relevant question is whether it is possible to utilize alternative measures and designs to determine "success" in such programs. We will host a lively discussion on the successes and challenges of evaluating release programs in corrections. Three evaluators will bring their experience and expertise in evaluating corrections programs to discuss how timing can influence an evaluation by way of data gathering context, program maturity, reporting audience, and relevant outcomes. Our panel will facilitate this discussion using examples from our own evaluation experiences that had differing data collection contexts, levels of program maturity, and end results that uncovered both unique and standard outcomes of success.

Life Skills Program Evaluation: Evaluating a Release Program Prior to Release Marian Kimball Eichinger, The Improve Group, mariane@theimprovegroup.com The Minnesota Department of Corrections designed a Life Skills program for incarcerated men and women at four correctional facilities. The program was developed after a comprehensive process that involved piloting portions of the program at one facility, developing relationships with partners, and conducting a needs assessment. The result was a long-term, personalized program to teach life skills in areas such as self-development, relationships, conflict management, making community connections, and generally improving behavior upon release. The evaluation was conducted while the program was in progress and utilized a variety of data collection methods including a needs assessment, participant interviews, class observations and an outcome survey. Results show that the Life Skills program significantly and positively impacted program participants. The panel discussion will raise the question of whether it is possible to determine the success of a program such as this while the participants are still incarcerated.

Minnesota Comprehensive Offender Re-entry Plan (MCORP): Using a Randomized Experimental Design to Evaluate a Prisoner Re-entry Project Grant Duwe, Minnesota Department of Corrections, grant.duwe@state.mn.us In 2008, the State of Minnesota implemented the Minnesota Comprehensive Offender Re-entry Plan (MCORP), a prisoner re-entry project. Based on the premise that offender re-entry begins as soon as offenders are admitted to prison, MCORP emphasizes increased collaboration between institutional caseworkers and supervision agents to provide planning, support, and direction for offenders to address their strengths and needs in both the institution and the community. It is hypothesized that MCORP will increase the extent to which offenders access employment, suitable housing, and programming in the community, which will lead to a reduction in recidivism. To evaluate whether MCORP is meeting these objectives, the Minnesota Department of Corrections has implemented a randomized experimental design in which eligible offenders have been randomly assigned to either the experimental or control groups. The presentation will focus on the challenges involved in preserving the integrity of a randomized experimental design implemented within a correctional setting.

Evaluating a Community-Based Release Program Under Legislative Mandate in a Short Demonstration Phase Timeframe Rebecca Stewart, The Improve Group, beckys@theimprovegroup.com This presentation explores independent evaluation methods and design for a community-based Minneapolis release program; both the program and the evaluation were funded by the Minnesota Legislature. This context provided the evaluator with access to State Corrections data and a mandate to evaluate the program in just over a year. The program was in a demonstration phase during this year, with program and data collection systems still developing throughout the evaluation. While this is short time frame to form findings on recidivism outcomes for an evolving program, the program and legislators wanted to achieve evaluation results that would inform whether or not state funding should be continued for the program. The presenter, the evaluation research manager, will explore challenges in gathering data in a community-based demonstration program, identifying appropriate outcomes for a short time frame evaluation and balancing the demands of different audiences in reporting.

Return to Evaluation 2009