Return to search form  

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Knowledge Management as a Guerilla Campaign: Leading the Horse to Water

Roundtable Presentation 701 to be held in the Boardroom on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Business and Industry TIG and the Integrating Technology Into Evaluation

Presenter(s):

Stephen Axelrad, Booz Allen Hamilton, axelrad_stephen@bah.com

Thomas E Ward II, United States Army, thomas.wardii@us.army.mil

Abstract: Many evaluation practitioners have experienced or observed the frustration of attempting to evaluate the effectiveness of a knowledge management (KM) project or initiative. This is particularly true 'after the fact' - when an evaluator is consulted to show how effective the implementation has been. Many organizations define successful KM initiatives when an intranet or online knowledge base is launched successfully, has no technical flaws, and is visited by a certain number of users. Few, if any, organizations understand how well KM resources and practices achieve their desired outcomes, which may include increased collaboration or greater innovation and creativity. The proposed roundtable discussion would explore how role evaluators can contribute to and/or lead an organization's effort to understand the impact of KM initiatives.

Roundtable Rotation II: How Can We Use Social Media for Program Evaluation?

Roundtable Presentation 701 to be held in the Boardroom on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Business and Industry TIG and the Integrating Technology Into Evaluation

Presenter(s):

Wendy DuBow, University of Colorado at Boulder, wendy.dubow@colorado.edu

Abstract: As evaluators in the 21st century, we cannot afford to ignore social media as an instrument for evaluation. Facebook, MySpace, YouTube, Second Life, Flikr, Twitter, and others are excellent places to find both homogeneous and heterogeneous groups of people. Blogs and discussion forums may be productive mediums for the evaluator. Yet, it can be difficult to figure out how to conduct evaluation in these new settings. Can we adapt our traditional evaluation techniques? Do we need to develop new techniques altogether? How does our approach impact the validity or generalizability of our studies? What has worked for you? Come to this roundtable to discuss how to make use of these media. Although also an interesting discussion, this roundtable is not intended to be about how to evaluate these media, but rather how to use them to conduct typical activities - program evaluation, outreach attempts, needs assessments, outcome assessments, and so on.

Session Title: Measuring Stakeholder Perspectives in College Access Program Evaluations

Multipaper Session 703 to be held in Panzacola Section F2 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the College Access Programs TIG

Chair(s):

Kurt Burkum,  ACT, kurt.burkum@act.org

Evaluating Equity of College Access in the Advanced Placement Initiative Presenter(s): Barbara Acosta, George Washington University, bacosta@ceee.gwu.edu Lori McGee, George Washington University, lmcgee@ceee.gwu.edu Julia Webster, Delaware Department of Education, jwebster@doe.k12.de.us Abstract: This presentation will examine lessons learned from an evaluation of the Advanced Placement Incentive Program (APIP), which aims to increase the participation of low-income students in rigorous coursework, improve academic outcomes, and prepare them for college. The evaluation design used Guskey's (2002) approach to evaluating professional development. Preliminary findings from the evaluation revealed that a major barrier to equity of college access had been ignored in the original design. Although all of the professional development objectives had been met for Guskey Levels 1 and 2, teachers had not changed their expectations regarding which students were capable of participating in rigorous coursework. In response to these findings, evaluators added an important indicator to measure changes in teacher attitudes. Audience members will learn the benefits and limitations of the Guskey Model for professional development evaluation, as well as additional considerations for evaluation design related to equity issues.

Factors Associated With Teacher and Student Perceptions of Readiness for College Presenter(s): Xinsheng Cai, American Institutes for Research, ccai@air.org Daniel Aladjem, American Institutes for Research, daladjem@air.org Sarah Bardack, American Institutes for Research, sbardack@air.org Victoria Marks, American Institutes for Research, vmarks@air.org Abstract: The purpose of the study was to examine the relationship among various classroom-, school-, and district-level factors and student and teacher perceptions of readiness for college. The data for the study were student and teacher surveys collected for the Evaluation of the GE Foundation Developing Futures program. Using hierarchical linear modeling, we found that students had high aspirations of college-going and felt prepared for college. Teachers, however, were less optimistic in their assessment of student readiness for postsecondary education. Student trust of their teachers and students' sense that teachers personalized instruction were the most robust predictors for student self-perception of college readiness. Teacher self-efficacy, which was associated with professional development, principal leadership, and teacher collaboration strongly predicted teacher perceptions. The findings emphasize the importance of creating a trusting classroom environment in which teachers feel confident to promoting college readiness through teacher training, principal support and teacher collaboration.

Understanding College Access Through the Lens of High School Seniors: A Case Study Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) North Carolina Presenter(s): Monifa Beverly, University of Central Florida, mbeverly@mail.ucf.edu Karyl Askew, University of North Carolina at Chapel Hill, karyls@email.unc.edu Michelle Jay, University of South Carolina, jaym@gwm.sc.edu Abstract: This presentation shares a case study of three successful GEAR UP NC programs. The purpose of this study is to explore what factors most influence post-secondary training choices among first time college attendees. In particular, the researchers wanted to ascertain whether participation in the GEAR UP NC program has an influence on participating students' career plans. This study was done, done in conjunction with a one-year evaluation of GEAR UP, asked high school seniors to tell their stories which potentially can illuminate or 'make real' their experiences in the GEAR UP NC program. The presenters will expand on case study and evaluation and will share useful strategies to engage in case study research effectively. This presentation also offers ideas about how faculty members can unite authentic learning for graduate students while at the same time providing quality services for social service agencies that are at no cost to them.

Session Title: Accounting for Contextual Factors in the Evaluation of Continuing Medical Education Programs

Multipaper Session 704 to be held in Panzacola Section F3 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Health Evaluation TIG

Chair(s):

Kevin Khamarko, University of California San Francisco, kevin.khamarko@ucsf.edu

Abstract: The AIDS Education and Training Centers (AETCs) supports a network of 11 regional centers that conduct targeted, multi-disciplinary education and training programs for health care providers treating persons with HIV/AIDS. The mission of the AETCs is to improve the quality of life for patients living with HIV/AIDS through the provision of high quality continuing medical education and training. Up to now, the AETCs have utilized Donald Kirkpatrick's typology for measuring four aspects of training: participants' reactions to training; learning; behavior change; and ultimately, the results for patients of trained providers. Although these domains are useful for conceptualizing and carrying out evaluation, we have found that application of training material is complicated by aspects of the setting, organization and social context in which the content of the training is applied. This session will provide specific examples of how the AETCs are accounting for contextual factors in their current training evaluations.

Understanding the Barriers and Facilitators to Applying the Content of AIDS Education Training Centers (AETC) Trainings

Kevin Khamarko, University of California San Francisco, kevin.khamarko@ucsf.edu

Kimberly Koester, University of California San Francisco, kimberly.koester@ucsf.edu

Janet J Myers, University of California San Francisco, janet.myers@ucsf.edu

The AIDS Education and Training Centers' National Evaluation Center (AETC NEC) provides evaluation development and technical support to the national AETC program. In an effort to better understand the pre-training, training implementation, and post-training conditions that influence training transfer, we qualitatively assessed the barriers and facilitators to implementing and sustaining new knowledge and skills gained during AETC training events. Twenty-two interviews were conducted with AETC faculty trainers throughout the country. Barriers to application included: 1) concerns about time and workload - in light of clinical priorities, training concepts may not be perceived as a high priority; 2) concerns about process and procedures necessary to apply concept; 3) lack of support from management; and 4) time lag between training and skill application. Trainers also identified several facilitators to training transfer, including: 1) provision of quality training; 2) approval of management; and 3) organization and time to implement training concepts.

Trainees, Settings and Contexts in Evaluation of Capacity Building: Does Our Training Program Enhance Capacity for HIV Care in Minority Communities?

Mari Millery, Columbia University, mm994@columbia.edu

Melissa Laurie, Columbia University, ml2489@columbia.edu

Daria Boccher-Lattimore, Columbia University, dmb82@columbia.edu

This presentation describes the efforts of the New York/New Jersey AIDS Education & Training Center (NY/NJ AETC) to evaluate its longitudinal HIV-training certificate program for minority and minority-serving healthcare providers in high HIV-prevalence minority communities. In the ACCESS (AETC Community Clinical Education & Sustaining Support) program, participants receive 20 or more hours of clinical training and are connected to HIV care information, resources and mentoring. Data sources for the evaluation include pre, post and 6-month follow-up surveys with trainees, trainings plans and interviews and focus groups with program staff. Indicators are included to capture increase in trainees "social capital". To date, 73 providers were enrolled in the program. The evaluation illustrates the importance of context on two levels: 1- individual trainees apply their learning within the contextual constraints of their work settings and 2- individual trainees are critical vehicles of bringing HIV knowledge to the context of high need communities.

Beyond Knowledge Change: An Exploratory Analysis of Clinical Contexts in a Longitudinal HIV Training Series

Blake Tyler McGee, Emory University, btmcgee@emory.edu

Rebecca Culyba, Emory University, rculyba@emory.edu

The Southeast AIDS Training and Education Center provides intensive, on-site training events for clinical and non-clinical service providers at outpatient facilities across the region over a one- to two-year period. To date, pre- and post-training data from this longitudinal training program reveal measurable improvements in HIV-related knowledge and skills. In particular, there was a statistically significant increase in mean score on the provider assessment tool over the first year of training at all participating sites. However, context may mitigate the application of new knowledge and skills. Therefore, we asked trainees to rate the presence of enabling factors in their clinic environments. These ratings formed an internally reliable scale that was not correlated with trainee test score change, suggesting that some learners with extensive knowledge acquisition are limited in applying training content. Additionally, trainees provided qualitative information about challenges to providing HIV care that may constitute further obstacles to training transfer.

Session Title: Tools and Processes for Monitoring and Evaluating Systems Change

Multipaper Session 705 to be held in Panzacola Section F4 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Systems in Evaluation TIG

Chair(s):

Bob Williams,  Independent Consultant, bobwill@actrix.co.nz

Keeping Track Under Complex Conditions: The Process Monitoring of Impacts Approach Presenter(s): Richard Hummelbrunner, OEAR Regional Development Consultants, hummelbrunner@oear.at Abstract: This approach systematically observes those processes, which are expected to lead to results or impacts of an intervention. It builds on the assumption that inputs (as well as outputs) have to be used by someone to produce desired effects. A set of hypotheses are identified on the desired use of inputs or outputs by various actors (e.g. partners, project owners, target groups), which are considered decisive for the achievement of effects. These hypotheses are incorporated in logic models as statements for 'intended use', and these assumptions are monitored during implementation - whether they remain valid, actually take place - or should be amended (e.g. to capture new developments or unintended effects). The paper describes the approach as well as the experience gained in Austria, where it has been applied for monitoring EU Structural Fund programs, to provide an adequate understanding of program performance under complex and dynamic implementing conditions.

Addressing the Challenges of Systems Change Evaluation: Tools for assessment Presenter(s): Anna F Lobosco, Developmental Disabilities Planning Council of New York State, alobosco@ddpc.state.ny.us Dianna L Newman, University at Albany - State University of New York, eval@csc.albany.edu Susan L Rogers, University at Albany - State University of New York, bottlecap22@hotmail.com Abstract: Existing definitions and models of systems change are reviewed, and practical challenges of systems change evaluation will be discussed. Meta-evaluation information will be used to identify the kinds of changes that occur when systems change efforts have been successful. Based on this, a set of performance indicators will be introduced for use in evaluation of systemic change efforts. Finally, efforts to assess within the context of ongoing systems change evaluation projects are discussed, and the use of a new assessment tool is introduced. Information from the fields of developmental disabilities, mental health, and education will be used as examples.

Evaluating Systems Change at the Massachusetts Department of Developmental Services Using the Build Initiative Framework Presenter(s): Jennifer Sulewski, University of Massachusetts Boston, jennifer.sulewski@umb.edu Abstract: Since 2006, the Massachusetts Department of Developmental Services has been working to make its system of day services and supports more focused on competitive employment outcomes. This effort, undertaken with assistance from the Massachusetts Medicaid Infrastructure and Comprehensive Employment Opportunities (MI-CEO) grant has involved intervention at multiple levels including: 1) changes to agency-wide and regional policies and practices, 2) technical assistance to local service providers, and 3) direct support staff training. Using the systems initiative evaluation framework developed by the Build Initiative, MI-CEO researchers developed a theory of change model incorporating the major strategies of the systems change effort. The model provided an organizing framework for a multi-method and multi-level evaluation that includes both system-wide evaluation strategies and evaluation of individual project components. For example, bellwether interviews assess high-level policy changes, case studies highlight successes and challenges at the local and individual levels, and participant surveys evaluate training outcomes.

Session Title: Supportive Pathways for Emerging Indigenous Evaluators: From Corporate, Business and Community Lives to Evaluation

Multipaper Session 706 to be held in Panzacola Section G1 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Graduate Student and New Evaluator TIG

Chair(s):

Kataraina Pipi, FEM (2006) Limited, kpipi@xtra.co.nz

Abstract: Three evaluators with varying levels of expertise describe aspects of their journeys into evaluation as indigenous women from Aotearoa, New Zealand. They entered the evaluation field from varying cultural contexts - one woman entered from the corporate world, and the other two women came with business, social services and community backgrounds. All three women however, practice from a cultural context with a firm view to utilization of Kaupapa Maori (indigenous worldview) approaches. Each person describes her evolution into evaluation, highlighting the factors contributing to her progress. Each presentation concludes with key evaluative learnings, together with affirmations of supportive strategies for the growth of student and emerging indigenous evaluators by indigenous and non-indigenous practitioners.

From Flying High in the Travel Industry to Tentative Steps in Program Evaluation

Vivienne Kennedy, VK Associates Limited, vppk@snap.net.nz

This presentation tells the story of how a student evaluator came from a management position in the airline industry to conducting community-based research and program evaluation with her people. Her journey describes starting from scratch by transcribing interviews, note taking at meetings, and developing skills for analysis and synthesis of information, through to report writing. The value of mentors and a supportive network to guide and assist her in her new vocation is described, as are some of the skills and evaluative knowledge gained on the journey of working with the indigenous people of New Zealand. This story aims to give tips to emergent evaluators - applicable to indigenous and non-indigenous practitioners, to help them to realise their aspirations, as well as to provide a message to professional practitioners as to the value, merit and worth of their contributions to the development of upcoming evaluators.

Language Activist = Professional Evaluator

Kirimatao Paipa, Kia Maia Limited, kiripaipa@ihug.co.nz

Kiri will present her journey from being a rurally-based Maori language activist to a city-based emergent evaluator. Her journey highlights how cultural knowledge continues to affect the development of professional evaluation knowledge and practices. Kiri will talk about how language and culture is a lens through which evaluation is understood and practiced. This story would be applicable to emergent evaluators worldwide, and particularly for indigenous people who ask themselves, "Do I want to be an evaluator? How does being an evaluator add value and support the emancipatory goals of indigenous peoples?"

Facilitating Evaluative Melodic Moments

Kataraina Pipi, FEM (2006) Limited, kpipi@xtra.co.nz

This presenter is a facilitator, an evaluator and a composer of music who has over ten years practical experience in many aspects of evaluation. Her story relates to how she has been able to weave her collective strengths together, utilising Kaupapa Maori practice in evaluation to affirm and validate the work of her people. She is currently undertaking academic studies to add to her basket of knowledge and to provide the theory that gives credence to the approaches and methods she uses. Those new to evaluation as well as emerging and indigenous evaluators can learn much about how to bring together various strands of personal experience and skills to this discipline, whilst being true to oneself in order to support nationhood building.

Session Title: Online Surveys in Evaluation: Issues and Lessons Learned From Four Evaluation Projects

Panel Session 707 to be held in Panzacola Section G2 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Integrating Technology Into Evaluation

Chair(s):

Hsin-Ling Hung, University of Cincinnati, hunghg@ucmail.uc.edu

Abstract: The use of technology in evaluation, for example, online (web-based or e-mail) surveys in data collection, has become increasingly widespread evaluation activities due to many advantages including time and cost efficiencies, ease of survey administration, and convenience for respondents. However, researchers and evaluators likely encounter methodological issues (e.g., sampling, response rate, instrumentation, etc.) and data collection challenges in conducting online surveys. This panel presentation attempts to address these issues and challenges through discussion of both the theoretical foundations and practical perspectives regarding online surveys. Linking the ideal to the practical world of conducting online surveys in the field of evaluation will be the focus of this presentation. Challenges encountered, possible solutions and lessons learned in real-life evaluation utilizing online surveys will be discussed.

Handling Technological Tools to Have Meaningful Evaluation Data Imelda Castaneda-Emenaker, University of Cincinnati, castania@ucmail.uc.edu This presentation focuses on the use of online survey in the evaluation of Client K's teacher professional development program integrating literacy across the curriculum. No names or usual form of student identification was used on the online survey because the client was concerned about non-response. The main challenge lies in matching the students' responses with their teachers'. The college system provided a database of 6000 students of all the teachers in the program. Teachers were handling several courses and students were taking different courses so the list contains duplicate names of teachers, students, and email addresses. The email addresses were used as passwords to enter the survey. Eventually, there were only few students that participated with the online survey. The dilemma about identification issues and the evaluator's naiveté about a database easily solving matching concerns led to data that are not very meaningful for the evaluation processes.

Context-Driven Use of Online Surveys in Educational Program Evaluations Janet Matulis, University of Cincinnati, janet.matulis@uc.edu This presentation discusses how the context of an educational program helps determine whether data collection is best accomplished via online survey, pencil-or-paper survey, or some combination of the two. Although the tendency is for technology-based survey administration to be considered the ideal, this methodology may not be the most appropriate choice for an educational program evaluation. When online surveys are deemed appropriate for an evaluation, contexts of the educational program can also dictate whether data integrity would best be attained through survey administration via email and/or a website link. The rationale and process for administering a student survey via both online and paper formats as part of the evaluation of an art museum's K-12 school-based programs will be discussed.

Online Survey on International Participants and Using Technology in International Collaboration: Challenges and Lessons Learned Hsin-Ling Hung, University of Cincinnati, hunghg@ucmail.uc.edu This presentation will be based on an electronic, Delphi study of educational program evaluation (EPE) in the Asia-Pacific region conducted by a team of researchers in the U.S. and Taiwan. Unlike the other examples in this panel presentation, this study used both e-mail and commercial service for web-based survey for data collection. Challenges associated with commercial web-based survey as well as issues of online survey specific to the Delphi methods and Needs Assessment will be discussed. General methodological consideration of the survey research will be introduced. In addition, utilization of technology in international research collaboration will also be discussed. The presentation would be of interest to evaluators and researchers in online survey as well as application of technology in research and international research collaboration.

Challenges and Lessons learned From a Community-Based Initiative Evaluation Project Mary Marx, University of Cincinnati, mary.marx@uc.edu This presentation is based on community health project to conduct community-based participatory research focusing on nutrition and physical activity of minority adults and youth. An initial challenge was related to instrument development, specifically the difficulty in convincing the community agency of the need to shorten the instrument. As experienced in this project, clients often feel it is important to take advantage of a data collection opportunity by insisting upon an extremely long and cumbersome online survey to gain what they perceive to be high value for the money they spend on an evaluation. A technical issue involving the survey software's management of "missing values" was also encountered. A third issue involved data analysis, particularly alternating back and forth between different software packages. When sharing the data with a researcher consultant working with the community agency, care was required in recoding where needed to assure that the calculations would be accurate.

Session Title: AEA Student Travel Awards Winners in Dialogue

Panel Session 708 to be held in Panzacola Section H1 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Presidential Strand

Chair(s):

Linda Schrader, Florida State University, schrader@coe.fsu.edu

Discussant(s):

Doug Horton, Independent Consultant, d.horton@mac.com

Kevin Ly, Urban Policy Student at Milano, ly.kevin@gmail.com

Sergio Prada, University of Maryland, sprada1@umbc.edu

Debra Rog, Westat, debrarog@westat.com

Kristy Bloxham, Utah State University, kristy.bloxham@gmail.com

Haili Cheng, University of Massachusetts Amherst, hailicheng@gmail.com

Brandi Gilbert, University of Colorado at Boulder, brandi.gilbert3227@gmail.com

Douglas Grane, University of Iowa, douglas.grane@gmail.com

Mark Hansen, University of California Los Angeles, hansen.mark@gmail.com

Deborah Smith, University of Southern Maine, deborah.smith.2006@gmail.com

Jennifer Terpstra, University of British Columbia, jlterp@interchange.ubc.ca

Session Title: Propensity Scores: Statistical Analyses for Strengthening Causal Validity

Multipaper Session 709 to be held in Panzacola Section H2 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Quantitative Methods: Theory and Design TIG

Chair(s):

MH Clark,  Southern Illinois University at Carbondale, mhclark@siu.edu

Application of Data Integration and Matching Techniques in Evaluation Presenter(s): Mukaria Itang'ata, Western Michigan University, mukaria.itangata@wmich.edu Abstract: Often evaluators using non-experimental or observational data sets are faced with situations where they must conduct comparative studies between two or more programs or outcomes. In many of these situations there may be one large population compared to a much smaller foci subpopulation. In such situations, a logical way to study the subpopulations is to create one or more matched comparison samples drawn from the larger population. However, a question arises: how best to match the subpopulation participants when randomization is not possible? This presentation will compare different data integration and matching techniques to determine if these matching techniques result in differential evaluation conclusions under different experimental conditions.

Issues in Selecting Covariates for Propensity Score Adjustments Presenter(s): William Shadish, University of California Merced, wshadish@ucmerced.edu Abstract: When randomized experiments cannot be used for ethical or practical reasons, nonrandomized experiments are often used instead. However, selection bias may result in doubt about bias of the resulting effect estimates. Various methods for adjusting those estimates have been proposed, including ordinary least squares covariate adjustment and propensity score analysis. This paper explore issues in the selection of covariates used in such adjustments. The paper will introduce a study that randomly assigned participants to be in a randomized or nonrandomized experiment, and then used various adjustments to the nonrandomized experiment to reproduce the results from the randomized experiment. The paper then explores various features of the covariates to shed light on how one might choose better covariates for such analyses.

A Comparison of Instrumental Variables and Propensity Score Approaches to Adjusting Group Treatment Selection Bias Presenter(s): Ning Rui, Research for Better Schools, rui@rbs.org Abstract: One of the key challenges in identifying the causal effect of a group treatment has been to effectively control for group selection bias in order to properly estimate the treatment effect. Scholars have gained tremendous insights into the use of propensity score matching (PS) and instrumental variables (IV) strategies to address selection bias in observational studies. However, little is known about whether PS and IV methods are likely to generate similar estimates of treatment effects. The present study aims to compare both approaches to adjusting for selection bias in estimating the effect of community colleges on overall educational attainment using the High School and Beyond (HS&B) data. Both approaches agreed on the direction but differed substantially on the size of estimates for community college diversion effect on educational attainment. Two methods' relative advantages and disadvantages are discussed and suggestions for future studies are provided.

Use Resampling to Estimate and Compare the Selection Bias of Propensity Score Matching Techniques Presenter(s): Haiyan Bai, University of Central Florida, hbai@mail.ucf.edu Wei Pan, University of Cincinnati, wei.pan@uc.edu Abstract: Propensity score matching has become increasingly popular in program evaluation when randomization is improbable. However, selection bias could still exist in the extant propensity score matching techniques. This present study utilizes resampling to estimate and compare the bias of the four types of commonly-used propensity score matching techniques: Nearest neighborhood, caliper, stratification, and kernel. Monte Carlo simulation study is conducted with three sample sizes (small, medium, and large) across the following four conditions: (a) without resampling; and (b) with resampling implemented (b1) before propensity score estimation, (b2) between propensity score estimation and matching, and (b3) after propensity score matching. The standard errors and confidence intervals for the treatment effects are compared on the totally 48(4x3x4) different simulation conditions. This study would provide researchers scientific advice on which matching technique performs the best under what data conditions and when to use resampling to obtain less biased estimates of the treatment effects.

Session Title: Demonstrating How to Facilitate Data-Driven Decision Making: Three Tools For Providing People With the Data They Need, for Conducting Organized and Systematic Needs Assessments, and for Building Client Capacity to Assess Progress and Outcomes

Demonstration Session 710 to be held in Panzacola Section H3 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Evaluation Use TIG

Presenter(s):

Laura Feldman, University of Wyoming, lfeldman@uwyo.edu

Humphrey Costello, University of Wyoming, hcostell@uwyo.edu

Rodney Wambeam, University of Wyoming, rodney@uwyo.edu

Abstract: In this demonstration, the three presenters will each highlight a different technique for making data central to informing programming, policy, and funding decisions. As an example of how to make data more accessible and to increase its use, the first presenter will demonstrate an interactive graphing website that includes options for generating data fact sheets. The second presenter will introduce participants to a workbook that gives clear, step-by step instructions on conducting a comprehensive needs assessment. The third presenter will discuss working with communities to create local level evaluation plans during site visits using an evaluation workbook and based upon previous needs assessment and strategic planning documents. The presenters will emphasize how each tool can be adapted for use in different contexts, across a variety of fields, and among a diverse group of stakeholders.

Session Title: A Conversation Among Emerging and Seasoned Professionals in Evaluation and Diversity: Challenges, Opportunities, and Lessons Learned

Think Tank Session 711 to be held in Panzacola Section H4 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Non-profit and Foundations Evaluation TIG

Chair(s):

Gerri Spilka, OMG Center for Collaborative Learning, gerri@omgcenter.org

Rodney Hopson, Duquesne University, hopson@duq.edu

Presenter(s):

Pauline Brooks, Brooks Consulting and Associates, pbrooks_3@hotmail.com

Domingo Moronta, OMG Center for Collaborative Learning, domingo.moronta@gmail.com

Patricia Patrizi, Patrizi and Associates, patti@patriziassociates.com

Debra Perez, Robert Wood Johnson Foundation, dperez@rwjf.org

Gerri Spilka, OMG Center for Collaborative Learning, gerri@omgcenter.org

Howard Walters, OMG Center for Collaborative Learning, howard@omgcenter.org

Huilan Yang, WK Kellogg Foundation, hy1@wkkf.org

Vivian Figueredo, OMG Center for Collaborative Learning, vivian@omgcenter.org

Elvis Fraser, Bill & Melinda Gates Foundation, elvis.fraser@gatesfoundation.org

Astrid Hendricks, The California Endowment, ahendricks@calendow.org

Brenda Henry, Robert Wood Johnson Foundation, bhenry@rwjf.org

Katrina Herbert, HighScope Educational Research Foundation, kherbert@highscope.org

Summer Jackson, David and Lucile Packard Foundation, snjackson22@gmail.com

Jill Jim, Wilder Research, jilljim2003@hotmail.com

Ricardo Millett, Millett & Associates, ricardo@ricardomillett.com

Abstract: A number of evaluation readings and resources in the last decade or so focus on the need for evaluation to be relevant in diverse populations and communities (Hood, et.al, 2005; Hopson, 2004; Kirkhart, 1995; Madison, 1992; Mertens, 2005). In philanthropic and funding agencies, evaluation in culturally diverse communities is increasingly important (Council on Foundations, 2008; Greene, et.al, 2004; National Science Foundation, 2002, 2001, 2000). The philanthropic and non-profit world is at a critical juncture in contributing to social change agendas were issues of difference and diversity matter regarding race, ethnicity, disability, gender, and so on. The recent blossoming of pipeline and pathways efforts are perceived as one potential way to bring evaluators and emerging professionals of color in particular to the field of evaluation. This interactive panel session brings together emerging and seasoned professionals as a way to share and learn experiences in the practice of evaluation, especially around topics of diversity, culture, social justice, and social change.

Session Title: Accountability, Context, Transparency, and President Obama: Evaluation in the New Administration

Panel Session 712 to be held in Sebastian Section I1 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Government Evaluation TIG

Chair(s):

David J Bernstein, Westat, davidbernstein@westat.com

Discussant(s):

David J Bernstein, Westat, davidbernstein@westat.com

Abstract: This panel will explore the impact of U.S. President Barack Obama and his Administration on accountability, transparency, evaluation, and the context of U.S. federal evaluation. A distinguished panel of experts will provide their well-informed options to address these and other questions: * Was then Senator Barack Obama’s campaign focus on increased accountability and transparency campaign rhetoric or part of a larger plan to increase government efforts to capture information about government programs through evaluation and performance measurement and make that information accessible to the public? * What impact is the economic stimulus package having on federal accountability and evaluation-related efforts? * What is the status of the Program Assessment Rating Tool, and what changes have occurred or are envisioned? * What have we learned about the new Administration’s approach to evaluation policy? * What has been the impact on evaluation practice within federal agencies?

What's Up Now in the Federal Government? George F Grob, Center for Public Program Evaluation, georgefgrob@cs.com U.S. President Barack Obama has announced his principles of accountability, performance, and transparency. He wants to change the Program Assessment Rating Tool (PART) to put results in context, avoid ideological bases for performance measures, and involve the Congress and the public in setting program goals. We have seen aspects of these concepts in the economic stimulus legislation, and in health and education proposals. AEA has presented its "Evaluation Roadmap for a More Effective Government" to the federal Office of Management and Budget (OMB). This presentation will report where Federal evaluation policy stands. Potential topics include the role of Inspectors General (IGs) in the stimulus package, evaluation policies in the budget, the role of the Chief Performance Officer and Performance Council, the Government Performance and Results Act (GPRA), PART, OMB evaluation initiatives, and how the evaluation function is playing out in education, and health reform, and international development.

Will the New Calls for Accountability in Federal Spending Lead to More and More Useful: Program Evaluation? Stephanie L Shipman, United States Government Accountability Office, shipmans@gao.gov The past year's political transition in the federal government has strengthened interest in ensuring government transparency, accountability, and results for the American people. Large public investments of the economic stimulus package are accompanied by a blizzard of reporting requirements to show the public how their money is spent. The President demonstrated his commitment to improving results through announcing a budget scrub for wasteful and redundant spending, and promising improvements in evaluation initiatives begun under the last Administration (such as PART). Congress proposes to increase program oversight and more frequently demands "rigorous" evaluation evidence to reduce wasteful and ineffective spending. From the perspective of Congress' primary oversight agency and AEA's Evaluation Policy Task Force, Ms. Shipman will discuss some opportunities and challenges identified in GAO's work with Congress to help ensure that a diverse array of program evaluation activities will provide credible results that can best inform efforts to improve government.

Evaluation and Measurement in the New Administration: View From the Trenches Kathryn E Newcomer, George Washington University, newcomer@gwu.edu With each election that results in a change in administration, it is reasonable to expect that there will be changes not only resulting from the appointment of high-level government officials, but also dramatic changes in policy, perspectives, programmatic focus, and government operations. The Administration of President Obama is no exception. The President's campaign emphasis on accountability and transparency, combined with concerns about the earlier economic stimulus efforts involving the banking and finance industry, has heightened the expectations for the new Administration. This presentation will include a discussion of the challenges and opportunities for federal agencies and federal evaluators that result from these heightened expectations. Specific agency reactions and responses will be discussed. Finally, the presentation will address whether the increased demands for evaluation and performance information are being met with resources to carry out such activities.

Accountability, Transparency, and Effectiveness Under President Obama: The United States Department of Education's (ED)'s Evaluation Response Alan L Ginsburg, United States Department of Education, alan.ginsburg@ed.gov The U.S. Department of Education (ED) is receiving about $50 billion fiscal stimulus fund in 2009-2010, nearly double its annual budget. ED is committed to being accountable for effective spending of the new funds. ED is committed to using the opportunity created by the expanded funding to weed out ineffective programs and make existing programs work better. This presentation will describe how ED's evaluation strategies are responding to the need for improved accountability, transparency and effectiveness in the use of ED funds within a complex federal-state-local system through a multi-method evaluation approach that includes: - Strengthening the relevance and data quality of ED's performance indicators. - Assessing the effectiveness of the many smaller ED programs that lack evaluation evidence. - Using research and evaluation evidence to provide formative guidance for stimulus and program spending. - Presenting evaluation information to the Congress and the public in a timely and clear way.

Session Title: Alternatives to the Conventional Statistical Counterfactual: Assessing Effects and Impacts in Real-world Evaluations

Think Tank Session 713 to be held in Sebastian Section I2 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Quantitative Methods: Theory and Design TIG

Presenter(s):

Michael Bamberger, Independent Consultant, michaelbamberger@gmail.com

Discussant(s):

Jim Rugh, Independent Consultant, jimrugh@mindspring.com

Fred Carden, International Development Research Centre, fcarden@idrc.ca

Michael Bamberger, Independent Consultant, michaelbamberger@gmail.com

Abstract: Probably no more than ten percent of project or program evaluations use strong quantitative designs where impacts are estimated using a statistically defined counterfactual. Yet clients and other stakeholders of the remaining ninety percent (or more) of programs are equally concerned to know the extent to which their interventions have achieved demonstrable impact. Most evaluators believe that estimating program effects and impacts requires a methodology to define and eliminate alternative explanations (rival hypotheses) for the observed changes in the program population. So what advice can we offer to evaluators on alternatives to the conventional statistical counterfactual (e.g. randomized control trials and quasi-experimental designs)? The think tank will review some of the alternative approaches mentioned in the literature including: logic models, concept mapping, theories of change, most significant changes, participatory rapid appraisal and other participatory group methods, pipeline evaluation designs and creative uses of secondary data.

Session Title: Evaluation in an Ever-changing Context: Two Differing Approaches to Evaluating Homeless Programs

Multipaper Session 714 to be held in Sebastian Section I3 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Special Needs Populations TIG

Chair(s):

Kristin Stainbrook,  Advocates for Human Potential Inc, kstainbrook@ahpnet.com

Adapting Evaluation Measures and Methods When the Local Environment Changes Presenter(s): Kristin Stainbrook, Advocates for Human Potential Inc, kstainbrook@ahpnet.com Abstract: Useful evaluations of treatment programs reveal how well programs perform in relation to client outcomes. Frequently, programs must alter their program services to adapt to environmental changes. In turn, evaluations must also adapt to ensure the usefulness of findings. This presentation will describe the ways an evaluation of a trauma treatment program for homeless mothers was modified when changes in the local economy and housing market necessitated altering the program. Clients' lengthened shelter stays meant some outcomes, including residential stability, could not be assessed with existing methods. In response, the evaluation secured secondary data sources in attempt to these outcomes. The presenter will discuss the context in which program changes occurred; how program changes altered the evaluation questions; how questions about client outcomes were modified; how changes in evaluation were negotiated with program managers; and the changes to methodologies to ensure relevant information on client outcomes was collected.

Social and Political Context: A Discussion of the Evaluation of the Miami-Dade County Homeless Trust Presenter(s): Amy Donley, University of Central Florida, adonley@mail.ucf.edu James Wright, University of Central Florida, jwright@mail.ucf.edu Jenna Truman, University of Central Florida, jltruman@mail.ucf.edu Abstract: This paper presents findings and discusses the ways in which the evaluation of the Miami Dade Homeless Trust was conducted by researchers at the Institute of Social and Behavioral Sciences at the University of Central Florida. The Miami Dade Homeless Trust funds a continuum of care which provides services to homeless people ranging from outreach and emergency shelter to long term permanent housing with wrap-around services. The evaluation consisted of a multi-method design which combined quantitative and qualitative data analysis techniques. While the Miami Dade County homeless continuum of care is considered to be a model around the country, one of the main purposes of the evaluation was to determine what improvements could be made within the system.

Session Title: Study Quality in Context: Mechanisms to Improve and Assess Evaluations

Panel Session 715 to be held in Sebastian Section I4 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Research on Evaluation TIG

Chair(s):

Jeremy Lonsdale, National Audit Office United Kingdom, jeremy.lonsdale@nao.gsi.gov.uk

Discussant(s):

Robert Orwin, Westat, robertorwin@westat.com

Abstract: Study quality is inherently linked to context yet context also shapes how quality is perceived and the purposes evaluation serves. The first paper reflects on practice. By comparing and contrasting multiple perspectives and contexts, a framework is developed to help situate the definitions and uses of metaevaluation in theory and practice. The second paper looks across evaluation recommendations made by GAO in its oversight role. Context sophistication, a key GAO standard, helps it examine agency performance. GAO reviews have resulted in a variety of recommendations to assist agencies through improving their evaluations and capacity. In the U.K. value-for-money context, the third paper argues for a more sophisticated use of models and methods to evaluate programs with complicated delivery mechanisms situated in complex environments. The last paper refers back to the framework, draws on practice, and examines systematic reviews and associated metaevaluative tasks for assessing or synthesizing evaluations in their context.

Untangling Definitions and Uses of Metaevaluation Leslie J Cooksy, University of Delaware, ljcooksy@udel.edu Valerie J Caracelli, United States Government Accountability Office, caracelliv@gao.gov The term "metaevaluation" is used in multiple ways and the action of metaevaluation has multiple purposes. For example, "metaevaluation" has been used to refer to the individual evaluation of an evaluation, on the one hand, and as a synonym for meta-analysis, on the other. Metaevaluation serves as a means of assessing study quality in meta-analysis but the terms are not interchangeable. Metaevaluation, the action, has also been used in evaluation synthesis, assessment of capacity needs, and as one of the Program Evaluation Standards (Joint Committee, 1994). This paper will untangle the definitions and uses of metaevaluation and provide a framework for the concept. It will provide a backdrop for approaches to metaevaluation used in practice illustrated by other papers in the session.

The External Review and Oversight of Evaluation: Some Insights From the Experiences of the US Government Accountability Office (GAO) Martin de Alteriis, United States Government Accountability Office, dealteriism@gao.gov GAO, the oversight body for the United States Congress, promotes evaluation in a variety of ways. It conducts evaluations of federally-funded programs, supports capacity building in the federal government, and examines how federal agencies evaluate their programs. This last oversight role can lead to metaevaluation or associated recommendations about evaluation made to agencies. In the last three years, well over one hundred reports made recommendations about evaluation and are listed in GAO's publication database. A preliminary examination of this database shows that GAO has made recommendations for improvements in the following areas: Considering Evaluation in Program Design and Implementation; Planning for Evaluations; Conducting Impact Evaluations; Strengthening Existing Evaluations; Using the Results of Evaluations; and Improving Agencies' Ability to Evaluate. A subset of reports will be reviewed to determine the issues, researchable questions, and methods used to support recommendations and the implications for improving agency evaluation in the federal government.

Evaluating Complex Systems and Contexts: The Use of Operational Modeling in Performance Audit Elena Bechberger, National Audit Office United Kingdom, elena.bechberger@nao.gsi.gov.uk The UK's National Audit Office (NAO) produces 60 reports annually which assess the economy, efficiency and effectiveness with which central government bodies use public money. Evaluating the 'value for money' of programs and organizations is often challenging, particularly where the delivery chain is complex and performance strongly influenced by the social and economic context in which they operate. In order to deal with these challenges, the NAO has made increasing use of innovative methods, including advanced quantitative techniques such as operational modeling. This paper discusses the use of systems analysis frameworks to evaluate complex delivery chains in complex environments. Based on practical examples from NAO studies in the previous two years, it outlines both the value such approaches can add to assessing and improving performance, but also the limitations and difficulties associated with using them in the context of performance audits.

Metaevaluative Tasks in Evaluation Synthesis and Evidence-Based Practice Grading Systems Valerie J Caracelli, United States Government Accountability Office, caracelliv@gao.gov Leslie J Cooksy, University of Delaware, ljcooksy@udel.edu Drawing on aspects of the framework discussed in the first paper, as well as empirical examples from GAO and other contexts, this presentation addresses metaevaluation projects that examine a body of evidence pertaining to specific policy questions. Metaevaluative tasks requiring judgments of quality are frequently a part of such evaluation methodologies as peer review, evaluation synthesis, and meta-analysis. As governments, non-profits and other funders face fiscal constraints, the question of "what works" is one of importance. Systematic reviews in the contexts of health, education, justice, and social welfare are being conducted by agencies and organizations concerned with providing information to policy makers and practitioners to address important social problems. This paper will address the strength and limitations of assessing study quality in the context of such evidence based grading systems. It will also contrast such use with metaevaluations conducted for other purposes, including serving as a standard of evaluation practice.

Session Title: Country Context in Evaluation: Palestine, India, and the Middle East

Multipaper Session 717 to be held in Sebastian Section L1 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Denis Jobin,  National Crime Prevention Centre, denis_jobin@yahoo.ca

Discussant(s):

Denis Jobin,  National Crime Prevention Centre, denis_jobin@yahoo.ca

The Field of Evaluation in India: The Current Situation, the Changing Opportunities, and Why Context Matters Presenter(s): Katherine Hay, International Development Research Centre, khay@idrc.org.in Abstract: Anecdotal evidence in India suggests a gradual decline in demand for development evaluation, weaknesses of public sector evaluation institutions, poor (and reducing) quality of evaluations, a trend towards line Ministry run evaluations (with arguably limited evaluation use and uptake), and an increase in private/civil society evaluation. Despite discussion of the declining state of evaluation in various fora, there is little (if any) documented evidence of the state of evaluation in India, or of the factors underlying the current situation. Drawing from roundtable discussions, focus groups, and commenting on initial results from surveys a meta-evaluation on evaluation quality and the available literature, this paper will synthesize and analyze key aspects of the supply and demand of development evaluation in India. The paper will look at issues of evaluation capacity, evaluation demand and supply, and evaluation quality. The paper is intended to inform the work of various State and non-State stakeholders currently involved in strengthening the field of evaluation in India.

Employees' Performance Assessment Systems in the Middle East: Moving Beyond Western Logics Presenter(s): Antonio Giangreco, IESEG School of Management, a.giangreco@ieseg.fr Andrea Carugati, Aarhus School of Business, andreac@asb.dk Antonio Sebastiano, Universitŕ Carlo Cattaneo LIUC, asebastiano@liuc.it Abstract: This paper critically examines the background conditions that define the success or failure of performance appraisal systems (PASs) valid in Western contexts to extend their use in the Middle-East (ME). We present a view point based on a systematic literature review focused on the logics used to justify and to use PASs. We have analysed 52 articles which allow us to identify five guiding logics behind PAS. Examining the logics behind PAS in the ME, this research uncovers key elements that determine the value of PAS and that are usually backstaged in favour of more operational issues. These elements allow us to pose key questions for future PAS research in the ME and more in general in non-Western contexts. The main implication for practitioners intending to implement PAS in MEern contexts is that they should find their own motivation for doing it and be mindful of the contextual emergent usage logics.

Session Title: Quality Global Partnership Evaluations Depend on Good Planning and Governance Practices: Findings of a Review of 60 Evaluations

Panel Session 718 to be held in Sebastian Section L2 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Cheryl Gray, World Bank, cgray@worldbank.org

Abstract: The number of Global and Regional Partnership Programs (GRPPs) tackling issues from environmental preservation to communicable diseases has grown exponentially during the 1990s and 2000s, and the value of periodic evaluation is recognized. But the decision-makers in these multi-stakeholder programs often miss opportunities to lay the groundwork for credible evaluations by failing to address early on: M & E policy and framework; oversight roles; team selection; quality assurance; budgeting. The World Bank's Independent Evaluation Group (IEG) was asked by international evaluation networks to survey enabling conditions and planning processes, and the analytical tools used, in a sample of over 60 GRPP evaluations, to identify good practices. These are being published in a "Guidebook" which will complement the work published in IEG's earlier "Sourcebook for Evaluating GRPPs". This AEA panel, one of two which will highlight the findings of that survey work, focuses on enabling conditions and good planning processes.

Building an Enabling Environment for Quality Monitoring and Evaluation in Global Partnership Programs (GRPPs) Aaron Zazueta, Global Environment Facility, azazueta@thegef.org Dale Hill, World Bank, dhill@worldbank.org Elaine Wee-Ling Ooi, World Bank, eooi@worldbank.org Ideally, the GRPP founders are intentional about results-based management, accountability to stakeholders, and continued learning, at program inception. This means looking ahead and laying the groundwork to increase the probability of credible, useful evaluations later on. Examples of good practice will be given where: the GRPP charter specifies periodic independent evaluation; the governance framework anticipates the roles of various actors in monitoring, reporting and evaluation functions, and provides for sound budgeting decisions; and a formal policy elaborates the principles, roles and procedures for M& E, considering both activity- and program-level needs. In some cases, GRPPs adopt host agency policies and processes for team selection methods, evaluation ethics, dissemination and translation of products, or use of standard reporting systems. IEG found a particular lack in GRPP monitoring frameworks and systems, which complicated, and sometimes compromised quality of later evaluations. Good practices vary by program by necessity, but have common lessons.

Time for an Evaluation: Good Practice Planning Tips for Decision-makers in Global and Regional Partnership Programs (GRPPs) Dale Hill, World Bank, dhill@worldbank.org Elaine Wee-Ling Ooi, World Bank, eooi@worldbank.org A key constituent, or program policy, calls for an evaluation in the future. What are needed steps to help ensure it will be high-quality, independent, useful to stakeholders, and acted upon? A good-practice first step is a situational analysis. Is the timing right? Are there past evaluations? What is the state of the monitoring system and available data? Is the budget fixed or flexible? What policies and principles exist for guidance? Different GRPP case examples will be presented. After those questions are answered, the next step is setting the purpose, timetable and budget for the evaluation, and the process for team selection. Also, ideally, processes for evaluation review, dissemination, and response to recommendations will be planned in advance. Often a key consideration is the availability of evaluation expertise and experience among the decision-makers. Different approaches that GRPP officials have taken will be presented to show both good and second-best practice.

Ensuring Credibility and Usefulness: Good Governance in Overseeing Evaluations of Global and Regional Partnership Programs (GRPPs) Chris Gerrard, World Bank, cgerrard1@worldbank.org Dale Hill, World Bank, dhill@worldbank.org Elaine Wee-Ling Ooi, World Bank, eooi@worldbank.org Credibility of all evaluations depends on both design and implementation. The evaluation must meet standards of rigor and independence, while following program policies on transparency and stakeholder involvement. To ensure independence, ideally, the evaluation will have been commissioned by the Governing Body, and the roles of the Governing Body, subcommittees, host agency officials, and management defined in advance. The design will have also been set with appropriate expert and stakeholder input, considering the need to collect new data, possibly at multiple levels, with ground-truthing and triangulation. The characteristics of GRPPs, with their joint decision-making, multi-layered results chain, emphasis on showing good results to attract funding, and varying donor rules, can make both design and oversight complex. Mid-term corrections are sometimes needed. Finally, the ultimate usefulness may depend on dissemination, accessibility of evaluation products, and the process of considering recommendations. Both good practice and second-best examples will be presented.

Session Title: Constructing A Home for Evaluation by Building Capacity One Step at a Time

Panel Session 719 to be held in Sebastian Section L3 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

Chair(s):

Michele Mercier, Centers for Disease Control and Prevention, zaf5@cdc.gov

Discussant(s):

Hallie Preskill, FSG Social Impact Advisors, hallie.preskill@fsg-impact.org

Abstract: Building capacity for evaluation in federally funded programs that engage a complex network of actors from the private, public, and nonprofit sectors poses multiple challenges as well as exciting opportunities. In this panel, speakers will draw upon former and current experiences of the Center for Disease Control and Prevention's National Asthma Control Program to explore strategies for developing evaluation capacity in social programs that engage a variety of actors to accomplish shared goals. Panelists will share details about this real-world case using a recent model of evaluation capacity building developed by Preskill and Boyle (2008) as a guiding framework. The entire continuum of evaluation capacity building within the National Asthma Control Program will be addressed including the history of garnering support for evaluation, initial collaborative efforts that were used to begin building evaluation capacity among actors, and ideas for sustaining evaluation capacity as well as expanding initial capacity building efforts.

Laying the Foundation: Garnering Support for Evaluation Leslie Fierro, Centers for Disease Control and Prevention, let6@cdc.gov Michele Mercier, Centers for Disease Control and Prevention, zaf5@cdc.gov One of the first steps in building evaluation capacity is establishing a climate that is supportive of, or at least intrigued by, evaluation itself. To facilitate the development of an environment that is ripe for generating the support needed to build evaluation capacity, our experiences indicate that good work on the part of a small group of "advocates" for evaluation, periodic experiences that point to the importance and value of evaluative information, and fortuitous timing are needed. In this presentation, the genesis of an evaluation team within the National Asthma Control Program at the Centers for Disease Control and Prevention will be shared. Significant milestones for garnering support for evaluation within this program will be explained as well as subsequent efforts to continue fostering the understanding and importance of evaluative information.

Constructing the Floor From Which to Build Future Evaluation Capacity Building Activities Linda Winges, Battelle Centers for Public Health Research and Evaluation, winges@battelle.org Shyanika Wijesinha Rose, Battelle Centers for Public Health Research and Evaluation, rosesw@battelle.org After garnering support for evaluation activities, initial steps to build evaluation capacity can be undertaken. In this presentation, we describe the process used by the National Asthma Control program at the CDC to model the first four steps in the CDC Evaluation Framework as they initiated efforts to develop evaluation tools for use in state asthma programs. Stakeholder input was gathered through a highly interactive workgroup process and was critical for describing the program, grounding program theories in reality, prioritizing evaluation questions, and generating a data collection instrument for use in program monitoring. Products from this effort provide a starting point for states to build evaluation capacity and provide a consistent base upon which ongoing evaluation activities can be done within the National Asthma Program. Additionally, these efforts appear to have contributed to changes in participant's understanding and valuing of evaluative information.

Adding the Second Story: Building Evaluation Capacity in the Field Carlyn Orians, Battelle Centers for Public Health Research and Evaluation, orians@battelle.org Joanne Abed, Battelle Centers for Public Health Research and Evaluation, abedj@batelle.org To make a long-term positive impact on evaluation practice within state asthma programs, building the knowledge base and tools to support effective evaluations is necessary but not sufficient. This knowledge must be transferred in a systematic fashion through communication, training, technical assistance and other strategies to support evaluation capacity building among those who are tasked with carrying out the programs. The presenter will describe current efforts underway in the National Asthma Control Program to develop an evaluation guidance handbook that will support both self learning and directed training among state asthma programs, building on the knowledge base described by the previous panelists.

How Do You Keep What You Have Built: Sustaining Evaluation Capacity Maureen Wilce, Centers for Disease Control and Prevention, muw9@cdc.gov Sheri Disler, Centers for Disease Control and Prevention, sjd9@cdc.gov The National Asthma Program is undertaking a collaborative and comprehensive process to build evaluation capacity. Complementing the processes for creating capacity are efforts to maintain a culture of inquiry and sustain capacity to conduct evaluations and use data effectively over time. These efforts include reinforcing the focus on a shared, clear vision for evaluation; ensuring alignment of evaluation activities with program priorities; supporting engagement of key stakeholders throughout the evaluation process; routinely enhancing appropriate knowledge and skills for evaluation through technical assistance; and maintaining consistent resources and expectations for roles for evaluators at all levels. The presenters will show how various tools, including the management information system, training materials, and procedural guides have been designed to promote institutionalization of systems and structures supporting evaluation practice and use of findings. The presenters will also discuss strategies to tailor and implement these tools and share indicators used to monitor capacity over time.

Session Title: Evaluation Judgment: Context, Credibility and Communication

Multipaper Session 720 to be held in Sebastian Section L4 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Theories of Evaluation TIG

Chair(s):

James Griffith,  Claremont Graduate University, james.griffith@cgu.edu

'Credible' Judgment in Program Evaluation Presenter(s): Marthe Hurteau, Université du Québec ŕ Montréal, hurteau.marthe@uqam.ca Sylvain Houle, Université du Québec ŕ Montréal, houle.sylvain@uqam.ca Abstract: Through the scientific process, researchers generate findings which must convince peers of their validity, reliability and acceptability (Ginnell, 2009).Transposed it to the context of program evaluation, would not the process imply that evaluators provide evidence and generate judgments that convince the stakeholders? The present research aims at exploring, depending on the context, what constitutes a credible judgment and what are the conditions required to reach it, In order to do so, in-depth interviews were conducted with 6 subjects (3 experienced evaluators and 3 "evaluation clients"). After transcribing the interviews, the QDA Miner was used to encode and analyze the produced data. Among many things, findings raises the absence of modalities to mix the information in an adequate manner in order to generate a solid argumentation that will eventually support an evaluative judgment. Impacts of the study on the evaluative practice, as well as prospects for further research will be presented.

Evaluation Inquiry and the Discourse of Practice Presenter(s): William Rickards, Alverno College, william.rickards@alverno.edu Abstract: The role of communicating results and reporting to stakeholders holds a significant place across a range of evaluation's theoretical perspectives and approaches, from experimental designs to deliberative democratic, participatory, and mixed methods to program theories, just as in AEA's Guiding Principles. Given this, it is critical to understand how individual communities of users and stakeholders interact in relation to evaluation evidence. The structures and norms of these conversations constitute a discourse of practice with potential to shape evaluation theories, approaches, data analysis, reporting and meaning-making processes. This discourse can also become a medium for understanding the effects of interventions in select communities (e.g., how do complex interventions affect the conversations through which practitioners, such as teachers or probation officers, accomplish their work?). This presentation examines the discourse of practice as a key consideration in conceptualizing evaluations, designing processes for taking up evaluation findings, and ethical considerations for evaluators.

Session Title: Challenges to Extension Organizations and Collaboration in Evaluation

Multipaper Session 721 to be held in Suwannee 11 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Extension Education Evaluation TIG

Chair(s):

Mary Crave,  University of Wisconsin-Extension, crave@conted.uwex.edu

Extension Evaluation Research Collaborations: Will They Kill You or Make You Stronger? Presenter(s): Marc Braverman, Oregon State University, marc.braverman@oregonstate.edu Abstract: Evaluation research collaborations between Extension teams and non-Extension researchers provide many benefits and are becoming more common. These partnerships may involve theory building, new program development, or delivery of completed programs. Besides the benefits, however, there are characteristic challenges in making the collaborations successful, stemming from differences in the partners' expectations, goals, and reasons for collaborating. The role of Extension evaluators can be pivotal in ensuring projects' productivity. This presentation will identify issues that Extension evaluators should consider when they are involved in building research partnerships, whatever the level of research expertise within the Extension team. The presentation will identify critical topics, provide illustrative examples from previous experience, and offer recommendations for practice. Examples of the major topics include structuring the project budget, making decisions about scholarly products, negotiating evaluation rigor, formulating roles for Extension county-based faculty, and determining timelines for disseminating both the evaluation results and the program itself.

Framework for Building Evaluation Capacity of Extension Personnel: Challenges and Alternatives Presenter(s): Koralalage Jayaratne, North Carolina State University, jay_jayaratne@ncsu.edu Abstract: Building evaluation capacity of Extension educators is necessary for enabling them to conduct quality evaluations and document program results. Extension evaluators are responsible for building evaluation capacity in Cooperative Extension. This paper presentation discusses challenges, alternatives and a framework for building evaluation capacity in Cooperative Extension. This paper is based on the information gathered from literature and the author's experience in helping build evaluation capacity of Cooperative Extension Services in two states. The framework presented in this paper has direct implications for Cooperative Extension and other nonprofit organizations. This is a timely topic for evaluators who face the reality of building evaluation capacity in their organizations. It contributes to the evaluation practice by laying out a framework for building organizational evaluation capacity.

Optimizing Conditions for Success: An Extension Case Study in Cross-Program Surveys Presenter(s): Gwen Willems, Willems and Associates, wille002@umn.edu Abstract: Surveying is a highly popular method that has been used for decades to gather evaluative data. The advantages are numerous: it's a relatively low-cost and straight-forward way to obtain data from many people in a short period of time, as described by Gary Henry. One of the difficulties is standardizing surveys across a variety of programs. Much of the literature and discussion of surveys gives attention to survey design, error reduction, the audience, and sampling of respondents. This presentation will step back, instead doing a meta-analysis that focuses on a process and environment that led eventually to successful design and adoption of cross-program end-of-educational-session and follow-up surveys for a section of the University of Minnesota Extension. The presenter will describe this case study, the multi-activity process she used with Extension educators, challenges in that environment, and factors that contributed to success of the cross-program survey initiative.

Session Title: A Comprehensive State-wide Model for Evaluating Tier 1/Universal Response to Intervention for Behavior

Demonstration Session 722 to be held in Suwannee 12 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Karen Childs, University of South Florida, childs@fmhi.usf.edu

Don Kincaid, University of South Florida, kincaid@fmhi.usf.edu

Heather George, University of South Florida, hgeorge@fmhi.uf.edu

Abstract: This session will describe a comprehensive state-wide model for evaluating Tier 1/Universal Response to Intervention for Behavior (Positive Behavior Support) that seeks to answer critical questions about building a scalable and sustainable PBS system. Participants will receive a practical portrayal of an evaluation system that provides data for decision-making at the state level, and expands to include decision-making at the school and district levels as well. A description of Florida's state-wide evaluation system will be provided. This model, driven by 12 questions includes topics of implementation fidelity as both a dependent and independent variable and expands the traditional questions of statewide evaluations to include consumer satisfaction, team processes, barriers and facilitators to implementation, and attrition, Participants will receive comprehensive information about web-based data collection procedures, instruments/tools, data analysis and training.

Session Title: Embedded Evaluation in Out of School Programs

Multipaper Session 723 to be held in Suwannee 13 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Patricia Jessup, InSites, pjessup@insites.org

Discussant(s):

Alyssa Na'im, Education Development Center Inc, anaim@edc.org

Abstract: Evaluation is embedded in each of the National Science Foundation funded programs discussed in this session in a different way with each program situated in a different out of school context. The first paper focuses on embedding evaluation in the program using a particular evaluation tool - concept mapping' with both teachers and students. The second paper focuses on embedding evaluation in a culturally responsive way that benefits both students and program leaders. The third paper looks at embedding evaluation in the after school community and education infrastructures to help sustain the program in both a school district and a community organization. How to integrate evaluation into programs and their contexts, embed evaluation, in ways that enhance the value of evaluation is an important issue in the formal K-12 education system as well. These papers provide important ways to strengthen evaluation in both formal and informal education settings.

Increasing Participant Investment in the Evaluation Process Through Concept Maps

Rachel Becker-Klein, PEER Associates, rachel@peerassociates.net

Andrew Powers, PEER Associates, andrew@peerassociates.net

The Community Science Investigators (CSI) program employs a professional development model of program delivery. CSI staff conduct workshops and ongoing professional development with classroom educators, in the areas of Geographic Information Systems (GIS), Augmented Reality (AR) and Service-learning (SL). In turn, the educators use their newfound knowledge in these three areas to structure an out-of-school time program for middle-school students that engages students in learning about science, technology, and nature. During the professional development training, CSI staff have educators fill out concept maps before and after learning about GIS, AR, and SL, in order to assess educators' knowledge in these areas. The educators' concept maps fulfill the purposes of: 1) assessing changes in educators' content knowledge; 2) increasing educators' investment in the evaluation process; and 3) demonstrating the value of using concept mapping to assess students' content knowledge. CSI staff expect that educators will have students complete concept maps over the course of the year.

Culturally-responsive Evaluation as Reciprocal Practice: How Evaluation Can Give-Back to Participants

Phyllis Ault, Northwest Regional Educational Laboratory, aultp@nwrel.org

This presentation examines implications of using culturally-responsive and transformative approaches to increase our understanding of how evaluation can contribute to improving outcomes for underrepresented students. Increasingly, evaluators recognize that cultural understanding is needed to derive meaning from student performance and measure changes over time. Yet cultural competence is arguably a minimum expectation of responsiveness to community norms and values. In projects such as those with American Indian students, evaluators should also be attentive to ways in which evaluation "gives-back" to students, enhancing their understanding or skills, and building on successes. This reciprocity in evaluation seeks to find methodologies enabling project leaders and evaluators to gauge the influence of a given project while simultaneously helping participants grow. Using the example of Salmon Camp Research Teams funded by the National Science Foundation, the presentation will discuss methodologies for data collection that have positively influenced participating students (all of whom have American Indian affiliation).

Using Evaluation to Encourage Sustainability of Out-of-School Programs

Patricia Jessup, InSites, pjessup@insites.org

Tirupalavanam Ganesh, Arizona State University, tganesh@asu.edu

Beverly Parsons, InSites, bparsons@insites.org

The three-year Learning through Engineering Design and Practice program (funded by the National Science Foundation) offers seventh and eighth grade students informal education experiences. The program, located in Arizona, includes an after-school component where participants learn engineering and information technology skills through activities such as simulating desert tortoise behaviors, researching and developing designs to mitigate the urban heat island, and designing autonomous rovers capable of navigating Mars-like terrain. They also participate in leadership development activities including summer activities serving as docents for younger children at the local science center and internships with a local water conservation project. The paper discusses the evaluation approaches used to investigate what students are learning, the differences in curriculum for out-of-school versus in-school programs, and issues of sustainability of the work. The evaluation blends external and embedded evaluation and is grounded in supportive relationships between the evaluators and the program director.

Session Title: Using District and State Administrative Data to Evaluate Education Policies and Practices

Panel Session 724 to be held in Suwannee 14 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Kellie Kim, Academy for Educational Development, kkim@aed.org

Discussant(s):

Jeff Sellers, Florida State Department of Education,

Abstract: Panelists will discuss their experiences using administrative data to evaluate district and state policies and practices in elementary, secondary, and postsecondary education. Each presenter has used district- or state-maintained student records to investigate critical education issues, such as developing behavioral indicators of at risk status (i.e., "early warning signals"), secondary education reforms, and predictors of STEM degree attainment at the postsecondary level. The presentations will describe the benefits and challenges of using student records to evaluation policy and practice.

Creating a High School Ready Indicator Using District of Columbia Public School Data Becky Smerdon, Academy for Educational Development, bsmerdon@aed.org Aimee Evan, Academy for Educational Development, aevan@aed.org The panelist will discuss a 12-month collaborative project with the District of Columbia to use D.C. Public Schools' administrative data to develop a clear understanding of the roots of the dropout problem in the District. Because the drop out problem is most pronounced at the ninth grade level, the project focused on the years just prior to ninth grade, middle school, and the transition from middle to high school. Using extant, student-level longitudinal data, this collaborative effort aimed to provide a definition of "high school readiness" that can be used to identify at-risk middle school students and DCPS middle schools for targeted interventions. Designed to serve as a model for how districts (and research organizations) can build research partnerships and employ data-based decision-making and strategic planning across all levels of the system, results from this project provide important lessons about the strengths and limitations of using administrative data.

Evaluating High School Reform Using Baltimore City Public Schools Data Jennifer Cohen, New America Foundation, cohenj@newamerica.net The researchers used student-level administrative data from the Baltimore City Public School System (BCPSS) to assess the impact of a five year high school reform initiative on student sorting, mobility between schools, achievement and school climate. The study's outcomes suggest that the district's new "innovation" schools outperformed other non-selective schools while attracting a slightly less disadvantaged student population. Data were primarily obtained from BCPSS and manipulated for research purposes. Challenges include navigating school enrollment as schools opened, closed, or broke apart, and identifying school membership and attendance in a highly mobile population. Data used include demographics (race, gender, special education status, English language learner status, age), previous achievement (8th grade scores in mathematics and reading), attendance rate, and achievement on English and Algebra standardized tests. Methods employed include descriptive statistics, OLS regression, and hierarchical linear modeling. The advantages and limitations of using BCPSS data will be highlighted.

Career Academies in Florida: How Many, What Type, and Who Enrolls? Kathryn Borman, University of South Florida, kborman@cas.usf.edu Bridget Cotner, University of South Florida, bcotner@cas.usf.ed Corinne Alfeld, Academy for Educational Development, calfeld@aed.org Robert Bozick, Academy for Educational Development, rbozick@aed.org Reggie Lee, University of South Florida, rlee@cas.usf.org This National Science Foundation funded study begins by mapping the current landscape of career academies in Florida. Specifically, this study describes what type and how many career academies are available to high school students in Florida and the characteristics of students who enroll in career academies. This information will help decision makers in Florida and elsewhere to determine the extent to which career academies focus on the high-need industries (e.g., STEM) and enroll students who are historically underserved. This study will also include a coursetaking analysis to provide insights into the extent to which core subject areas and career-oriented content are integrated and the balance of coursetaking in core subject areas and career-oriented topics in career academies and other school settings. Use of Florida's student-level, longitudinal data to address the evaluation questions will be discussed.

Studying the Effects of Early College High Schools Using North Carolina Data Nina Arshavsky, University of North Carolina at Greensboro, narshavs@serve.org The presenter will discuss using NC Extant Data to Determine the Efficacy of the Learn and Earn Early College High School Model. Schools participating in this experimental study use a lottery to select their students. The study then links these students to extant data already collected by the North Carolina Department of Public Instruction. Results are compared students who were randomly selected to attend and those who were randomly selected not to attend on the following outcomes: student attendance, achievement, coursetaking patterns, school leaving and dropout status, and disciplinary outcomes. Original survey data on attitudes and school experiences are also collected and linked to the longitudinal North Carolina data, which are housed at the North Carolina Education Research Data Center.

Session Title: Quasi-experimental Designs: Studies in PreK-12 Evaluation

Multipaper Session 725 to be held in Suwannee 15 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Pre-K - 12 Educational Evaluation TIG and the Quantitative Methods: Theory and Design TIG

Chair(s):

Tom McKlin,  The Findings Group, tom.mcklin@gmail.com

Evaluating the Effects of Arts-based Instructional Strategies on Student Achievement and Character in Grades K-8 Presenter(s): Melinda Mollette, North Carolina State University, melinda_mollette@ncsu.edu Richard Benjamin, Pioneer Regional Educational Service Agency, rbenjamin@pioneerresa.org Mary Lasris, Pioneer Regional Educational Service Agency, mlasris@pioneerresa.org Abstract: This paper presents results from years 1-3 of the School Transformation: Character through the Arts project being implemented with over 1800 students in grades K-8 at three north Georgia schools. The project framework uses the Artful LearningG™ model, developed at the Leonard Bernstein Center/Grammy Foundation, as well as the eleven principles of effective character education, developed by the Character Education Partnership, to increase the leadership capacity of teachers, and expand instructional creativity and effectiveness, and student engagement. Additionally, this project is explicitly designed to integrate proven character education strategies with the arts as a vehicle for promoting character growth along with academic growth. This evaluation utilizes a quasi-experimental longitudinal design, involving pre and post-testing in both project and matched comparison schools. The implementation process, along with results from formative and summative instruments, such as surveys, classroom observations and achievement tests, will be discussed, along with plans for future evaluation measures.

Applications of Quasi-Experimental Research Designs: An Innovative Approach to Increasing Rigor in Evaluations Presenter(s): Elena Kirtcheva, Research for Better Schools, kirtcheva@rbs.org Abstract: This paper presents an innovative use of quasi-experimental research designs and social scientific methodology to rigorously evaluate the impact of a professional development program on the gains in content knowledge of elementary and middle school science teachers. It addresses the pronounced need in the field of content knowledge assessment for methods that enable efficient controlling for spurious relationships without increasing the cost of evaluations. The author uses data from a professional development intervention to illustrate the use of the Non-Equivalent Dependent Variable (NEDV) and Non-Equivalent Groups (NEG) research designs in evaluation.

Integrating Quasi-Experimental Design and Case Study to Address Diverse Stakeholders: An Evaluation Design for a Teaching American History project Presenter(s): Leanne Kallemeyn, Loyola University Chicago, lkallemeyn@luc.edu Crystal Johnson, Chicago Metro History Education Center, cjohnson@chicagohistoryfair.org Abstract: Evaluations of U.S. Department of Education Teaching American History Grantees are required to utilize experimental or quasi-experimental designs, as wells as report on specified Government Performance and Reporting Act (GPRA) indicators. The purpose of this paper is to describe the evaluation design of a Teaching American History grant, which was a case study with an embedded time series, non-equivalent control group quasi-experimental design that utilized the school district's existing criterion-referenced U.S. History Test to assess student outcomes. I will discuss how this evaluation design met federal requirements for a quasi-experimental design and GPRA indicators, while also being responsive to local stakeholder information needs. Finally, I will critically examine the strengths and weaknesses of this approach based on the first year of the evaluation.

Session Title: Facets of Context: Place, Profession, and Program Stage

Multipaper Session 726 to be held in Suwannee 16 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Non-profit and Foundations Evaluation TIG

Chair(s):

Linda Schrader,  Florida State University, schrader@coe.fsu.edu

Impact of Context on the Choice of Evaluation Strategies in the Ongoing Evaluation Process of the Pradler Program for Social Change Presenter(s): Chen Lifshitz, Branco Weiss Institute, chenl@brancoweiss.org.il Pamela Deutsch, The Pradler Program, pamdeu@gmail.com Abstract: The Pradler Program, established by the Pratt Foundation, assists in building the capacity of Israeli NGOs in the field of resource development. Over the 5 years of its existence, the Program initiated evaluation processes, internal and external, at 3 critical junctures of its development. At each juncture, the type of evaluation utilized took into account the internal and external contexts in which the Program operates. This included the social and political context, i.e. social change organizations in Israel, and the ongoing development of the Program taking into account the needs of shareholders, staff, and clients. At each stage, the evaluation examined different issues, using differing tools and sources of information. Adapting an appropriate evaluation strategy at each juncture ensured the relevance of the findings at two levels; the Program's impact on social change in Israel and strengthening the use of the findings in enhancing the effectiveness of the Program.

Lawyer Discipline and Deterrence: Are American Bar Association (ABA) Sanctions Effective? Presenter(s): Charles Gray, University of Saint Thomas, cmgray@stthomas.edu Abstract: Informational asymmetries in the market in legal services can give rise to a moral hazard problem that potentially threatens market viability. The American Bar Association, a 501(c)(6) nonprofit association, has implemented a discipline system intended to deter unethical behavior and maintain consumer faith in the legal services market. This paper applies the economic model of crime and punishment to the issue of lawyer discipline in an effort to evaluate the effectiveness of the ABA's system. The fundamental questions are: -Is lawyer discipline justified by a market failure? -Does the probability of sanctions imposed under the existing state-by-state systems of lawyer discipline deter unethical, illegal, and other inappropriate behaviors by lawyers? -Does the severity of sanctions likewise have a deterrent effect? The empirical results are consistent with the theoretical expectation that punishment deters and support the efficacy of the ABA's program.

Community Information: Defining Context for the Evaluation, Non-Profit and Public Sector Communities Presenter(s): Charles Naumer, University of Washington, naumer@u.washington.edu Lisa Montaqu, Piton Foundation, lmontagu@piton.org Abstract: The role of community information resources to support the definition of context is discussed in terms of its role in supporting the work of evaluators, nonprofits, public sector agencies and foundations. The results of research conducted in Denver, Colorado will be discussed in terms of the role of community information in defining context, the role this information plays in shaping evaluation questions, how evaluation results are utilized and framed in terms of the defined context, and which stakeholders might be incorporated into evaluation processes as a result of a better understanding of context. Additionally, future directions and promise for better developed community information resources to inform the work of those in the social sector will be discussed.

Session Title: Use of Context in Developing and Implementing a 10-year-old Collegiate Leadership Studies Program

Panel Session 727 to be held in Suwannee 17 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Assessment in Higher Education TIG

Chair(s):

Valerie K York, Kansas State University, vyork@ksu.edu

Abstract: Leadership Studies and Programs (LSP) at Kansas State University includes undergraduate academic coursework, a value-centered learning community, and an array of programs that include civic leadership and service-learning activities. In 2007, a decade after LSP's founding, its leadership decided that program evaluation should be undertaken. The Office of Educational Innovation and Evaluation took an evaluation approach that was multidimensional and utilized a variety of qualitative and quantitative methods. The methodology involved two major components: 1) Program history and characteristics, including review of archival data, interviews and content analysis, stakeholder surveys, and comparison and validity studies; and 2) Impact studies, including a short-term impact study of current LSP students, and a long-term impact study of a decade of program graduates. This evaluation of LSP provided a dynamic retrospective study of the first 10 years of the program and essential data for staff to use for ongoing program improvement and strategic planning.

Discovering the History and Context of a Leadership Studies Program Through Evaluation Sarah A Bradford, Kansas State University, sbradfor@ksu.edu Linda P Thurston, Kansas State University, lpt@ksu.edu Lindsay L Edwards, Kansas State University, llr2@ksu.edu Evaluators developed a historical organizational case study as part of the evaluation of Leadership Studies and Programs (LSP) at K-State. The historical case study of LSP included a document analysis and an oral history. The oral history was used to determine the perceptions of the founders about the creation of LSP, the constraints and challenges of the program, and the founders' expected outcomes and evaluation needs. The document analysis of 10 years of program materials allowed the evaluators to define critical events and put them into a historical context within the University and within the field of student leadership development. Combined into a historical organizational case study, the evaluators were able to tell the "story of LSP," thus providing a basic understanding of the program, the context and history of the program, and substantive aspects of the program for stakeholders.

Examining the Voices of Stakeholders to Inform the Evaluation Valerie K York, Kansas State University, vyork@ksu.edu Linda P Thurston, Kansas State University, lpt@ksu.edu B Jan Middendorf, Kansas State University, jmiddend@ksu.edu Jennifer E McGee, Kansas State University, jemcgee@ksu.edu Sarah A Bradford, Kansas State University, sbradfor@ksu.edu The Office of Educational Innovation and Evaluation identified expected outcomes of the Leadership Studies minor held by various program stakeholders, including faculty/staff, founders, the Advancement Council, students, and alumni. The evaluation team designed and implemented impact evaluations that: 1) assessed students' attitudes, competencies, and dispositions at various stages in their progress through the minor, and 2) examined the program's contributions to students' success in and beyond their university experiences. Expected outcomes collected from faculty, founders, the Advancement Council, and the LSP Community Student Learning Outcomes were the basis for instruments used to collect data from students and alumni. Surveys, interviews, and focus groups were used to identify the perceptions of and expected or experienced program impact on students in the four courses required for the Leadership Studies minor and on program alumni.

Utilizing Accepted Standards to Evaluate an Academic Leadership Development Program at Kansas State University Linda P Thurston, Kansas State University, lpt@ksu.edu Valerie K York, Kansas State University, vyork@ksu.edu B Jan Middendorf, Kansas State University, jmiddend@ksu.edu Evaluators from the Office of Educational Innovation and Evaluation sought and utilized appropriate standards to use as part of the evaluation of Leadership Studies and Programs (LSP) at Kansas State University. The evaluation question was: How does K-State's program measure up using College Student Leadership Development Program and W.K. Kellogg Foundation criteria? The standards were utilized to consolidate data across all 15 studies in the evaluation and to determine the extent to which the K-State program "measures up" to these national criteria. The LSP program met over half of the 73 criteria for student leadership development programs established by Chambers (1992). Of the 32 Kellogg Foundation Hallmarks (Zimmerman-Oster & Burkhart, 1999) 66.8% were affirmed. The standards allowed the evaluators to identify some outstanding qualities of the program. The evaluators made recommendations to the program related to the 31 un-met standards.

Improving and Sustaining Leadership Studies Through Evaluation Utilization Susan M Scott, Kansas State University, smscott@ksu.edu Mary Hale Tolar, Kansas State University, mtolar@ksu.edu Linda P Thurston, Kansas State University, lpt@ksu.edu Leadership Studies and Programs at K-State had existed within the College of Education for ten years when it commissioned the Office of Educational Innovation and Evaluation to conduct a program evaluation. The program was on the cusp of becoming a School of Leadership Studies, and evaluation was needed for strategic planning, documenting successes, and fundraising. One of the five purposes of the evaluation was to "provide accurate and valid data to stakeholders for various purposes, such as strategic planning." The evaluation results have been utilized in several ways. For example, the faculty have implemented strategies to examine important aspects of their curriculum and make substantial improvements. Information about perceptions of expected student outcomes has allowed the program to address the type and content of information they provide about the program. The impact evaluation and the historical case study have been essential in sustaining the program in times of budget constraints.

Session Title: Collaborative Evaluations

Skill-Building Workshop 728 to be held in Suwannee 18 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

Liliana Rodriguez-Campos, University of South Florida, lrodriguez@coedu.usf.edu

Abstract: Do you want to engage and succeed in collaborative evaluations? Using clear and simple language, the presenter will outline key concepts and effective tools/methods to help master the mechanics of collaboration in the evaluation environment. Building on a theoretical grounding, you will explore how to apply the Model for Collaborative Evaluations (MCE) to real-life evaluations, with a special emphasis on those factors that facilitate and inhibit stakeholders' participation. Using discussion, demonstration, hands-on exercises and small group work, each section deals with fundamental factors contributing to the six model components that must be mastered in order to succeed in collaborations. You will gain a deep understanding of how to develop collaborative relationships in the evaluation context.

Session Title: Evaluating Changes in Knowledge, Identity and Behavior: New Approaches

Multipaper Session 729 to be held in Suwannee 19 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Human Services Evaluation TIG

Chair(s):

Todd Franke,  University of California Los Angeles, tfranke@ucla.edu

Discussant(s):

James Sass,  Independent Consultant, jimsass@earthlink.net

Identification of Child Maltreatment: Public Child Welfare Worker Training Evaluation Outcomes Presenter(s): Chris Lee, University of California Berkeley, clee07@berkeley.edu Maria Hernandez, University of California Berkeley, hernandezm@berkeley.edu Abstract: To increase the knowledge base of public child welfare workforce training evaluation, this paper presents findings from a cross-sectional data analysis of training participant characteristics and successful achievement on an embedded evaluation assessing skills in the identification of child maltreatment. Trainees first underwent training on child maltreatment identification, divided into two parts: one on identifying neglect, physical abuse and emotional abuse (CMI-1), and the other on the identification of sexual abuse and exploitation (CMI-2). Through the use of logistic regression we explored the relationship between public child welfare trainee characteristics and skill acquisition, defined by a binary dependent variable of passing/not passing the evaluation. Our findings suggest a relationship between certain trainee background characteristics and skill acquisition for this evaluation of child maltreatment identification.

Measuring Identity Change in Evaluation Using Identity Exploration (IDEX) Instruments Presenter(s): Elaine Hogard, University of Chester, ehogard@me.com Abstract: Many social programs aim to change not just behaviour but deeper attitudes and views of self. For example a smoking cessation proigramme aims not only to reduce smoking behaviour but to bring about a change in how recipients view themselves and what kind of identity they aspire to have. This brings into play the important but elusive concepts of self and identity. This paper explores the extent to which identity and identity change has been measured in reported program evaluations and introduces a novel theoretical formulation and measuring tool known as Identity Structure Analysis and IDEX. ISA is a synthetic theory drawing on a number of key theorists. IDEX is a framwework tool which can be tailored for use in particular situations and hence program evaluations.The tool is based on constructs and entities relevant to the area of study coupled with anchor elements related to identity in its societal and temporal context. The devising and use of the tool is described together with several case studies of its use in program evalautions.

Evaluation of National Community Coalition Initiative Presenter(s): Ayana Perkins, Georgia State University, aperkins@ikataninc.com Kimberly Leonard, The MayaTech Corporation, kjleonard@mayatech.com Wilma Wooten, National Medical Association, wjwooten@cox.net Roslyn Douglas, National Medical Association, rdouglas@nmanet.org Abstract: Multi-methods are needed to provide a comprehensive assessment of programs that promote prevention and proper management of diabetes among African Americans. A goal based evaluation conducted by the National Medical Association for its National Diabetes Education Program. The original evaluation model was broadened after initial assessment to add more qualitative methods to conduct cross site comparisons and process-oriented data. Data collection involved face-to-face interviews, site visits, telephone interviews, document reviews, and online surveys. The qualitative findings indicated that coalitions successfully adopted their prevention approach to the regional culture. Each coalition achieved at least one indicator of strength that could benefit other coalitions in the cohort. In addition, qualitative findings from site visits and interviews indicated that the transition from demonstration phase to active phase should involve the use of alternative health models that are more efficacious in promoting disease awareness and the integration of technical assistance based upon the successes documented by the six coalitions.

Exploring Community Volunteerism: Evaluating the Literacy AmeriCorps Program Presenter(s): Keith Trahan, University of Pittsburgh, kwt2@pitt.edu Yuanyuan Wang, University of Pittsburgh, yywang.crystal@gmail.com Abstract: The Greater Pittsburgh Literacy Council (GPLC) is a respected organization that serves as a nation-wide pioneer in adult education and family literacy programs. In this paper, we examine one of its programs, Literacy AmeriCorps, with an emphasis on program capacity-building through AmeriCorps volunteer contributions and program impact on volunteers' perceptions of community services. Through our evaluation survey of volunteers and agencies located in six different metropolitan areas around the US, we explore the impact of residence in the appointed service community on volunteers' willingness to continue working in community service and the impact of volunteers on program service capacity. In general, the intriguing aspect of our study lies in its focus on investigating the reciprocal relationships between Literacy AmeriCorps agencies and their volunteers. More importantly, our evaluation findings provide other community service agencies with characteristics and structures that may increase program capacity and success.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Capturing Context in a Multi-site Evaluation: An Examination of Magnet Program Impact and Implementation in Four School Districts

Roundtable Presentation 730 to be held in Suwannee 20 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG

Presenter(s):

Amy Burns, Brown University, amy_burns@brown.edu

Erika Taylor, Brown University, erika_taylor@brown.edu

David Lopez, Brown University, david_lopez@brown.edu

Deborah Collins, Brown University, d_collins@brown.edu

Abstract: The proposed roundtable seeks to share the Education Alliance's approach to the 'rigorous' evaluation of four Magnet School Assistance Program-funded districts and the nuances in district-level contexts that influenced study design. As part of the description of the approach, the Alliance will briefly describe both the statistical models and tools utilized in this process, including logic models and rubrics. The proposed presentation also identifies the benefits and challenges associated with a multi-faceted approach and discusses with the group ways in which these challenges might be addressed.

Roundtable Rotation II: Lost in Translation: Evaluation Capacity Building in a Multi-site, Multi-method Evaluation of a State-wide Collaborative Building Effort - Implications for Real-world Practice

Roundtable Presentation 730 to be held in Suwannee 20 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG

Presenter(s):

Ann Price, Community Evaluation Solutions Inc, aprice@communityevaluationsolutions.com

Steve Erickson, EMSTAR Research Inc, ericksoneval@att.net

Rebekah Hudgins, Hudgins Consulting, rhudgins@mindspring.com

Abstract: Evaluators often use the term 'evaluation capacity' as a desired goal of evaluations. But what does this term really mean for grassroots prevention program programs? How will we know that capacity has been achieved? What is the role of an evaluator in creating and sustaining evaluation capacity? This session will discuss capacity building in the real world using examples from a multi-site, state-wide collaborative program that seeks to help communities improve outcomes for children and families. The three presenters will discuss capacity building from three different perspectives: the lead state-wide evaluator, an evaluation consultant that provides technical assistance to a specified region of communities within the state, and a local evaluator who works with community collaboratives to implement state-level evaluation requirements. We will attempt to answer these questions from these very different, but related perspectives. Participants will be asked to share their successes and challenges in building evaluation capacity.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Shared Values of Formal and Systematic Evaluation in Non Western Context

Roundtable Presentation 731 to be held in Suwannee 21 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Evaluation Use TIG and the International and Cross-cultural Evaluation TIG

Presenter(s):

Julien Kouame, Western Michigan University, julienkb@hotmail.com

Abstract: Although evaluation is believed to be a useful tool for organizational strengthening (Merrill, 2005), the utilization of evaluation findings cannot be guaranteed. Why evaluation finding are not used is a concern explored in the last 20 years by numerous evaluation researchers. This study is an extension of these researches while it is investigating the shared values of a formal and systematic evaluation across institutional responsibility levels in non western context such as a country in West Africa.

Roundtable Rotation II: Contrasts and Misuses of Evaluation Theory and Practice in the Confucian-influenced Cultures

Roundtable Presentation 731 to be held in Suwannee 21 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Evaluation Use TIG and the International and Cross-cultural Evaluation TIG

Presenter(s):

Deborah M Oh, California State University Los Angeles, doh2@calstatela.edu

Chuan Chang, University of Hawaii, chachuan@gmail.com

Abstract: This paper explores the contrasts and the goodness of fit of evaluation theory and practice in the Confucian-influenced cultures, namely the ways in which the evaluation purposes and implementation in the western perspective is counter productive from the eastern perspectives. Culture and culturally competent practices, as well as cultural differences in perceptions are discussed. An Eastern based evaluation processes are reflected against resistance: evaluation taken personally, acceptance: acceptance of a program without questioning, and facilitation: wanting to but not having the means to respond. In terms of misuse, the paper characterized cultural misunderstandings of evaluation practices as issues resulting in validity and credibility rather than misuse in terms of the following: Awareness, knowledge-base, perceptions of evaluation, attitudes, Finally, the adaptability and applicability of evaluation from the eastern perspectives versus the western perspectives and the idea of accountability and perspectives on what are relevant and appropriate criteria among cultural groups are elaborated.

Session Title: Addressing Culture and Cultural Competence Within the Evaluand

Multipaper Session 732 to be held in Wekiwa 3 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Multiethnic Issues in Evaluation TIG

Chair(s):

Stella Hargett,  Morgan State University, drslhargett@gmail.com

Discussant(s):

Stella Hargett,  Morgan State University, drslhargett@gmail.com

How Do Evaluators Communicate Cultural Competence? Indications of Cultural Competence Through an Examination of the American Evaluation Association's Career Center Presenter(s): Stephanie Evergreen, Western Michigan University, stephanie.evergreen@wmich.edu Kelly Robertson, Western Michigan University, kelly.n.robertson@wmich.edu Abstract: While it is widely accepted that evaluators need to be culturally competent, how the concept is operationalized and communicated in practice is not yet well understood. Thus, the goal of this study is to create an organic definition of cultural competence within evaluation by examining the ways in which it is expressed in practice by evaluators and employers through examination of the American Evaluation Association's (AEA) Career Center. To formulate a definition, we conducted a review of all resumes and job postings available on AEA's Career Center, from January to October 2009. The evaluation-specific expressions of cultural competence were grouped across themes, producing a list of skills, responsibilities, and experiences that are currently used to indicate an otherwise intangible construct. An even larger lesson may be the professional development process through which the raters of the Career Center documents were calibrated.

A Participatory Approach to Ensure Cultural and Linguistic Competence in Survey Research: The Healthy Start Participant Survey Presenter(s): Sasigant So O'Neil, Mathematica Policy Research Inc, so'neil@mathematica-mpr.com Julia Silhan Ingels, Mathematica Policy Research Inc, jingels@mathematica-mpr.com Lisa M Trebino, Mathematica Policy Research Inc, ltrebino@mathematica-mpr.com Margo L Rosenbach, Mathematica Policy Research Inc, mrosenbach@mathematica-mpr.com Abstract: As the population of the United States becomes increasingly diverse and concerns about health disparities grow, research to understand the health needs and outcomes of disadvantaged populations is necessary. Consequently, cultural and linguistic competence is a consideration not only for health service delivery, but also health services research. A participatory research approach can help address cultural and linguistic competence needs in evaluation. In this article, we use the Healthy Start participant survey as a case study and describe participation in survey design and administration at the individual, organizational, and community levels for eight program sites; we discuss these interactions in the context of key components of cultural and linguistic competence, including awareness of diversity, knowledge of cultures, understanding of the dynamics of differences, development of cross-cultural skills, and adaptation to diversity. We conclude with a discussion of the benefits and challenges to using this participatory research approach.

The Development and Validation of the Cultural Competence of Program Evaluators (CCPE) Scale Presenter(s): Krystall Dunaway, Old Dominion University, kdunaway@odu.edu Bryan Porter, Old Dominion University, bporter@odu.edu Jennifer Morrow, University of Tennessee at Knoxville, jamorrow@utk.edu Abstract: Although most program evaluators embrace the idea that evaluation should be shrouded in cultural competence, there is currently no measure of cultural competence in existence for the field. Therefore, the goal of this study was to develop and validate a measure of cultural competence for use with program evaluators. Items from three established measures in the fields of counseling, therapy, and healthcare were selected and altered to better suit the field of program evaluation, and then these altered items were combined, along with qualitative and demographic questions, to create the Cultural Competence for Program Evaluators (CCPE) instrument. The researchers pilot tested the CCPE via purposive heterogeneity sampling. Specifically, online surveying of members of AEA and SEA was used. The reliability and validity of the CCPE was assessed, as well as differences in level of cultural competence among program evaluators based on several demographic variables. Results and implications will be discussed.

Session Title: Peace and Democracy in the Face of Corruption: Culturally-responsive and Ethical Evaluation

Multipaper Session 733 to be held in Wekiwa 4 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Patty Hill,  EnCompass LLC, phill@encompassworld.com

Discussant(s):

Patty Hill,  EnCompass LLC, phill@encompassworld.com

Peace Education and Culturally Responsive Evaluation: Emerging Theory and Practice Presenter(s): Helga Stokes, Duquesne University, stokesh@duq.edu Abstract: Peace education programs address conflicts that are rooted in a perception of otherness. Historical and ongoing injustices and inequities have led to mistrust and contention. Culture, ethnicity and diversity have become a source of conflict instead of an enrichment of societal life. Cultural responsiveness in the evaluation of such programs is therefore unquestionably a necessity. The debate is not about the need for cultural responsiveness in the evaluation of peace education programs, but rather about best practices and capacity building among evaluators. The presentation will discuss the complex nature of evaluating peace education programs, specifically programs serving populations who lived through conflict situations and strive to recover. The need for evaluators skilled in conducting such evaluations is highlighted and a pipeline program designed to increase the availability and training of culturally competent evaluators will be described. Practice examples in evaluation settings are given.

Democratization and Corruption: An Evaluation of Anticorruption Strategies in Romania Presenter(s): Mihaiela Ristei, Western Michigan University, m2ristei@wmich.edu Abstract: Once a rarely discussed issue, corruption has become one of the most debated problems in the last fifteen years, particularly in the context of democratization. For post-communist countries, corruption has represented a particular challenge, undermining their process of democratic consolidation and transition to a market economy. The literature on corruption suggests that the strategies to contain corruption should be multi-pronged and include a variety of methods that focus on de-monopolizing decision making, limiting discretion, and strengthening accountability. As a candidate country to the European Union (EU), Romania had to demonstrate that it had the political will and the capability of effectively fighting corruption. This paper will evaluate the anticorruption strategies adopted in Romania prior to the EU accession and their effectiveness in achieving the desired effects. Additionally, the paper will make policy recommendations that would strengthen the existing anticorruption strategies and suggest new avenues to fight corruption.

Evaluation in Spain: Practice and Institutionalization Presenter(s): Maria Bustelo, Complutense University at Madrid, mbustelo@cps.ucm.es Jody Fitzpatrick, University of Colorado Denver, jody.fitzpatrick@ucdenver.edu Abstract: How is evaluation culture evolving in a relatively new democratic country in Europe, such as Spain? Which kind of evaluation practice is taking place in that new scenario? How are institutions reacting, and is there an evaluation policy taking place? Which lessons might be learned from contrasting other national realities and contexts? Based on an analysis of the most recent developments in evaluation practice and institutionalization in Spain, as well as in interviews and a survey targeted to Spanish evaluation practitioners and commissioners, this paper intends to draw a picture of the evaluation field in that country. A link with the recent democratic development of the country will be established in order to explain recent developments in evaluation practice, evaluation institutions, and professional and academic development. Policy domains, governmental level, size and types of evaluations, commissioners and stakeholders; purposes, approaches, questions and methods used; and evaluation utilization will be analyzed.

Session Title: Evaluation Capacity Building: Taking Stock of and Advancing Frameworks, Strategies, Outcomes and Measurement

Panel Session 734 to be held in Wekiwa 5 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Chair(s):

Yolanda Suarez-Balcazar, University of Illinois at Chicago, ysuarez@uic.edu

Abstract: Although the literature on ECB has grown, there is little synthesis of what we have learned thus far about ECB conceptualizations, strategies, and outcomes. This panel will address these issues. The first paper by Lubin et al. describes the method and findings from a synthesis of 100 ECB empirical studies using a framework developed by Duffy et al. The second paper by Duffy et al. describes the framework as it was initially conceptualized and discusses how it has been revised based on findings from the synthesis discussed in the Labin et al presentation. The third and fourth papers describe the validation of a contextual model of ECB using two complementary approaches; in the third, Suarez et al. describe a multi-method multiple case study approach and in the fourth, Taylor-Ritzler et al. describe a large quantitative validation approach. All presenters will discuss implications for ECB theory, research and practice.

Synthesis of Evaluation Capacity Building Literature Susan N Labin, Independent Consultant, slabin@pobox.com Jennifer L Duffy, University of South Carolina, jenniferlouiseduffy@gmail.com Duncan Meyers, University of South Carolina, meyersd@mailbox.sc.edu Abraham Wandersman, University of South Carolina, wandersman@sc.edu Evaluation capacity building (ECB) is a growing area reflecting an ever increasing expectation of evidence that programs are achieving their goals. Community-based organizations often must demonstrate their ability to engage in program evaluation to obtain funding. ECB is also reflective of the shift towards participatory evaluation methods. For example, capacity building is one of the basic precepts of empowerment evaluation. This presentation focuses on the method and findings from a synthesis based on detailed coding of approximately 100 ECB empirical studies. The conceptual framework used is discussed in the Duffy, et al. presentation. The broad-based synthesis method used derives from the evaluation syntheses developed at the Government Accountability Office and includes qualitative and quantitative data from a variety of evaluation designs. The research questions focus on ECB strategies, evaluations, and outcomes, and include some contextual and implementation variables. Reporting will also include challenges encountered and lessons learned.

A Conceptual Framework for Assessing the Empirical Literature on Evaluation Capacity Building Jennifer L Duffy, University of South Carolina, jenniferlouiseduffy@gmail.com Duncan Meyers, University of South Carolina, meyersd@mailbox.sc.edu Susan N Labin, Independent Consultant, slbain@pobox.com Abraham Wandersman, University of South Carolina, wandersman@sc.edu As the practice and research of evaluation capacity building (ECB) have grown, multiple useful frameworks have been developed to highlight important aspects of the ECB process. As preparation for a synthesis of the empirical research on ECB, key frameworks and theories were reviewed to guide coding and analysis. We then constructed a unified framework by integrating concepts from theory and existing frameworks into a general logic model format, including needs leading to ECB, activities characterizing ECB, and outcomes attributed to ECB. This presentation will describe the framework as it was initially conceptualized and discuss how it has been revised based on findings from the synthesis discussed in the Labin et al presentation. Elements of the framework that are lacking empirical research will be highlighted, and implications for ECB research and practice will be discussed.

Evaluation Capacity: Model Validation Using a Mixed Method Multiple Case Study Approach Yolanda Suarez-Balcazar, University of Illinois at Chicago, ysuarez@uic.edu Tina Taylor-Ritzler, University of Illinois at Chicago, tritzler@uic.edu Edurne Garcia-Iriarte, University of Illinois at Chicago, edurne21@yahoo.com Within the ECB literature, there are currently no published examples of validated instruments or systems for assessing evaluation capacity. Together with the fourth presentation in this panel, the purpose of this presentation is to present The Evaluation Capacity Building Contextual Model and share data on the validation of the model, using an in-depth mixed method multiple case study approach. The model was developed based on work with diverse human service organizations and on reviews of evaluation, cultural competence, and capacity building literatures. To begin to address the lack of validated models of evaluation capacity in their field, our team used a multiple case study approach to validate the ECB Contextual Model using survey, interview and document review data collected from multiple stakeholders within nine organizations. In this presentation we will share the measures, analytic methods and results of the case study approach to validation.

Evaluation Capacity: Model Validation Using Quantitative Methods Tina Taylor-Ritzler, University of Illinois at Chicago, tritzler@uic.edu Yolanda Suarez-Balcazar, University of Illinois at Chicago, ysuarez@uic.edu Edurne Garcia-Iriarte, University of Illinois at Chicago, edurne21@yahoo.com In this presentation we will present the methods and results of a large quantitative validation study of our ECB Contextual Model using a survey developed by our team. We will describe the survey methods, analytic results and final model and measure with audience members. We will present quantitative analyses of the variable importance of different individual factors (i.e., awareness of the benefits of evaluation, motivation to conduct evaluation and knowledge and skills related to conducting evaluation) and organizational factors (i.e., leadership, learning climate, resources, and funders' demands) on evaluation capacity outcomes (i.e., use and mainstreaming of evaluation practices). We will conclude the presentation by comparing and contrasting the results of the case study and quantitative approaches to validation and by discussing the implications for future research and practice related to conceptualizing and measuring ECB.

Session Title: Results Frameworks: Improving Evaluation of Socio-economic Outcomes of Research

Multipaper Session 735 to be held in Wekiwa 6 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Research, Technology, and Development Evaluation TIG

Chair(s):

Steve Montague,  Performance Management Network, steve.montague@pmn.net

Stakeholder Involvement and the Choice of Science and Technology Policy Outcome Evaluation Methods Presenter(s): Shan-Shan Li, Science & Technology Policy Research and Information Center, ssli@mail.stpi.org.tw Ling-Chu Lee, Science & Technology Policy Research and Information Center, lclee@mail.stpi.org.tw Wen-Chi Hung, Science & Technology Policy Research and Information Center, wchung@mail.stpi.org.tw Kai-Lin Chi, Science & Technology Policy Research and Information Center, klchi@mail.stpi.org.tw Chia-Hao Hsu, Science & Technology Policy Research and Information Center, chiahao@mail.stpi.org.tw Abstract: This paper aims to develop the linkage between stakeholder involvement and science & technology (S&T) policy outcome evaluation methods so as to allow adoption of suitable methods for responding to stakeholders' interests and concerns. The paper adopts the conceptual framework of impacts of publicly funded S&T programs which involves four phases: allocation of public funds, outputs and outcomes of S&T communities, cooperation of research communities, and impact of research utilization. Stakeholders' concerns on various issues and impacts in phrases of publicly funded S&T programs are discussed first from the requirements (demand-side) of publicly funded executors (S&T executive institutions, cooperative institutions, the end-users utilizing outputs, etc.), and concerns (supply-side) from publicly funding sponsors (funders, etc.). Then, based on their individual unique characteristics, different S&T policy outcome evaluation methods are used to assess different stakeholders' issues. In this manner, stakeholders'interests exert influences on the choice of S&T policy outcome evaluation methods.

Socioeconomic Effects and Meta Evaluation of Tekes Programs Presenter(s): Jari Hyvarinen, Tekes, Finnish Funding Agency for Technology and Innovation, jari.hyvarinen@tekes.fi Abstract: A goal of my paper is to determine meta evaluation results of Tekes programss in Finland in order to describe those socioeconomic effects that are defined in Tekes strategic goals and focus areas. The meta evaluation work has been targeted to the Tekes programss that have been carried out during 1995-2007. Tekes programss are the most important tool of the implementation of the Tekes strategy and strategy focus areas. The socioeconomic goals in Tekes strategy are welfare effects based on the sustainable energy and environment, quality of social and health system as well as services and platforms of information society. Moreover, our impact model concentrates on the effects of public R&D financing on the whole economy and society. We try to build up a framework of longer term societal welfare effects by focusing on the results of meta evaluation, Tekes strategic goals and focus areas, and Tekes impact model.

Evaluation's Contribution to Research, Technology and Development (RT&D): From Prescription to Menu to Guidelines Presenter(s): Steve Montague, Performance Management Network, steve.montague@pmn.net Rudy Valentin, Canadian Cancer Society, rvalentin@cancer.ca Christine Williams, Canadian Cancer Society, cwilliams@cancer.ca Anne Vezina, Canadian Cancer Society, avezina@cancer.ca Abstract: Evaluative thinking has influencesd the performance measurement of RT&D initiatves for decades. Tradtionally, evaluation has offered a prescription of key indicators to determine success. Early approaches have measured inputs, outputs and productivity in terms of research publications, often without providing a direct link of the contribution of research to achievement of mission bjectives. The publication of the report "Making an Impact" by the Canadian Academy of Health has provided a multi-facetted results logic and performance measurement menu for R&D. This menu represents an improvement over previous "presciptions" in that it broadens the mix of indicators to include policy, commercial innovation and a variety of mission related impacts. The authrors suggestr that a further refinement of the menu, using a results hierarchy as a guideline, will enhance performance measurement practices as applied to R&D. Specific examples will be discussed , in the context of the Canadian Cancer Society 's research programs.

Session Title: Building Data Systems to Support Evaluation in a Biomedical Research and Development (R&D) Environment

Multipaper Session 736 to be held in Wekiwa 7 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Research, Technology, and Development Evaluation TIG

Chair(s):

James Corrigan, National Institutes of Health, corrigan@mail.nih.gov

Discussant(s):

Robin Wagner, National Institutes of Health, wagnerr2@mail.nih.gov

Abstract: Gaining access to high quality analytically-useful data constitutes a common challenge that NIH program evaluations must address. The widespread nature of this challenge presents an opportunity for useful electronic tools to have broad impact on both NIH program management and program evaluations. Tools in various stages of development are addressing the needs of both external and internal users/audiences. External resources address the need for public transparency and internal resources provide the evidence base for program management decisions. Examples of data systems relevant to evaluation activities will be presented, highlighting the evaluation and assessment needs that these tools address. External and internal systems, as well as NIH-wide and Institute-specific systems will be covered. The systems to be discussed are: 1) NIH Research Portfolio Online Reporting Tool (RePORT); 2) Electronic Scientific Portfolio Assistant (e-SPA); and 3) CareerTrac.

Data Infrastructure and Access: The National Institutes of Health Research Portfolio Online Reporting Tool (RePORT)

James Onken, National Institutes of Health, onkenj@mail.nih.gov

In this presentation, we provide an overview of the National Institutes of Health (NIH) Research Portfolio Online Reporting Tool (RePORT). Easy access to information and the lack of a data infrastructure to support complex analyses of federal research programs can be a major impediment to program evaluation, reporting, and decision support. Meaningful assessments require the integration of disparate types of information from a variety of both structured and unstructured data sources. RePORT and its Expenditures and Results system, RePORTER, are experimental efforts to integrate multiple electronic sources of data on NIH research programs and provide public access to these resources. Through this initiative, the NIH hopes to meet its goals of promoting transparency in government, communicating the nature of NIH research programs and the scientific advances they produce, and building an infrastructure to support the "science of science."

The Electronic Scientific Portfolio Assistant (e-SPA): A Tool for Analysis of Research Project Portfolios

Kevin Wright, National Institutes of Health, wrightk@mail.nih.gov

The Electronic Scientific Portfolio Assistant (e-SPA) is an innovative software tool designed to provide staff of the National Institute of Allergy and Infectious Diseases (NIAID) with the capability to analyze outputs of research portfolios and individual investigators. The development of e-SPA grew out of program needs to ensure accountability and transparency of information about program performance and results. The tool generates user-defined portfolios of NIAID-funded projects and provides links to multiple R&D performance indicators including funding, publications, citations, impact factors, inventions, and patents. In addition to being robust, e-SPA is flexible and customized to meet the needs of a particular analysis or application. Portfolios are continually refreshed based on existing search parameters, allowing a "dashboard" like view of portfolio performance. Program managers can refine indicators to show citation counts with or without self-citation. The analysis can include views of productivity for various time horizons, perhaps capturing precipitous declines or increases.

CareerTrac: An Aid to Effective Evaluation

Linda Kupfer, National Institutes of Health, kupferl@mail.nih.gov

Ramkripa Raghavan, National Institutes of Health, raghavar@mail.nih.gov

This presentation will address the role of a data tool, CareerTrac, in enhancing the ability of NIH program managers to evaluate, monitor and report on their training programs. CareerTrac is a web based application used to collect data and track trainees supported on grants from the Fogarty International Center (FIC). The primary goal of this system is to create a complete trainee roster for all FIC research training programs and to monitor and evaluate outputs, outcomes and impacts of the programs. In addition to quantitative data such as publications and grant funding, CareerTrac uniquely captures qualitative information (such as trainee's career highlights, their role in policy development, honors/awards conferred and leadership roles) that is otherwise difficult to track. Data from CareerTrac will be potentially used to develop benchmarks for FIC programs and will allow FIC to better compare programs with each other and with other global health training programs.

Session Title: Rapid Assessment Procedures: A Tool for Evaluators

Demonstration Session 737 to be held in Wekiwa 8 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Qualitative Methods TIG

Presenter(s):

Ann Dozier, University of Rochester, ann_dozier@urmc.rochester.edu

Abstract: The demonstration will begin with a description of the purpose of RAP in the context of different types of evaluation. An overview will include how RAPS are designed: defining the narrow/broad goal of a RAP, the design decisions (who to interview; question development; stakeholder involvement); implementation (training; team deployment) and analytical strategies. Examples will be drawn from the author's experience in conducting RAP in a smoking cessation project in the Dominican Republic and a cardiovascular health project in Grenada. Dozier AM, et al. 2006. Tobacco use in the Dominican Republic: Understanding the culture first. Tobacco Control. Tobacco Control 15(suppl_1): i30-i36. Dozier, A et al. 2008. Cardiovascular health in the developing world: community perceptions from Carriacou, Grenada. CVD Prevention and Control, 3(3), 123-131. Dozier AM, et al. 2009. Health Care Workers in the Dominican Republic: Self-Perceived Role in Smoking Cessation. Evaluation & the Health Professions. Prepublished on line. DOI: 10.1177/1534734608

Session Title: Addressing Methodological Challenges in Evaluation of Environmental Programs and Policies

Multipaper Session 738 to be held in Wekiwa 9 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Environmental Program Evaluation TIG

Chair(s):

Matthew Birnbaum, National Fish and Wildlife Foundation, matthew.birnbaum@nfwf.org

Abstract: Evaluation has come later to the field of environmental programs and policies than to many other areas. Some of this area's important methodological challenges have had serious impacts on the concepts and practices of evaluations in this arena. This is a nascent field of evaluation, but the problems that evaluators are addressing here have significance to those outside the environmental arena. These features include: (1) conflicting and frequently long time horizons; (2) disparities in scaling; (3) problems in data credibility; and (4) the problem of a counterfactual given the inapplicability of RCT. These issues have formed the pillars of the first volume of New Directions for Evaluation (Issue #122, June 2009 release) devoted to the Environment.

Key Insights for the Design of Environmental Evaluations

Per Mickwitz, Finnish Environmental Institute, per.mickwitz@ymparisto.fi

Improving the future quality of program and policy evaluation in the environmental arena requires addressing four issues that emerged upon reading the earlier chapters in this special volume. Framing the evaluation requires careful consideration in choosing what the focus should be, what the context is, and what comprises potentially confounding factors. It means identifying the purpose of the evaluation and the stakeholders most important for inclusion. Addressing attribution requires recognizing the limits of establishing causality while maintaining counterfactual thinking. The frequent reality where bundles of interventions are implemented simultaneously makes precise assessment of relative impacts unrealistic and increases the importance of triangulation of qualitative and quantitative approaches. More effort is needed to increase the usefulness of evaluations. Multiple stakeholders come with their own perspectives; addressing this heterogeneity requires transparency and feedback during the design, implementation and follow-up to any evaluation study. Finally, improving the future quality of environmental evaluation will require continued deliberation and community building. Much of this is already underway, but the pluralism should still be promoted and our community enlarged. Evaluators in the environmental field further should continue to be mindful of contributions coming from other sub-fields of evaluation and applied research beyond the environmental community.

Design Alternatives to Evaluating the Impacts of Conservation Projects

Richard Margoluis, Foundations of Success, richard@fosonline.org

Caroline Stern, Foundations of Success, caroline@fosonline.org

Historically, examples of project evaluation in conservation have been rare. In recent years, however, conservation professionals have begun to recognize the importance of evaluation both for accountability and for improving project interventions. Even with this growing interest in evaluation, the conservation community has paid little attention to evaluation design. Recent literature has included some discussion of design, but it has focused primarily on experimental and quasi-experimental design and the use of counterfactuals. Real-life conservation projects, however, operate in complex and dynamic contexts and under conditions of limited resources, which limit the feasibility of counterfactual or experimental designs. There is, in fact, a wide range of design options to evaluate conservation interventions. The conservation community must educate itself about these options, and how different designs produce different results. This paper discusses evaluation design alternatives in light of the unique challenges that influence evaluation design selection in conservation.

Environmental Evaluation Practices and the Issue of Scale

Hans Bruyninckx, Catholic University, hans.bruyninckx@soc.kuleuven.be

Social scientists, natural scientists nor evaluators have properly defined the concept of scale for environmental problems. Environmental scale generally differs from social scale, which confounds the challenge of evaluating policies and governance arrangements in addressing environmental issues. Instead, social scales are generally based on traditional jurisdictional boundaries, which confounds effective decision making. Conversely, evaluators must be able to assess innovative governance arrangements as well as the outcomes of environmental problems since the two are interconnected. This becomes particularly true in looking at "cross-scale, social-ecological interactions." This has profound implications for policy evaluation since evaluators have to develop frameworks for connecting across various scales/levels in overcoming mismatches. Natural scientists probably need to be more humble in their ambitions, and evaluators will need to engage in interdisciplinary teams that blend the expertise of the social sciences with that from the natural sciences to assess outcomes to social and environmental scales.

A Perspective From Systematic Reviews in Environmental Management

Andrew Pullin, University of Bangor, a.s.pullin@bangor.ac.uk

To use environmental program evaluation to increase our effectiveness, predictive power and resource allocation efficiency, we need good data. Data require sufficient credibility in terms of fitness for purpose and quality to develop the necessary evidence base. Here we examine different elements of data credibility using experience from critical appraisal of studies on environmental interventions using systematic review methodology. We argue that critical appraisal of methodological quality is a key skill to improve both retrospective evaluation and prospective planning of monitoring of environmental programs. Greater transparency and data sharing among evaluators could facilitate rapid development in approaches to environmental evaluation that improve data credibility.

Issues of Scale and Monitoring Status and Trends in Biodiversity

Elizabeth Kennedy, Conservation International, e.kennedy@conservation.org

Issues of Scale and Monitoring Status and Trends in Biodiversity Designing a framework that can generate data to meet multiple audience objectives must consider interactions across levels and multiple scales of interest. One solution is to define targets using global criteria such that units are discrete. This aids identification of the types of data to collect in order to enable aggregation and reporting across relevant scales. By striving to standardize measurable targets and data requirements, we improve our ability to format monitoring information and tailor indicators for different reporting and decision making purposes. We outline the application of a nested approach presently used by Conservation International and describe some of its advantages and limitations.

Session Title: Teaching Evaluation in a Non-evaluation Context: How to Convince Practitioners That Evaluation is Worth Doing

Panel Session 739 to be held in Wekiwa 10 on Saturday, Nov 14, 9:15 AM to 10:45 AM

Sponsored by the Teaching of Evaluation TIG

Chair(s):

Neva Nahan, Wayne State University, n.nahan@wayne.edu

Discussant(s):

Neva Nahan, Wayne State University, n.nahan@wayne.edu

Abstract: Evaluation is conducted across a broad array of contexts and situations. In order to implement evaluation, evaluators collaborate with program practitioners across this spectrum. Non evaluators are often leery about the benefit of evaluation. This poses special challenges when teaching non evaluators about evaluation. They are often an unwilling audience. But this is the group that evaluators need to win over if evaluation is to be effectively conducted and findings utilized. This panel explores teaching non evaluators about evaluation in a variety of contexts.

Teaching Nonprofit and Public Administrators Francis Schweigert, Metropolitan State University, francis.schweigert@metrostate.edu I teach nonprofit and public administrators about evaluation. None of them are expected to be able to conduct an evaluation, but it is essential that they understand how it is done, what the limits and uses are, and how they can manage internal and external evaluators (evaluations). It is great fun to teach it. After three years, I think I have something worth sharing with others, regarding how it works. And I would be delighted to hear how others do this--the sooner the better!

Teaching Medical School Faculty Rick Axelson, University of Iowa, rick-axelson@uiowa.edu The course "Educational Research and Evaluation" is for Masters in Medical Education students to build foundational knowledge and skills needed to: 1) critically evaluate educational research; and 2) design basic research and evaluation projects in students' areas of educational interest. Challenges include: 1) developing course content that is focused on medical education research to maintain student interest, but, broad enough to cover essential issues in research design and evaluation; and 2) finding methods of delivery that accommodate busy schedules of the medical faculty who are the students. Students come with a wide range of background knowledge about research and have a great variety of interest areas. The course is online except four face-to-face sessions. The assignments help students process, summarize, and critique the foundational knowledge. Written feedback was provided each week to help deepen students' understanding. This presentation will describe course successes and shortcomings, as well as planned revisions.

Teaching Undergraduate Public Administration Majors Lora Warner, University of Wisconsin at Green Bay, lwarner@new.rr.com I offer the course "Evaluating Public and Nonprofit Programs" to undergraduate public administration majors. Our goal is to equip them with the skills needed to conduct an outcome evaluation through a series of applied exercises and a culminating experience designing an outcome measurement system for a local program. We employ "clickers" as a tool to enhance engagement. At the end of the semester, students deliver logic models, indicators, data collection plans, and survey instruments to the program directors. I have experimented with a variety of models to get students working with real local agencies and developing skills while providing meaningful products for practitioners.

Teaching Community-Based-Organization Leaders With Logic Models Arthur Hernandez, Texas A&M University Corpus Christi, art.hernandez@tamucc.edu I'd like to present on teaching about and using logic modeling in CBO's for planning and evaluation. This presentation will provide a brief overview of evaluation principles and frameworks applicable to operations of Community-Based Organizations, particularly (but not limited to) those focused on the improvement of health conditions. Although there are a large number of approaches available, many are beyond the sophistication of community based organizations to conduct. To be presented is an evaluation planning approach based on Logic Modeling, which can be easily tailored, organized and conducted. Participants will be provided with the model framework, and sample tools. This discussion will be of benefit to participants interested in developing materials for program and site promotion, applications for extramural (and internal) funding, and internal and external accountability purposes.

Teaching in Extension Gary Varrella, Washington State University, gvarrella@wsu.edu In extension, the key question is "so what," referenced to a logic model. This question focuses extension faculty and staff upon outcomes. In terms that are more gracious "so what" seeks to uncover what has been learned, what behaviors/practices have changed, and how conditions have improved. "So what" is an equally valuable question to consider before and after planning/completing programs and annual plans of work. The Extension model invites planning for evaluation and setting goals a priori and formative, process, and summative evaluation; however, there are at least three roadblocks to success: 1) Time and willingness to invest in evaluation, 2) skill level and confidence in gathering/making sense of germane data, and 3) uncertainty regarding the value of non-quantitative data and what to do with it once you have it! My comments will highlight recent experiences and successes in conducting workshops and collaborating with colleagues.

Return to Evaluation 2009