Search Results for All Sessions

Return to search form

Session Title: Exploring Contextual Factors and Assumptions That Shape Evaluation

Multipaper Session 288 to be held in Panzacola Section F1 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the AEA Conference Committee

Chair(s):

Curtis Mearns, Apex Education, curt@apexeducation.org

The Political Context of Program Evaluation

Presenter(s):

Anne Markiewicz, Anne Markiewicz and Associates Pty Ltd, anne@anneconsulting.com.au

Abstract: The presentation will contend that evaluation takes place within a political context influenced by the variety of positions presented by involved stakeholders. The political and stakeholder influenced context of evaluation is a reality that brings with it many benefits for the relevance and the use of evaluation findings. However, this context also presents a number of challenges for the evaluator in preserving their independence and objectivity with potential adverse consequences for the credibility of the evaluation. Many evaluators have experienced undue influence from commissioners of evaluations or key stakeholders involved in the process. This influence can affect each stage of the evaluation process but is particularly highlighted during the formulation of evaluation findings or recommendations. It is proposed that evaluators need to negotiate such challenges effectively. A credible evaluation is necessary to enable both commissioners of evaluations to make informed decisions and evaluators to produce quality work. The knowledge, skills and experience of the evaluator in attaining a balance between the impact of political and stakeholder influences and interests and delivering credible evaluation findings is critical to the future of evaluation and its utility.

Organizational Structures and Attitudes Towards Evaluation: An Examination of Contextual Factors That Shape Evaluation

Presenter(s):

Janet Lee, University of California Los Angeles, janet.lee@ucla.edu

Anne Vo, University of California Los Angeles, annevo@ucla.edu

Noelle Rivera, University of California Los Angeles, novari320@hotmail.com

Abstract: The manner in which evaluations are conducted are often shaped by the context in which they were commissioned. Programs' organizational structures are perhaps the best sources for understanding such contextual impact. In this presentation, we will discuss how the processes and activities of an evaluation for a federally funded educational program have been influenced by differences in organizational contexts. Specifically, we will examine various successes and challenges encountered while conducting an evaluation of small learning community (SLC) implementation at five high schools in a large, urban school district. These successes and challenges of conducting the evaluation will also be discussed in light of general attitudes towards program implementation, evaluation, and decision-making. Data collected through surveys, interviews and focus groups will be used. Understanding how these contextual factors may have impacted the evaluation provides important insight that can be useful for the planning of future evaluations.

Data Based Discussions: Uncovering the Unexamined Assumptions

Presenter(s):

Micah Fierstein, University of Alaska Anchorage, micah@uaa.alaska.edu

Ed McLain, University of Alaska Anchorage, ed@uaa.alaska.edu

Abstract: Teacher and evaluator conversations too often appear to be 'loaded' and influenced by participants' silent assumptions and frames. All too often these assumptions are unconsciously held, and unexamined. They are simply taken as granted as 'the ways things are' (Argyris, C.,1974). These silent assumptions can unwittingly constrain, frame, and ultimately debilitate the teachers' and evaluators' perspectives, and the their ability to understand and appreciate the data's underlying meaning. They can limit the teachers' ability to fully appreciate the situation and opportunities at hand - the teachers' role and potential for change and to influence the future. These blind spots undermine the teacher's and team's capacity to change practice - to make a difference. This paper identifies and examines these silent assumptions, and the influence they can have on a team's or individual's data-based meaning making in program review and evaluation.

Session Title: Transforming the Context of Child Welfare Practice: The Evaluation of Family to Family

Panel Session 289 to be held in Panzacola Section F2 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Human Services Evaluation TIG

Chair(s):

Lynn Usher, University of North Carolina at Chapel Hill, lynnusher@unc.edu

Abstract: Sponsored by the Annie E. Casey Foundation, the Family to Family foster care reform initiative is being conducted in 12 large urban child welfare systems in Arizona, California, Colorado, Kentucky, New York, North Carolina, and Ohio. The anchor-site phase began in 2006 and ends in 2009. The initiative seeks to transform child welfare practice through four core strategies: building community partnerships; resource family development and support; team decision-making; and self-evaluation. The underlying logic is that these strategies are mutually reinforcing and, if effectively implemented, will identify and draw community resources into decision-making and family support. In addition to helping sites develop their capacity for self-evaluation, the evaluation team has compiled data describing the implementation of each strategy, the exposure of individual children to key elements of Family to Family practice, and the safety and permanency outcomes they have experienced. The panel will present the results of analysis in each area.

Implementing the Core Strategies of Family to Family

Daniel Webster, University of California Berkeley, dwebster@berkeley.edu

Some Family to Family strategies are only indirectly linked to the experiences of individual children involved with the child welfare system. Monitoring implementation in these areas involves efforts such as tracking an inventory of foster families to assess the constraint the system faces in placing children with families rather than congregate care facilities. However, the strategy of team decisionmaking (TDM) uses a database to capture information about each TDM meeting, including participant characteristics, the presence of key practice elements, and recommendations from the meeting. Data are now available for more than 50,000 meetings at involving the removal of children from their homes and their movement from one placement setting to another. This paper will summarize what the TDM database reveals about the exposure of individual children to key elements of Family to Family practice and describe how the level of exposure is related to recommendations coming out of those meetings.

The Impact of Family to Family on Safety and Permanency Outcomes

Judith Wildfire, Wildfire Associates Inc, jwildfire@wildfireassociates.com

The Family to Family evaluation team has devoted considerable effort to building a database in each anchor site to track the experience of individual children from initial reports of abuse and neglect through any experiences in out-of-home care. By linking referral reports with data about each placement while in child welfare custody and arraying the data in a longitudinal format, it is possible to create a statistical case history for each child. A further linkage to TDM data makes it possible to assess the impact of key elements of Family to Family practice on safety and permanency outcomes. This paper will describe the multivariate analyses undertaken by the evaluation team to gauge impact based on varying levels of exposure to these key elements across anchor sites. The analysis will encompass the experience of all children whose initial involvement with the child welfare systems in these sites occurred between 2005 and 2008.

Building Effective Community Partnerships With Public Child Welfare Agencies: Lessons From the Field

David Crampton, Case Western Reserve University, david.crampton@case.edu

The Family to Family initiative seeks to develop partnerships between neighborhood organizations and public child welfare agencies. These partnerships can help recruit and support foster parents, and provide support to families who might otherwise receive child protection services. This presentation describes the implementation of the community partnership strategies in eleven diverse Family to Family communities in six states. A common set of implementation challenges are reviewed, including changes in agency leadership, limited buy-in from frontline staff, reduced budgets and staffing levels, and negative media coverage. Examples of how these communities addressed these challenges are also presented. We review the extent to which these practices are applicable across diverse communities and how these strategies have been adapted to meet local needs. Evidence of the impact of these partnerships is presented along with a discussion of how to evaluate community partnerships in child protection work.

Organizational Effectiveness and Family to Family Implementation: A Roadmap for Community-Based Program Improvement

Thomas Crea, Boston College, creat@bc.edu

Program evaluators are devoting increased attention to contextual influences on program implementation and practice fidelity. Once such influence is the extent to which organizational factors help or hinder program implementation. This presentation examines recent findings from a pilot survey of organizational excellence (the Survey of Organizational Excellence) given to child welfare workers and supervisors in two Family to Family anchor sites in Kentucky (N=284) and California (N=284). Specific attention is paid to how five dimensions of organizational effectiveness (work group, accommodations, organizational features, information, and personal) and their related constructs are related to implementation indicators of each of the four core strategies of Family to Family and overall perceptions of the initiative. Quantitative survey findings, and open-ended comments, are interpreted in light of their usefulness for agency managers to improve implementation of the initiative and changes in agency practices.

Session Title: Evaluation Impact With innovative Technologies and Media Strategies

Multipaper Session 290 to be held in Panzacola Section F3 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Evaluation Use TIG

Chair(s):

Jo Ann Intili, JBS International Inc, jintili@jbsinternational.com

Discussant(s):

Susan Tucker, Evaluation and Development Associates LLC, sutucker1@mac.com

Has Quality Assurance and Evaluation Made it Less Attractive to be a Teacher? A Survey in Five National Contexts

Presenter(s):

Carsten Stroemb�k Pedersen, University of Southern Denmark, csp@sam.sdu.dk

Peter Dahler-Larsen, University of Southern Denmark, pdl@sam.sdu.dk

Abstract: Do teachers think that quality assurance and evaluation (QAE) initiatives have made it less attractive to be a teacher? A cross-national survey in England, Denmark, Finland, Scotland, and Sweden identified both national and individual variations in teachers' responses to this question. In order to explain these variations, we look at the influence of QAE on teacher autonomy and on perceived social trust in teachers. We also check whether these phenomena are related to the perceived use of evaluation at the school level and to QAE as a management instrument.

Using Process Evaluation to Understand Context: A Practical Example

Presenter(s):

Lei Zhang, Centers for Disease Control and Prevention, lzhang2@cdc.gov

Martha Engstrom, Centers for Disease Control and Prevention, cpu5@cdc.gov

Abstract: Capturing context is key to understanding program effectiveness because organizational factors (e.g., structures, processes and resources) and environmental variables (e.g., political and cultural climate) can greatly influence program accomplishments. However, in practice, evaluations frequently fail to seek adequate understanding of the context within which programs are implemented. This paper uses a process evaluation conducted by the Office on Smoking and Health (OSH), Centers for Disease Control and Prevention as a practical example to demonstrate how process evaluation can be effectively used to improve OSH's understanding of the environment in which OSH provides its technical assistance to National Tobacco Control Program (NTCP) grantees.

Alternative Teacher Preparation: Evaluation as a Driver of Practice and Policy

Presenter(s):

Edith Stevens, ICF Macro, edith.s.stevens@macrointernational.com

Helene Jennings, ICF Macro, helene.p.jennings@macrointernational.com

Abstract: Alternative teacher preparation is critical to addressing teacher shortages in Maryland. The expansion of these programs has occurred at a rapid pace, from eight to 26 programs in just three years. With this kind of growth the State wants to ensure that these programs are of high quality and are sustainable. The project evaluator, Macro International, played a pivotal role in helping the State design and implement sustainable models. In this presentation, particular attention will be given to how the evaluator collaborated with program operators and the State to develop a reporting template to capture key inputs and outcomes across programs. Presenters will discuss how results were used to strengthen current programs, inform the development of new programs, contribute to the creation of an accountability structure that is embraced by all program partners, and establish state (and possibly national) standards.

Session Title: Empowering Non-profits through an Evaluation Learning Community

Panel Session 297 to be held in Panzacola Section H4 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Chair(s):

Wayne Parker, Virginia G Piper Charitable Trust, wparker@pipertrust.org

Discussant(s):

Peter York, TCC Group, pyork@tccgrp.com

Abstract: Far too often nonprofit evaluations are designed to satisfy the needs of funders who determine the definition of success. This session focuses on a process of bringing similar nonprofits in a community together - after school providers in this case - to identify their common definitions of success; to easily generate real-time demographic, output, and outcome data; and to utilize the data to drive programmatic decision making. Additionally, there is a focus on utilizing the data beyond the individual nonprofit in a learning community to build overall community evaluation capacity and to improve the quality of programming across organizations. Implications for both nonprofits and funders relative to evaluation practices are discussed.

Part I: Developing Common Evaluation Tools

Wayne Parker, Virginia G Piper Charitable Trust, wparker@pipertrust.org

This is a description of the process of identifying common definitions of success, translating them into effective measures, automating data collection, and developing quantitatively based evaluation reports for the statistically unsophisticated consumer.

Part II: Building Evaluation Capacity

Jane Dowling, Wellington Consulting Group Ltd, janedowling@msn.com

The participant nonprofit organizations varied widely in their level of experience with and understanding of the use of evaluation for program development and quality improvement. This presentation focuses on the development of a common language and base knowledge of evaluation as a necessary prerequisite to implementing an ongoing evaluation learning community.

Part III: Applications and Implications of Building a Learning Community

Catherine Jahnes, Virginia G Piper Charitable Trust, cjahnes@pipertrust.org

This presentation explores the impact of real-time data on nonprofit functioning and decision making. The evaluation learning community has the potential to minimize the power imbalance between nonprofits and their local funders. These and other implications of the project are explored and discussed. Future applications of the evaluative learning community process are examined.

Session Title: What Have We Learned About Education Programs, How Do We Use What We've Learned, and Where Do We Go From Here?

Multipaper Session 298 to be held in Sebastian Section I1 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Government Evaluation TIG and the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Alan L Ginsburg, United States Department of Education, alan.ginsburg@ed.gov

Discussant(s):

David Goodwin, Bill & Melinda Gates Foundation, david.goodwin@gatesfoundation.org

Abstract: The Office of Management and Budget (OMB) developed the Program Accountability Rating Tool (PART) by executive order of President Bush. PART required federal programs to provide evidence of program effectiveness with their budget requests. PART questions addressed relevant evaluations; OMB emphasized randomized-control trials. With a new administration, the context has changed: President Obama has called for fundamentally reconfiguring the PART and suspended reviews. This session examines the use of education evaluations from differing perspectives and contexts, including a review of PART assessments for 130 Federal education programs, a review of 75 evaluations conducted for the Department of Education concerning their rigor and relevance, a model for translating research into improved practice, and a new model for evaluation based on a systematic review of evaluations for college access and retention programs. We invite discussion on using research, evaluations, and performance measures to increase program effectiveness, accountability, and desired social change.

Education Programs: Program Assessment Rating Tool (PART) Assessments, Evaluation, Performance Measures and Targets - Past Patterns and Future Directions

Sharon Stout, United States Department of Education, sharon.stout@ed.gov

Jay Noell, United States Department of Education, jay.noell@ed.gov

Margaret Cahalan, United States Department of Education, margaret.cahalan@ed.gov

Zeest Haider, United States Department of Education, zh2124@barnard.edu

This paper reviews PART assessments for 130 federal education programs through 2008: 94 were Department of Education programs, with the remainder spread across 14 other agencies (including Defense, Health and Human Services, Interior, Labor, and the National Science Foundation). OMB's website, www.ExpectMore.gov, reports type of program, overall ratings, scores, answers to PART questions, and performance measures and targets. To this was added information from agency strategic plans, goals, and budgets. The analysis compares how programs were structured, and summarizes patterns within and across agencies. Program types and structures are analyzed with measures, targets, and scores on the PART components of program purpose and design, strategic planning, program management, and program results and accountability. In considering how linkages across education programs (and agencies) might be made more strategic and cross-cutting, suggestions are made to improve performance measures and reporting of program contributions to outcomes of interest to the public.

A Systemic Review of Studies Sponsored by the Department of Education to Evaluate Federal Education Programs: Lessons Learned Methodologically and Substantively

Margaret Cahalan, United States Department of Education, margaret.cahalan@ed.gov

Sharon Stout, United States Department of Education, sharon.stout@ed.gov

This paper reviews a set of 75 studies that were completed in the last decade by the Policy and Program Studies Service (PPSS) and by the Institute for Education Sciences (IES) to address evaluation questions concerning federal education programs. Most of these studies were either congressionally mandated or were undertaken to inform PART assessments. This paper adapts the systematic review methods developed by the Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre) at the University of London (http://eppi.ioe.ac.uk/cms/). The systemic review will address: questions asked, design, statistical methods, metrics developed, study validity and measurement error issues, and findings. A major focus of the review will be the lessons learned for future work, both methodological and substantive.

Translating Research to Practice in Education

Jennifer Ballen Riccards, United States Department of Education, jennifer.ballen.riccards@ed.gov

How are research findings best conceptualized, translated, and communicated to a practitioner audience? Any good translation identifies core meaning and considers how the message will be both sent and received. "Doing What Works" (www.dww.ed.gov) is a multimedia website sponsored by the Department of Education dedicated to helping educators identify and make use of effective, research-based teaching practices. The Doing What Works approach uses three steps to accomplish this goal: 1) consolidating research findings into recommended teaching practices; 2) translating recommended teaching practices into multimedia examples of teaching and other information on a website; and 3) communicating to practitioner audiences (teachers and professional developers involved in K-12 education) using these resources. We will discuss issues in conceptualizing research findings, maintaining fidelity to the research, considerations of audience in translating research to practice, and developing effective vehicles - including innovative media types - for communicating key ideas and promising practices.

Lessons Learned From the Decade in Designing a New Generation of Evaluations of College Access and Retention Studies

Margaret Cahalan, United States Department of Education, margaret.cahalan@ed.gov

This paper takes an in-depth look at six federal education programs within the area of college access and retention, reviewing their PART scores and the evaluation studies upon which the scores were based. The programs all have had evaluation studies of varying degrees of rigor in addressing program impact: Student Support Services, Upward Bound, Talent Search, McNair, GEAR UP, and CCAMPIS. The work focuses on the interaction of evaluation, PART, budget development, congressional action, and ED policy development. The paper addresses lessons learned both methodologically and substantively. The paper describes a new standards-based model for program evaluation that emphasizes partnership, reflection, and innovation.

Session Title: How to Prepare for the Evaluation of Community Change Initiatives and Other Nearly Impossible Tasks

Think Tank Session 299 to be held in Sebastian Section I2 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG

Presenter(s):

David Chavis, Community Science, dchavis@communityscience.com

Tom Kelly, Annie E Casey Foundation, tkelly@aecf.org

Astrid Hendricks, The California Endowment, ahendricks@calendow.org

Discussant(s):

Tom Kelly, Annie E Casey Foundation, tkelly@aecf.org

Astrid Hendricks, The California Endowment, ahendricks@calendow.org

Abstract: Community Change Initiatives (CCI's) have gained popularity because so many of our society's core social problems (e.g. poverty, poor education and health, and inadequate systems of care) find their root causes in community and larger systems. Little practical advice on design and management of these initiatives is available. This think tank will be designed to discuss practical questions evaluators must answer before embarking on this unique and highly complex type of evaluation. Two CCI case studies will initially be presented: Making Connections (Annie E. Casey Foundation) and Building Healthy Communities (The California Endowment). Using these case studies, small groups will be asked predetermined questions such as: What is the theory of these CCI's and what methods do you use to test these theories? How do you build realistic expectations? What do you measure when there is so much?

Session Title: Taming, Testing and Tapping Collaboration: A Panel Discussion on Multi-level Methods and Models for Evaluating Communities of Practice

Panel Session 300 to be held in Sebastian Section I3 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG

Chair(s):

Andrea M Hegedus, Northrop Grumman Corporation, ahegedus@cdc.gov

Abstract: There is still much to be learned as evaluators seek to improve the common understanding and measurement of the construct of collaboration. Collaboration is seen as one of the building blocks of communities of practice in that it supports social learning, shared resources, and partnerships among members. However, the definition of collaboration is often ubiquitous, elusive, and variable across different applications, making its measurement difficult and nongeneralizable. Using methods and models developed for the evaluation of communities of practice in multiple settings, members of this panel will present and engage participants in discussion about how they approach collaboration and its measurement. Drawing upon theory, multi-level methods, and lessons learned from practice, the panelists will provide evaluators with a conceptual framework for evaluating communities of practice. Additional take-aways will include specific tools such as social network analysis as well as strategies for integrating research and subsequent findings into the organization

Understanding and Evaluating Organizational Collaboration

Rebecca Gajda, University of Massachusetts, rebecca.gajda@educ.umass.edu

Practitioners, often mandated to forge communities of practice and strategic alliances, are not sure of what collaboration looks and feels like. They are not sure if their collective actions constitute true collaboration or if the structural, procedural, and interpersonal relationships among partners are as healthy as possible. In this presentation, I will assert that the evaluation field holds an important and unique position to help organizational improvement stakeholders meet contemporary health, human service, and educational reform challenges predicated on the structural and inter-professional power of collaboration. I will discuss how evaluators can use collaboration theory to demystify and operationalize the construct of collaboration. Methods for qualitative and quantitative assessment of collaborative integration and quality within and between communities of practice will be discussed. I will integrate lessons learned and ah-ha moments experienced as an evaluator of various health, social service, and education reform programs.

Evaluating Collaboration Effectiveness in Disperse and Diverse Contexts: The Case of Tobacco Quitlines in North America

Michele Walsh, University of Arizona, mwalsh@email.arizona.edu

Jessie Saul, North American Quitline Consortium, jsaul@naquitline.org

Scott Leischow, Arizona Cancer Center, sleischow@azcc.arizona.edu

The North American Quitline Consortium (NAQC) is committed to developing and supporting an infrastructure for conducting tobacco cessation quitline research that will be used by the quitline community to improve services. An example of such research is the Knowledge Integration in Quitlines: Networks to Improve Cessation (KIQNIC) project. KIQNIC is a 5-year NCI-funded research study that aims to better understand how NAQC members interact, share new evidence, make decisions on how and when to implement new knowledge, and ultimately adopt practices that they believe will improve quitline outcomes. We will present the theoretical framework for the project and will discuss the process of developing collaborative relations between KIQNIC researchers and the quitline community. We will discuss how building these relations allowed us to more effectively refine study instruments, facilitate data collection and determine which results, and in which format, would be most useful to quitline community members.

Understanding Collaboration Through a Multi-level Conceptual Framework for the Evaluation of Communities of Practice

Andrea M Hegedus, Northrop Grumman Corporation, ahegedus@cdc.gov

Awal Khan, Centers for Disease Control and Prevention, akhan@cdc.gov

Evaluation of communities of practice can be viewed as a dynamic systems process that incorporates different levels of analysis, use of theory to guide process and outcomes, as well as a mixed methods approach to align appropriate methodology to the questions asked. A conceptual framework is one way to unite disparate activities into a unified whole. It provides a valuable tool to evaluate constructs such as collaboration across different levels of the model. Specification of the framework also requires that collaboration be defined in accordance with its use and application (e.g., intra-organizational relationships versus inter-personal relationships). This presentation offers a conceptual framework used to evaluate communities of practice in a national public health informatics setting, specifies how collaboration is defined and measured in different levels of the model, as well as how this work improves the understanding and measurement of collaborative processes.

Session Title: The Inherent Void of Context Without Culture in Evaluation: Lessons Learned Along the Way by Three Culturally Responsive Evaluators

Panel Session 301 to be held in Sebastian Section I4 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Multiethnic Issues in Evaluation TIG

Chair(s):

Tamara Bertrand Jones, Florida State University, tbertrand@admin.fsu.edu

Discussant(s):

Elmima Johnson, National Science Foundation, ejohnson@nsf.gov

Abstract: In order to create and maintain successful programs with positive outcomes for African Americans, evaluations must truly reflect the group and the cultural context in which the group exists. Only a few evaluators have mandated that culture be an integral part of the evaluation process and have challenged to do so in their work. This panel reflects the views of three of those culturally responsive evaluators who through their scholarship, practice and service have strived to advance the field of culturally responsive evaluation. This panel first explores the history of culturally responsive evaluation and the contributions African Americans have made to this field. Second, the training of culturally responsive evaluators will be presented along with discussion about the necessity of such programs. The third presents an overview of the role of the culturally responsive evaluator and what makes this evaluator unique within the context of evaluation.

The Road Less Traveled: The Journey of Culturally Responsive Evaluation and Its Evaluators Then and Now

Stafford Hood, University of Illinois at Urbana-Champaign, slhood@illinois.edu

Cultural change has come to America and hopefully to the field of program evaluation. While this change has been slow and challenging there is promise. We are optimistic that the reign of ethnic and cultural ambivalence will be replaced with understanding and mutual respect within our social, educational and political institutions. Cultural diversity and the contributions of people of color have been an integral but unspoken part of this nation's history and discourse. The contributions of people of color to evaluation theory and practice, particularly in regard to cultural/contextual evaluation, have only recently been acknowledged by a select few. This presentation explores the evolution of culturally responsive evaluation and the specific contributions African American evaluators have made to evaluation theory, research and practice. This presentation also explores future trends in culturally responsive evaluation.

The Culturally Responsive Evaluator: My Perspective

Henry Frierson, University of Florida, hfrierson@ufl.edu

The concept and processes related to culturally responsive evaluation are open to various perspectives. Some may view the concept and related approaches as being synonymous with or tantamount to cultural competence or culturally competent evaluation. Others may see such notions as non-science and non-rigorous processes. The perspective of this presentation is that culturally responsive evaluation is not the same as cultural competence nor do advocates holding similar views feel the need nor likely have the capacity to reach such a level which would essentially require cultural immersion. Moreover, it is a process to be considered by those who are actually committed to obtaining valid data from which individuals can use to make sound decisions about programs, projects, etc. that involve individuals while taking into account the cultural context in which they operate.

Building a Legacy of Evaluators of Color: Lessons Learned in Training, Building Competencies, and Affecting Change

Rodney Hopson, Duquesne University, hopson@duq.edu

Over the last few years, efforts have been made to develop frameworks for curricula and evaluation field experiences in graduate degree and non-degree training programs in evaluation that emphasize culturally responsive evaluation strategies and to train these same evaluators to work in traditionally underserved communities and populations. Lessons learned from funded work (National Science Foundation, Kellogg Foundation, and Robert Wood Johnson Foundation) over the last five years will be shared, including ways to think innovatively about internships, curricula, training efforts, and other programs to bring these initiatives to the larger field. The purpose of the paper will be to reflect on the set of trainings, skills, and competencies potentially needed to arm a set of culturally responsive evaluators who desire to contribute to social change and social justice in traditionally underserved and marginalized communities.

Session Title: Sailing the Seven C's of Social Change: A Framework for Evaluating Social Change and Systems Change Initiatives

Demonstration Session 302 to be held in Sebastian Section J on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Systems in Evaluation TIG

Presenter(s):

Marah Moore, i2i Institute, marah@i2i-institute.com

Abstract: The Seven Cs Social Change Framework provides a method for conceptualizing, planning for, and evaluating social change initiatives. As the focus on "systems change" and social change initiatives continues to grow within both the funding and programming communities, the need for a method of measuring the progress of these initiatives is becoming more apparent. The Seven Cs Social Change Framework provides a method that helps to bridge the divide between direct service programming and systems change, providing a systems-based approach to assessment and evaluation that can be tailored to the unique aspects of different initiatives as well as the unique aspects of different sites within a single initiative. This workshop will demonstrate how to apply the conceptual framework provided by the Seven Cs Social Change Framework to a variety of change initiatives.

Session Title: Contextualizing Evaluation and Research: Indigenous Peoples Perspectives

Panel Session 303 to be held in Sebastian Section K on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Presidential Strand and the Indigenous Peoples in Evaluation TIG

Chair(s):

Joan LaFrance, Mekinak Consulting, lafrancejl@gmail.com

Discussant(s):

Joan LaFrance, Mekinak Consulting, lafrancejl@gmail.com

Abstract: Evaluators and researchers with experience working with American Indian tribes understand the importance of adapting their practice to honor cultural mores. This panel will discuss how researchers and evaluators have developed frameworks and methodologies that consider cultural values, tribal practices, and Indigenous ways of knowing. The three presenters describe a range of evaluation/research methodologies adapted for their situations. The Blackfoot project is centered within the context of language and traditions of one tribe. The North Dakota project addresses a framework to fit a state-wide program, and American Indian Higher Education Consortium is defining a framework for contextually responsive evaluation for the 36 tribal colleges in the United States. The presentations explore how language, values, and Indigenous ways of knowing influence evaluation and the mediating role these contextual elements play in ensuring meaningful participation and execution on an evaluation or research project.

Ihto'tsii Kipaitapiiwahsinnoon (Coming From Within): The Blackfoot Project

Iris Prettypaint, University of Montana, dprettypaint@yahoo.com

This collaborative graduate initiative was initiated by University of Montana's PACE Project and Research Opportunities in Science for Native Americans (ROSNA) in 2007 and includes 49 interested members from the Blackfoot confederacy, over half of whom are women. The purpose of the project is to increase the number of Indigenous students in graduate school, particularly women in science. The Project is interested in doing collaborative tribal community-based research to complete their degree requirements and utilizing Indigenous methods for data collection and analysis. The project identified four core research issues, which includes the rediscovery of Blackfoot inherent values, particularly the Blackfoot language; the acknowledgement of traumatic stress, which permeates the very fabric of Blackfoot families and communities; and the limitations of the Blackfoot Confederacy political system to strengthen the Blackfoot people.

Native American Research Paradigm

Carol Davis, North Dakota State University, carol.davis@ndsu.edu

Native science is concerned with relationships and the belief that all living organisms are connected. All life shares this correlation, including man. In an effort to engage tribal college students in research that honors this concept, North Dakota EPSCoR promotes the integration of indigenous knowledge with multicultural inquiries that are posed through science, technology, engineering and mathematics. The research task is assigned to ten research teams that consist of a university researcher, one tribal college faculty member, and two tribal college students. The teams designed the North Dakota Tribal College Faculty Research Model: Guiding Undergraduate Student Research to guide investigations and avoid conflict with tribal customs. The studies often integrate Western concepts with indigenous knowledge to arrive at a unique conclusion.

Indigenous Evaluation Framework

Richard Nichols, Colyer Nichols Inc, colyrnick@cybermesa.com

In the American Indian Higher Education Consortium's Indigenous Evaluation Framework, developed under a project funded by the National Science Foundation, context plays a critical role in ensuring that evaluations are culturally responsive to tribal communities, values and traditions. Rather than focusing on generalizability of results, the framework emphasizes the critical importance that qualitative and quantitative evaluation approaches, designs and strategies that integrate tribal concerns in order that results and findings be utilized. Furthermore, Indigenous evaluation methodologies are meant to enhance tribal and personal sovereignty and to empower and build capacity among community stakeholders. Developed over a two-year process of tribal community member consultations, and pilot-testing with Tribal Colleges and Universities, the Indigenous Evaluation Framework represents a research-based approach to conducting evaluations in Indian Country and other Indigenous communities.

Session Title: Complementary Approaches to Evaluation at the World Bank

Panel Session 304 to be held in Sebastian Section L1 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Cheryl Gray, World Bank, cgray@worldbank.org

Abstract: This session will illustrate how various kinds of evaluation approaches complement each other in the workings of a large international organization. Specifically, the panel will draw on the experience of the World Bank in evaluating agricultural programs it has supported.

Using Multiple Methods to Evaluate Development Programs

Nalini Kumar, World Bank, nkumar@worldbank.org

This presentation will provide an overview of the evaluation. It will discuss the various methods used and the rationale for the approaches taken to conduct the study, as well as the principal findings. The objective of this presentation is to establish how complementary approaches helped to derive the evaluative findings and conclusions, and specifically how the use of multiple approaches strengthened the evaluation design. The presentation will cover a range of approaches, including population analysis, analysis of project samples, country case studies, and impact evaluations.

Evaluating Effectiveness: Case Study of Agriculture in India

Regina Birner, International Food Policy Research Institute, r.birner@cgiar.org

The second presentation will be based on a case study carried out in India. Altogether, 11 country case studies were conducted, including nine randomly selected countries and special studies of India and China. They were designed to assess the extent to which World Bank assistance supported agricultural production in the overall country programs, the appropriateness of the support in relation to agricultural potential, and the outcomes of the support. The presentation will show how the India case study evaluated the effectiveness of World Bank support at the country level and how it contributed to the overall assessment of the effectiveness of World Bank support for agricultural development globally.

Assessing Impact Using Mixed Quantitative and Qualitative Methods in Malawi

Ximena Del Carpio, World Bank, xdelcarpio@worldbank.org

The final presentation will discuss the use of impact evaluations in the context of the broader study. Specifically, this presentation will show how a combination of quantitative and qualitative methods was used to carry out an impact evaluation in Malawi. The study used a quasi-experimental design to carry out the quantitative assessment, and interviews and "nominal group processes" to conduct the qualitative analysis. The presentation will discuss how these methods complemented each other and fit into the broader evaluation.

Session Title: International Perspectives on Evaluating Scientific Policies, Programs, and Collaborations

Panel Session 305 to be held in Sebastian Section L2 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Krishna Kumar, United States Department of State, kumark@state.gov

Abstract: This panel presents on how evaluation procedures, practices, and techniques are implemented in scientific programs, policies, and/or collaborations in the United States, Russia, selected Eurasian countries and New Zealand. The presenters highlight the complex nature of these evaluations and discuss critical issues in developing core metrics that can be applied across the borders in different socio-cultural contexts to measure long-term program outcomes of science-related programs. The panelists represent a wide range of experienced professionals with international, regional and cross-cultural expertise in program management and program evaluation. The panelists discuss best practices and shortcomings in the evaluation studies that they conducted on programs related to science and scientific collaboration in their respective countries and address issues relating to cross-cultural sensitivity and the context in which the presented programs operate. An exchange of ideas will be encouraged about evaluation methodologies and techniques that are applied internationally to measure science and engineering programs.

Using National Data to Evaluate the International Collaborations of United States Scientists and Engineers

John Tsapogas, National Science Foundation, jtsapogas@nsf.gov

Science and Engineering doctoral degree holders are of special interest to many decision makers because they represent some of the highest educated individuals in the U.S. workforce. This presentation will discuss the process involved in using a national survey of doctoral scientists and engineers, the Survey of Doctorates Recipients (SDR), to evaluate the extent and character of international collaboration among U.S. doctoral degree holders. A description will be made on the procedures used to develop and append a set of questions on international collaborations to existing questionnaires of doctoral degree recipients. Data from the survey were used to evaluate how international collaborations are impacting the U.S. doctoral workforce. Data on international collaborations from the SDR will be presented by sex, research/teaching faculty status, sector of employment (industry, government, and academia), minority status, citizenship, presence of children in household, field of study of doctorate, and year of doctorate receipt.

Evaluating Micro-Ventures Development Through Science-Business Partnerships in Eurasia

Liudmila Mikhailova, United States Civilian Research and Development Foundation, lmikhailova@crdf.org

Fostering micro-ventures and facilitating new technology transfer through the formation of science-business partnerships is critical in building a market economy in Eurasia. This presentation will discuss the Science and Technology Entrepreneurship Program (STEP) and the findings from its impact evaluation conducted in Armenia, Azerbaijan, Georgia and Moldova. STEP was launched by the U.S. Civilian Research and Development Foundation (CRDF) in 2004 in several Eurasian countries under the State Department funding to nurture the creation of micro-ventures through science-business collaboration and to facilitate new technology transfer. Testimonials of each country officials about the program emphasized that STEP was recognized as an innovative development model for building entrepreneurial clusters and promoting a new generation of scientists with entrepreneurial thinking. Data from interviews and surveys will also address participation of young scientists and students in STEP grant projects; job creation opportunities and engagement of former weapon scientists in the civilian research.

Evaluating Institutional Programs: Linking Research And Education At Russian Universities

Irina Dezhina, Russian Academy of Sciences, dezhina@crdf.ru

This presentation elaborates on evaluation results of the "Basic Research and Higher Education" (BRHE) Program, a large institutional initiative aimed at strengthening linkages between research and education at Russian universities. Twenty "Research and Educational Centers" (RECs) were established at geographically spread Russian universities within the BRHE. Data from a longitudinal evaluation study show RECs performance and their impact on the Russian university system. The BRHE started in 1998 with the funding from McArthur Foundation and is cost-shared by the Russian government. The program was very timely in the Russian context, which is characterized by the separation of research and education; limited research funding at universities; weak linkages with the Academy of Science institutes; and other research and development (R&D) organizations. The evaluation results show that the BRHE Program is an effective reform effort in Russia and proves that integration of research into higher education institutions is beneficial for both areas.

Evaluation Practices In The New Zealand Government: A Case Study Of The Ministry Of Research Science And Technology

Yelena Thomas, Ministry of Research Science and Technology New Zealand, yelena.thomas@morst.govt.nz

Best evaluation practices in the area of science policy are topics of interest for many policy-makers. This presentation will focus on how evaluation practices and procedures are implemented at policy level in the New Zealand government and discuss shortcomings and stumbling blocks of this process. Many government agencies have research or performance and evaluation units that are separate from policy teams. That practice allows evaluators to stay independent and objective. The presenter will showcase a policy cycle and how it incorporates the development of a comprehensive evaluation framework when evaluation specialists work closely with policy advisors and contract managers as a team. The majority of policies are introduced to the Cabinet with a high level of evaluation framework and the timelines for reporting. The presentation will conclude with lessons learned and recommendation to the practitioners who work in the area of policy level assessment.

Session Title: Understanding Contextual Factors to Build Capacity Among People Who Don't Think They Are Evaluators

Think Tank Session 306 to be held in Sebastian Section L3 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

Presenter(s):

Julianne Manchester, Case Western Reserve University, julianne.manchester@case.edu

Discussant(s):

Molly Engle, Oregon State University, molly.engle@oregonstate.edu

Julianne Manchester, Case Western Reserve University, julianne.manchester@case.edu

Abstract: Often programersrs ignore evaluation in favor of developing the program and think of evaluation only after the program has been implemented. Building evaluation capacity of programersrs who deliver trainings in complex organizations will contribute to useful evaluations. By considering the importance of relational and organizational factors on program planning and evaluation, session participants will identify the contextual factors affecting their evaluations. Questions posed to the audience will include, "What characteristics of partnerships affect programming? What characteristics enhance access to practice-based evidence? Can trainee data and the utility of results with multiple groups (curriculum designers, faculty) be used to build capacity? How is evaluation planning shaped in the absence of adequate partners? How can using logic modeling as a tool be useful in building evaluation capacity among program planners? What logic model formats maximize evaluation capacity?" Knowing when to introduce systems tools may facilitate evaluation capacity building.

Session Title: Taking It to the Next Level: Challenges, Strategies and Lessons Learned in Linking Implementation and Outcomes

Panel Session 307 to be held in Sebastian Section L4 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Theories of Evaluation TIG

Chair(s):

Susan Berkowitz, Westat, susanberkowitz@westat.com

Discussant(s):

Larry L Orr, Independent Consultant, larry.orr@comcast.net

Abstract: Evaluators are often faced with the problem of how to effectively link evaluation of the implementation of a program or intervention with the outcomes of these interventions. Maintaining a strict separation between process evaluation, on the one hand, and outcome evaluation, on the other, seems unproductive. However, there is no clearly indicated path to follow in addressing the middle ground occupied by these linkages. This panel will focus on key issues that arise in meeting the challenges of linking implementation and outcomes for a range of education, mental health, and substance abuse prevention interventions. Panelists will discuss strategies for testing a program's underlying theory of change, ways of measuring implementation and treatment fidelity, and analytic techniques employed to make the implementation-to-outcome linkage. They will assess the relative success and utility of these different approaches and offer lessons learned. The discussant will draw together their arguments and suggest potentially promising future paths.

Linking Implementation and Outcomes for Three Educational Interventions: Challenges, Strategies and Lessons for the Future

Joy Frechtling, Westat, joyfrechtling@westat.com

Joy Frechtling, an evaluator who has conducted a wide range of studies looking at intervention programs and their outcomes, will discuss several evaluation studies in which the connections between implementation and outcomes were examined. Focusing on educational interventions, she will describe three studies, one examining an arts education reform, and two examining reading programs, in which various strategies and analytic techniques were used to test the theory of change underlying the program. Relationships based on both an overall implementation score and multi-part implementation scores will be discussed. She will describe the extent to which these various endeavors were successful in tracking connections, challenges encountered, and emergent ideas about how to more effectively assess these linkages.

The Hype and Futility of Measuring Implementation Fidelity

David Judkins, Westat, davidjudkins@westat.com

In qualitative evaluation there is tremendous enthusiasm for "looking inside the black box." A recent popular book from a well respected evaluator,"Learning More from Social Experiments," provides encouragement that statistically rigorous methods can be developed to understand the role of implementation fidelity and other mediators in the results of randomized trials. David Judkins, a statistician, recently led the analysis of a group randomized trial of alternative preschool curricula in Even Start projects. He will argue that attempts at relating fidelity to intervention outcomes are for the most part well-intentioned but ill-considered and expensive endeavors doomed to failure. Worse, they tend to warp the design of experiments, lowering power for primary experimental endpoints due to the need to lower the cost of fidelity measurement. He will review technical challenges in the statistical inference of mediated effects and provide recent examples to illustrate his points.

Examples of Success in Using Implementation/Fidelity Measures to Understand Cross-site Variation in Multisite Evaluations

Joseph Sonnefeld, Westat, josephsonnefeld@westat.com

Robert Orwin, Westat, robertorwin@westat.com

Use of treatment fidelity and implementation measures has been repeatedly recommended in evaluating multisite interventions, which are often characterized by substantial cross-site variation in both the program models being implemented and success of implementation. Responding to arguments that individually randomized designs implemented at multiple sites may be usefully analyzed without reference to variation in intervention fidelity, Joe Sonnefeld and Rob Orwin, experienced evaluators of national substance abuse and mental health initiatives, examine ways that well-measured fidelity to an evidence-based model, or cross-site differences in program "dose," contributed to understanding variations in effectiveness, led to meaningful recommendations, and prevented false negative conclusions about overall program effectiveness. They draw on examples from two Center for Mental Health Services (CMHS) initiatives, Access to Community Care and Effective Services and Supports (ACCESS) and Consumer Operated Programs and an ongoing evaluation of the Center for Substance Abuse Prevention (CSAP)'s Strategic Prevention Framework State Incentive Grants.

Session Title: Clinical and Translational Science Awards: Evaluation Approaches and Dimensions of Quality

Multipaper Session 308 to be held in Suwannee 11 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Health Evaluation TIG

Chair(s):

Don Yarbrough, University of Iowa, d-yarbrough@uiowa.edu

Discussant(s):

Don Yarbrough, University of Iowa, d-yarbrough@uiowa.edu

Abstract: The NIH NCRR established the Clinical and Translational Science Awards in 2005 with the goal of creating a national consortium to transform clinical and translational research in the United States. The national consortium now consists of three cohorts totaling 38 academic health centers. Each participating institution, typically under the aegis of a new or existing "Institute for Clinical and Translational Science," has proposed and implemented a site-level "monitoring, tracking, and evaluation" component (EC). However, the individual institutes' evaluations vary widely in structure, purpose, and function. These papers provides a detailed look at site-level dimensions of difference, for example, 1) EC location (internal or external to the Institute), 2) administration and governance, 3) reporting mechanisms and structure, 4) funding and resources available 5) identified purposes 6) anticipated challenges and barriers, 7) general evaluation approaches and methods, and 8) definitions and dimensions of evaluation quality.

Clinical and Translational Science Awards (CTSA) Evaluators' Survey: An Overview

Cath Kane, Weill Cornell Clinical Translational Science Center, cmk42@cornell.edu

Sheila K Kessler, Northwestern University, s-kessler@northwestern.edu

Knut M Wittkowski, Rockefeller University, kmw@rockefeller.edu

At the 2008 CTSA evaluators meeting in Denver, several members gathered structured input from their colleagues towards the development of a basic CTSA Evaluators Survey. The goal of the survey was to investigate common trends in three critical areas: 1) the management of evaluation, 2) data sources and collection methods, and 3) analysis or evaluation activities currently being conducted. The survey was conducted in the Spring of 2009 with CTSA Evaluation staff intended as both the survey participants and primary audience for findings. The goal of this survey was to provide a comparative overview of all current CTSA evaluators, to allow each team to solicit best practices, and to allow the various CTSA evaluation teams to orient themselves relative to their peers. As such, a review of the survey results will act as a fitting introduction to the series of case studies gathered for this multipaper of CTSA Evaluators.

Evaluation in a Complex Multi-Component Initiative, University of Wisconsin's Institute for Clinical and Translational Research

D Paul Moberg, University of Wisconsin, dpmoberg@wisc.edu

Jan Hogle, University of Wisconsin, jhogle@wisc.edu

Jennifer Bufford, Marshfield Clinic Research Foundation, bufford.jennifer@mcrf.mfldclin.edu

Christina Spearman, University of Wisconsin, cspearman@wisc.edu

UW-ICTR's Evaluation Office provides program evaluation support to nearly 30 components from throughout the University and Marshfield Clinic with 2.5 FTE (4 people) located in both places. Organizationally, Evaluation is part of Administration, integrating cross-component evaluation, priority setting, reporting, accountability, and quality improvement, as well as providing informal "ethnographic" access to the Institute's daily activities. Building internal rapport is key to leveraging interest in program evaluation in an environment unfamiliar with the concepts. We additionally focus on cross-component evaluation addressing ICTR goals and specific aims, and participate in national consortium activities. Our role involves helping components to better articulate goals and objectives; assisting with annual report narratives; conducting key informant interviews and investigator surveys; and compiling case studies. Our Evaluation Working Group represents all cores in a utilization-focused, participatory approach to monitoring and evaluation. The presentation will further discuss our approach, findings, and possible indicators of "evaluation quality".

Atlanta Clinical Translational Science Institute: An Evaluation Framework

Iris Smith, Emory University, ismith@sph.emory.edu

Tabia Henry Akintobi, Morehouse School of Medicine, takintobi@msm.edu

Brenda Hayes, Morehouse School of Medicine, bhayes@msm.edu

Andrew West, Emory University, awest2@emory.edu

Cam Escoffery, Emory University, cescoff@sph.emory.edu

The Atlanta Clinical Translational Science Institute (ACTSI) represents a multi-institutional partnership between Emory University, Morehouse School of Medicine, Georgia Institute of Technology and several community organizations. The evaluation function is highly collaborative and organized around an evaluation team consisting of evaluators from each of the primary evaluation user groups, i.e. the collaborating academic institutions, the Institute's Bioinformatics key function program, ACTSI administration and a part time research assistant. The evaluation team meets bi-weekly to plan evaluation activities, review data and prepare evaluation reports for ACTSI leadership. All decisions are made through consensus. This collaborative approach to evaluation has been beneficial in a number of ways by: facilitating evaluation "buy-in;" providing a vehicle for rapid communication of evaluation plans and findings; providing centralized coordination of evaluation activities across the ACTSI key functions; strengthening existing partnerships; and fostering the development of additional collaborative activities among team members.

Duke Translational Medicine Institute (DTMI) Evaluation

Rebbecca Moen, Duke Translational Medicine Institute, rebbecca.moen@duke.edu

Melissa Chapman, University of Iowa, melissa-chapman@uiowa.edu

Vernita Morgan, University of Iowa, vernita-morgan@uiowa.edu

The DTMI evaluation tracks for continuous quality improvement and is focused on measuring how each component contributes to the goals of the national CTSA Consortium and the DTMI. Using a decentralized model, the DTMI component leadership and staff monitor metrics for their respective components and provide reports to DTMI Administration. Our evaluation efforts are fully integrated within our administrative leadership structure and strategic planning processes. We have developed a tool called a "Zerhouni-gram", derived from instruments used to develop the NIH Roadmap, which each component describes its strategic goals for the following year, including: 1) activities that "definitely can get done," 2) activities that "should get done," 3) stretch goals, and 4) national CTSA Consortium goals. Each component completes worksheets on an annual basis to describe its progress toward these strategic activities, its individual component goals, and if applicable, its progress on issues that were identified in the previous review by our External Advisory Committee.

ICTS Evaluation and Metaevaluation: The University of Iowa Approach

Emily Lai, University of Iowa, emily-lai@uiowa.edu

Antionette Stroter, University of Iowa, a-stroter@uiowa.edu

Douglas Grane, University of Iowa, douglas.grane@gmail.com

The CTSA Evaluation structures and functions at the University of Iowa are distributed across all key functions. The responsibility for conducting on-going and annual evaluations resides with the key function directors. The Center for Evaluation and Assessment, an independent 3rd party, regents-approved evaluation and assessment center since 1992, is responsible for formative and summative metaevaluation. The Director of the CEA consults with the ICTS Executive Committee and Key Function members on a bi-weekly basis. Staff members at the CEA are available for on-going formative consultation on evaluation activities and for evaluating specific sub-components that fall outside the reach of individual key functions. In addition, the CEA works with the Informatics Key Function to review institute-wide information needs for the overall governance function. Beginning in Year 2, the CEA assumed responsibility for an overall metaevaluation of all Key Function Monitoring, Tracking and Evaluation components and products.

Session Title: Evaluating Technical Assistance for Capacity Building: Learning When to Lead and When to Follow

Panel Session 309 to be held in Suwannee 12 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

Chair(s):

Marilyn L Ray, Finger Lakes Law & Social Policy Center Inc, mlr17@cornell.edu

Discussant(s):

Judith Ottoson, Independent Consultant, jottoson@comcast.net

Abstract: Capacity building efforts are important to the evaluation and consulting fields. As accountability requirements have increased in recent years in social and nonprofit programs, interest in capacity building has increased commensurately. The major strategy for increasing capacity has been training. However, research indicates that training alone is insufficient to achieve long-term change (e.g., Ringwalt, et al, 2002; Hall & Hord, 2005; Fixsen, et al, 2005) and follow-up technical assistance is recommended to optimize potential for change. We do not know what constitutes best practices for technical assistance in social programs and little literature reports results from evaluations of different types of technical assistance approaches. The panel presentation will help begin to fill the gap in the literature with results from four capacity building efforts. Challenges from the study sites, including resources and measurement will be presented. To conclude, an expert discussant will remark on the works presented.

The Added Value of Offering Technical Assistance Following a Training Program

Dana Keener, ICF Macro, dana.c.keener@macrointernational.com

Technical assistance (TA) is growing in popularity as a strategy for capacity building, but little is known about the added value of offering TA relative to other capacity building efforts which may be less costly (such as training). This presentation will describe four elements of TA identified from the literature that may be linked to its effectiveness: (1) relationship quality; (2) individualization; (3) proactive design; and (4) dosage. In addition, results of a longitudinal, quasi-experimental evaluation study designed to assess the relative impact of training and TA components of a capacity building intervention for violence prevention professionals will be shared along with recommendations for future TA programs. This presentation may be especially useful to those who design and/or evaluate capacity building interventions and to those who allocate funds for capacity building.

Training and Technical Assistance Approaches to Building Prevention Capacity

Catherine A Lesesne, Centers for Disease Control and Prevention, clesesne@cdc.gov

Capacity building projects employing training and technical assistance (T/TA) strategies are continually responding to challenging environments. It is important that capacity building efforts have clearly articulated desired outcomes as well as the potential for sustainability despite real-world challenges such as staff turnover. However, little is known about the best practices for capacity building T/TA. Evaluations of capacity building programs need to address the relative success of proactive and reactive T/TA strategies in meeting stated capacity outcomes. This presentation will describe the T/TA strategies (proactive and reactive) being used in a multi-site, federally-funded, capacity building project to build prevention capacity in the area of teen pregnancy prevention. The project includes 3 national, 9 state, and 4 regional technical assistance grantees all using a variety of approaches to capacity building.

A Proactive Approach to Evaluation Technical Assistance

Jennifer L Duffy, South Carolina Campaign to Prevent Teen Pregnancy, jduffy@teenpregnancysc.org

Polly Edwards-Padgett, South Carolina Campaign to Prevent Teen Pregnancy, jduffy@teenpregnancysc.org

Mary Prince, South Carolina Campaign to Prevent Teen Pregnancy, mprince@teenpregnancysc.org

Shannon Flynn, South Carolina Campaign to Prevent Teen Pregnancy, jduffy@teenpregnancysc.org

Erin Johnson, South Carolina Campaign to Prevent Teen Pregnancy, jduffy@teenpregnancysc.org

Experience gained from working with community-based organizations has demonstrated that traditional training and reactive technical assistance (TA) do not achieve the level of organizational capacity necessary to sustain on-going evaluation activities, especially when an organization has limited resources. This presentation describes how proactive TA was introduced into an existing model of training and TA to build organizational evaluation capacity. Prior to starting proactive TA, in-depth interviews were conducted with 14 community-based organizations to assess evaluation needs. Evaluation materials and user-friendly, flexible forms were developed to help organizations implement their evaluation strategies including logic models, goals and objectives, evaluation plans, process and outcome measures, data management and analysis, report writing, and using data for continuous quality improvement. Results from the needs assessment, TA dosage, satisfaction with TA, and lessons learned from the field also will be discussed.

A Case Study Report From a Technical Assistance Effort

Marilyn L Ray, Finger Lakes Law & Social Policy Center Inc, mlr17@cornell.edu

Training, follow-up technical assistance and capacity building go hand-in-hand and are in growing demand, yet we know little about best-practices for technical assistance. This paper will report preliminary findings from a case study of follow-up technical assistance provided to nine state and three regional organizations to improve their capacity to train on science-based programs found successful for preventing teen pregnancy. Training of grantees on a science-based program took place in February 2009 and follow-up technical assistance is being provided until late August 2009. Results will be reported from self-administered pre/post surveys of the grantees, logs with amounts and types of technical assistance provided, and semi-structured interviews with both the recipients of the technical assistance and the technical assistance providers. Also discussed will be a number of challenges to measuring the change resulting from this training and technical assistance capacity building effort.

Session Title: Evaluating Teacher Professional Development: Randomized Control Studies

Multipaper Session 310 to be held in Suwannee 13 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Bruce Frey, University of Kansas, bfrey@ku.edu

Discussant(s):

Haiyan Bai, University of Central Florida, hbai@mail.ucf.edu

Striving to Link Teacher and Student Outcomes: Results From an Analysis of Whole-School Literacy Interventions

Presenter(s):

Kelly Feighan, Research for Better Schools, feighan@rbs.org

Elena Kirtcheva, Research for Better Schools, kirtcheva@rbs.org

Eric Kucharik, Research for Better Schools, kucharik@rbs.org

Abstract: Researchers examined the effects of a federal Striving Readers initiative on teachers' pedagogy and student performance on a state assessment and Iowa Test of Basic Skills. The study was conducted in eight middle schools situated in a large, urban district. Half of the schools participated in the two-year intervention and half served as randomly-selected comparison schools. Designed to foster greater classroom literacy integration, the intervention comprised weekly classes for content teachers, in-school coaching assistance, and a principal seminar. Analysts used hierarchical linear modeling to account for contextual factors at the school-, teacher-, and student-level and to determine any relationship between teachers' reported literacy strategy use and their students' academic outcomes. Results showed modest gains in the treatment teachers' (N = 66) implementation of specific strategies; however, the association between teacher and student outcomes was weak. Evidence culled from observations and interviews is mined for insight into the findings.

Rice Model for Teacher Professional Development

Presenter(s):

Wallace Dominey, Rice University, wdominey@rice.edu

Dana Diaconu, Rice University, ddiaconu@rice.edu

Abstract: The Rice Model for Teacher Professional Development started in 2006 as an innovative professional development program designed to create systemic change in elementary science instruction in large urban school districts. Using the Rice Model, 80 elementary grade level science teachers are trained each year from more than a dozen districts in the greater Houston area. These districts represent more than a million students, the majority of which are high needs students from historically underrepresented minority groups. The Rice Model is applicable for any STEM education discipline and any K-12 grade level. Preliminary results show increased student achievement. For 2008-09, we have adopted a Clustered Randomized Design with Treatment and Control groups randomized at the school level. We evaluate program outcomes using quantitative and qualitative methods and a variety of assessment instruments, including the RTOP classroom observation instrument and newly created valid and reliable measures of student achievement in science.

Evaluating Teachers' Professional Development Experiences: Data From a Two-Year Randomized Controlled Trial Study

Presenter(s):

Andrew Newman, Mid-continent Research for Education and Learning, anewman@mcrel.org

Abstract: Teachers' participation in professional development (PD) activities was examined across both years of a just concluded large-scale randomized control trial (RCT). The intent of this two-year RCT was to investigate the use of a specific PD program focused on classroom assessment in participating elementary schools. For this paper, baseline, Year 1 and Year 2 data from over 300 teachers were analyzed regarding professional development activities other than the RCT intervention. Teachers' participation in PD was examined including frequency, type of activity, and the perceived impact of the PD on classroom practice. Additionally, demographic and geographic characteristics of the sample were looked at contextually to see how they relate to various types of PD in which teachers engage. Evaluators will learn about data findings, PD trends over the two years of the study and the implications for future analyses.

Session Title: Indicators for Evaluating Pipeline Programs

Think Tank Session 311 to be held in Suwannee 14 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Tom McKlin, The Findings Group, tom.mcklin@gmail.com

Discussant(s):

Betsy Bizot, Computing Research Association, bizot@cra.org

Sarah Hug, University of Colorado at Boulder, hug@colorado.edu

Daryl Chubin, American Association for the Advancement of Science, dchubin@aaas.org

Abstract: Many federal and state funders focus on pipeline programs, projects designed to move participants along a pre-determined path such as from high school to undergraduate to graduate school and into a career in a specific domain. Funders spend hundreds of millions of dollars supporting these programs, and they are uniquely difficult to evaluate. A team of evaluators working on a set of National Science Foundation programs has spent months drafting a set of common, core indicators that may be applied across all pipeline evaluations. Therefore, this think tank will present the draft of common, core indicators; seek critical feedback from other evaluators; test the indicators by asking other evaluators to envision applying the indicators to their own programs, and brainstorm improvements.

Session Title: Evaluation Adjustments for Imperfect Fidelity in the Implementation of Social Experiments

Panel Session 312 to be held in Suwannee 15 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Cristofer Price, Abt Associates Inc, cristofer_price@abtassoc.com

Discussant(s):

Barbara Goodson, Abt Associates Inc, barbara_goodson@abtassoc.com

Abstract: In 2005, the U.S. Department of Education awarded eight Striving Readers grants to projects around the country. The goal of Striving Readers is to improve literacy skills and achievement for struggling readers. This panel includes insights from several of the research teams involved in evaluating the reading programs implemented in grantee schools. Specifically, the panel focuses on the adjustments made by the evaluation teams in response to various implementation fidelity shortcomings in the field. Presenters will discuss the types of fidelity issues faced in their respective evaluation sites, efforts made to correct these issues during the course of the evaluations, and techniques utilized during data analysis to correct for potential biases introduced by the fidelity issues. By combining experimental data with quasi-experimental statistical methods, evaluators were able to leverage more information from their study designs than otherwise would have been possible.

Enhancing the Precision of Program Evaluation Results by Converting Intention-to-Treat Estimates to Impact-on-the-Treated Estimates: Striving Readers in Newark, New Jersey

Jennifer Hamilton, Westat, jenniferhamilton@westat.com

The READ 180 literacy curriculum, implemented in Newark, NJ as part of the US Department of Education's Striving Readers grant program, is being evaluated using a cluster randomized trial design. In the evaluation 19 schools have been randomly assigned to the treatment (READ 180; 10 schools) or control (standard literacy curriculum; 9 schools) groups. At issue in the evaluation is the fact that over 10% of the students assigned to treatment schools never received the READ 180 curriculum. As a result, the study has focused on the intention-to-treat effect of the program. However, program officers would like information on the effect of the program for those students who actually received the treatment. To provide such an estimate, the Bloom adjustment was utilized to convert intention-to-treat estimates into impact-on-the-treated estimates. The adjustment allows the evaluation team to account for the high mobility context within which READ 180 operates.

What Matters Most: Relating READ 180 Inputs to Student Outcomes

Deb Coffey, Research for Better Schools, coffey@rbs.org

The randomized control trial of Scholastic's READ 180 in the Memphis Striving Readers Project showed no difference in impact during the first two years of implementation. However, various local issues and challenges led to substantial differences in the fidelity of implementation across schools in the study. At what point do we decide between "the program doesn't work for this population" and "the intervention wasn't implemented as planned and fidelity of implementation should be improved before making judgments about the program"? Evaluators of the MSRP targeted intervention collected implementation data related to a variety of inputs in 19 classrooms in eight schools. These data are used in multiple regression analyses to determine which inputs contribute most to student outcomes. Results can be used to inform additional implementation analyses and provide recommendations to the district about where to focus improvement efforts.

Using Fidelity Data in Impact Estimates

Bonnie Faddis, RMC Research Corporation, bfaddis@rmccorp.com

Margaret Beam, RMC Research Corporation, mbeam@rmccorp.com

As part of its Striving Readers grant, Portland Public schools implemented the Xtreme Reading program (developed by the Center for Research on Learning at the University of Kansas) in Grades 7-10 at 10 low achieving middle and high schools. Students were randomly assigned to treatment and control conditions within schools and grade levels. To measure fidelity of implementation, the evaluation team collected data through classroom visits, teacher surveys, and teacher logs of classroom activities, and also quantified teacher characteristics, group professional development hours, school literacy coach assistance, and professional developer assistance for each teacher. Additional data on implementation issues were collected through interviews with professional developers, district staff, school literacy coaches, and school administrators. This presentation will summarize the district's implementation fidelity issues and the evaluation team's efforts to incorporate fidelity data into the impact analyses.

Implications for Evaluation: Barriers to Implementation Fidelity in a Randomized Controlled Trail

Kimberly Sprague, Brown University, kimberley_sprague@brown.edu

The Education Alliance at Brown University is conducting a randomized controlled trial (RCT) evaluating the effectiveness of two literacy interventions on the reading achievement of students identified as "struggling readers." Adjustments to the evaluation design have resulted two forms of implementation barriers. Structural barriers include "unknowns" at the time of proposed plans and relate to "on-the-ground" operational issues including district operational challenges, administrative, school, and teacher communications, general school configuration and scheduling feasible for implementation as planned, hiring and allocation of teachers as well as intervention changes/tailoring for implementation and general issues around screening and placement as well as data documentation and tracking. Classroom implementation barriers include those related to information available from developers regarding model components by which measures can be developed to indicate fidelity, ways in which to weight components to identify levels of implementation, and what how best to indicate fidelity of implementation in general.

Session Title: How we Work: Studies of Evaluation Practice

Multipaper Session 313 to be held in Suwannee 16 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Research on Evaluation TIG

Chair(s):

Christina Christie, Claremont Graduate University, tina.christie@cgu.edu

The Evaluation Field in Brazil

Presenter(s):

Daniel Brandao, Instituto Fonte, daniel@fonte.org.br

Martina Otero, Instituto Fonte, martina@fonte.org.br

Rogerio Silva, Instituto Fonte, rrsilva@fonte.org.br

Abstract: This is a project led by Instituto Fonte e Funda��o Ita� Social which is oriented to strengthening the evaluation field in the Brazilian community and investigates the size and meaning of social program evaluations in Brazil. In 2008, a survey conducted with 211 companies brought data on how this player relates with the evaluation of social projects. A study of the Brazilian academic production on evaluation (Msc and Phd dissertations) was also developed. In 2009 a study on how NGOs relates with evaluation is being conducted. The paper will present data from these studies and aims to contribute with an understanding of how the evaluation field is under development in a Latin American country, showing its opportunities and challenges. The paper can contribute to discussions on the international development of evaluations and is of interest to those committed to international evaluation networks.

Developing a Measure of Organizational Evaluation Capacity

Presenter(s):

Tim Aubry, University of Ottawa, taubry@uottawa.ca

J Bradley Cousins, University of Ottawa, bcousins@uottawa.ca

Swee Goh, University of Ottawa, goh@telfer.uottawa.ca

Catherine Elliot, University of Ottawa, elliott.young@sympatico.ca

Abstract: While knowledge about building the capacity to do evaluation has developed considerably, less is understood about building the organizational evaluation. This presentation reports findings of factor analyses of responses from a pan-Canadian survey of 340 evaluators working in the public sector, not-for-profit sector, or in the private sector on a measure assessing organizational capacity to do and use evaluation. The examined measure was made up of 114 items assessing various elements of contextual circumstances for organizational evaluation capacity, capacity to do evaluation, and capacity to use evaluation. Confirmatory factor analyses will be conducted to determine if factors corresponding to the proposed components for each of these areas are present. These will be followed by principal components analyses to break down the measure into interpretable subgroups of items. Implications of the findings on theory related to organizational evaluation capacity and on using the measure in future research are discussed.

"You Want Me to Do WHAT?" Evaluators and the Pressure to Misrepresent Findings

Presenter(s):

Michael Morris, University of New Haven, mmorris@newhaven.edu

Abstract: Evaluators' experiences of being pressured by stakeholders to misrepresent findings were explored via a survey of a random sample of non-student members of the American Evaluation Association. A response rate of 37% was obtained (N = 940). 42% of the respondents indicated that they had felt such pressure during one or more of their evaluations. The study examines the varieties of misrepresentation that were sought, the types of evaluation that pressure was applied to, the range of stakeholders exerting pressure, evaluators' perceptions of stakeholder intent, the outcomes of influence attempts, and evaluators' views of what could have been done to prevent such pressure from occurring. Implications of the study for understanding the nature of influence in evaluation, as well as for safeguarding the integrity of evaluation reports, are discussed.

Exploring the Practices of School District Evaluators

Presenter(s):

Susan T Hibbard, Florida Gulf Coast University, shibbard@fgcu.edu

Abstract: Little is known about the background, training, and practices of the people conducting evaluations in school districts. This study seeks to answer the calls for research on evaluation, focusing on the evaluation activities carried out in different contexts of school district evaluations. The background of school district evaluators, their practice of evaluation, and the link between training and practice in regards to evaluation theory is explored. Findings of the study highlight the patterns and characteristics of current practicing internal evaluators and variability across those evaluators in different contexts. Interviews, focus groups, and a revised version of the Theory to Practice instrument (Christie, 2001) were used to collect data. Exploring the people who play a role in the decision making process of the public schools is essential for the advancement of education evaluation, evaluation as a profession, and public education.

Session Title: Higher Education and Science, Technology, Engineering and Mathematics (STEM) Careers: Evaluation Challenges

Multipaper Session 314 to be held in Suwannee 17 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Assessment in Higher Education TIG

Chair(s):

Andrea Fuhrel-Forbis, University of Michigan, andreafuhrel@yahoo.com

Discussant(s):

Summers Kalishman, University of New Mexico, skalish@salud.unm.edu

Context, Pedagogy, and Gender: Evaluating Engineering Education

Presenter(s):

Mary Moriarty, Smith College, mmoriart@smith.edu

Abstract: This paper reports on the results of a two year evaluation of the overall operation and impact of the Picker Engineering Program at Smith College. Smith's engineering program was founded in 1999 specifically to respond to the critical challenges of recruiting and training women engineers. The program, now in operation for 9 years has graduated 117 women engineers, 22% of whom identify as women of color. The program provides a unique context in that it is 1) all women, and 2) is situated in a liberal arts institution. There are also a number of challenges associated with the evaluation of a program in such a context with little comparative data. The evaluation focused on defining the nature of the program through an examination of the beliefs and practices of departmental faculty and an examination of how the single-sex environment influence the progression, choice, and growth of women in the engineering?

Outcomes Evaluation for Simulation-Enhanced Education in Healthcare: Kirkpatrick Revisited

Presenter(s):

Kathryn Parker, Hospital for Sick Children, kathryn.parker@sickkids.ca

Kelly McMillen, Hospital for Sick Children, kelly.mcmillen@sickkids.ca

Abstract: Donald Kirkpatrick's outcomes evaluation model (1959) has enjoyed remarkable utility and transportability since its inception. Business, professional development, and health sciences have used variations it to determine the merit of educational programming (Kaufman, Keller, & Watkins, 1995; Molenda, Pershing, & Reigeluth, 1996; Phillips, 1994). Educational researchers, however, argue that the model is flawed; specifically, that the assumptions of the model are problematic (Alliger & Janak, 1989; Bernthal, 1995). The absence of strong inter-correlations and evidence of causal linkages between the levels speak to the need to incorporate learning and behaviour change theories into an outcomes model. The impact of simulation as a pedagogical tool on the learning process is one informed by behaviourist, constructivist, and contextualized learning theories and as such requires an evaluation framework that reflects their theoretical underpinnings. This paper describes an outcomes evaluation model which incorporates components of these theories. Applications of this model are also illustrated.

Transitioning From Industry Into Teaching: Lessons Learned From an Alternative Route to Mathematics and Science Teaching Program

Presenter(s):

Bonnie Swan, University of Central Florida, bswan@mail.ucf.edu

Theresa Becker, University of Central Florida, tbecker@fullsail.com

Abstract: Because of the teacher shortage, new pathways into mathematics teaching are emerging. This study focuses on evaluation results from the fifth year study of Transition to Mathematics and Science Teaching (TMAST), a university-based induction program. TMAST is popular for individuals with interest in becoming teachers as a second career choice, who have backgrounds in the areas of science, technology, engineering and mathematics (STEM). The purpose was to investigate the program's effectiveness, sustainability, a generic model for longitudinal evaluation, and guidance for improving the program. Specific information was sought concerning the impact that TMAST had on teaching effectiveness, professional development, and perceptions of program participants and about how their participation may have affected their students' motivation and their students' achievement, what factors led them into teaching, what surprised them the most, what elements of support they needed the most, and what is needed for them to remain in the teaching field.

A Longitudinal Analysis of Science, Tehnology, Engineering and Mathematics (STEM) Faculty Engagement in a Math and Science Partnership Project

Presenter(s):

Ning Rui, Research for Better Schools, rui@rbs.org

Jill Feldman, Research for Better Schools, feldman@rbs.org

Abstract: Drawing on four years of survey data, the present study explores sources of variance that predict breadth and intensity of engagement among college faculty in a National Science Foundation Math & Science Partnership project during 2003-2007. Building upon descriptive studies of faculty engagement and developmental theories on breadth and intensity as two separable dimensions of engagement, a three-level hierarchical linear model is applied to explain the proportion of variance attributable to temporal, individual, and institutional level variables, respectively. Results indicate that effects of between-person predictors on engagement vary across institutions for intensity but not for breadth. Implications for practice and research about effective engagement of college faculty in similar efforts to reform secondary mathematics and science education are discussed.

Session Title: Using Empowerment Evaluation to Build the Prevention and Evaluation Capacity of Federally-funded Programs

Panel Session 315 to be held in Suwannee 18 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Chair(s):

Rita Noonan, Centers for Disease Control and Prevention, rnoonan@cdc.gov

Discussant(s):

Corinne Graffunder, Centers for Disease Control and Prevention, cgraffunder@cdc.gov

Abstract: The Division of Violence Prevention (DVP) within the Centers for Disease Control and Prevention has utilized an empowerment evaluation approach to build the capacity of three of its Programs since 2002. The Empowerment Evaluation with Programs Designed to Prevent First-time Male Perpetration of Sexual Assault Program focused on building evaluation capacity within local organizations. The Domestic Violence Prevention Enhancements and Leadership Through Alliances (DELTA) Program focuses on building prevention and evaluation capacity within community coalitions. The Enhancing and Making Programs and Outcomes Work to End Rape (EMPOWER) Program focuses on building prevention and evaluation capacity within a state level prevention system. We will explore why DVP pursued an empowerment evaluation approach with these Programs; how the approach worked within each of these Programs; capacity outcomes; successes, challenges and lessons learned within and across these Programs - specifically addressing what was most helpful in achieving increases in prevention and evaluation capacity.

The Centers for Disease Control and Prevention's (CDC's) Empowerment Evaluation With Programs Designed to Prevent First-time Perpetration of Sexual Assault

Rita Noonan, Centers for Disease Control and Prevention, rnoonan@cdc.gov

Deborah Gibbs, RTI International, dag@rti.org

The lofty goal of pursuing empowerment principles sounds appealing, but why should organizations dedicate resources to it and what does that process entail? This presentation will highlight why CDC pursued an empowerment evaluation strategy with 4 sexual violence prevention programs and then outline key steps in the process. Key steps include the engagement of participating organizations in a technical assistance process based on the FORECAST (formative evaluation consultation and systems technique) model. Using FORECAST enabled the programs to develop logic models, identify evaluation markers and measures, interpret meaning from evaluation data, and expand each program model to include broader social ecological perspectives. Tips and tools will be shared with attendees.

Empowering Programs With Evaluation Technical Assistance: Outcomes and Lessons Learned

Deborah Gibbs, RTI International, dag@rti.org

Rita Noonan, Centers for Disease Control and Prevention, rnoonan@cdc.gov

Although empowerment evaluation has gained widespread currency, few reports have described its outcomes. This presentation combines perspectives of participants and technical assistance providers to describe the process and outcomes of CDC's empowerment evaluation with four sexual violence prevention programs. Participating programs reported substantial enhancements in evaluation capacity, resources devoted to evaluation, and the extent and sophistication of their evaluation practice, as well as numerous examples of the application of evaluation findings to program improvement. Experiences from evaluation technical assistance providers identified aspects of the process that were particularly useful in achieving these outcomes, including: the importance of investing in collaborative relationships; maximizing participation among program staff; tailoring the content and form of technical assistance to program preferences; and the value of combining structured learning with program-specific technical assistance.

Building the Evaluation Capacity of Local Coalitions That Address Intimate Partner Violence Through Empowerment Evaluation

Pamela Cox, Centers for Disease Control and Prevention, pcox@cdc.gov

The Centers for Disease Control and Prevention's DELTA Program utilizes an Empowerment Evaluation approach to build the evaluation capacity of the 59 DELTA-funded local coalitions operating in 14 states. Each DELTA Program state has its own empowerment evaluator who is responsible for coaching the DELTA-funded coalitions in increasing their evaluation capacity and evaluating their capacity building progress as well as specific intimate partner violence prevention strategies implemented by the coalition. These empowerment evaluators have coached these local coalitions in needs assessments, assessments of strategy theory and design, process evaluations, and outcome evaluations. Empowerment evaluators have been encouraged not to utilize a one-size fits all model, but to take each local coalition to its next level of capacity based in its current capacity and resources. This presentation will describe some of the success, challenges and lessons learned to utilizing the Empowerment Evaluation approach to build capacity within DELTA-funded local coalitions.

Using an Empowerment Evaluation Approach to Build Prevention System Capacity

Karen Lang, Centers for Disease Control and Prevention, klang@cdc.gov

Using empowerment evaluation (EE) to build individual and organizational capacity is well documented in the literature. Less common is using empowerment evaluation to build system capacity. CDC's EMPOWER Program is designed to build individual and system level capacity for sexual violence (SV) prevention using an empowerment evaluation approach in six states. We define a system as a network of individuals, groups, organizations or subunits that, through their interaction, have the potential to enhance the primary prevention of SV using a public health approach. This presentation will describe the SV prevention system capacity (PSC) framework, how the framework was developed using EE principles, how EMPOWER states assessed their SV PSC and key findings from a cross-site evaluation of SV PSC. Opportunities and challenges related to building SV prevention system capacity will be explored.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Triangulation of Observation Data in a Non-optimal Sample Program Evaluation

Roundtable Presentation 316 to be held in Suwannee 19 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Extension Education Evaluation TIG and the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Marlene Hurley, Empire State College, marlene.hurley@gmail.com

Fernando Padro, Cambridge College, fpadro@msn.com; fernando.padro@cambridgecollege.ed

Abstract: This presentation focuses on creating credible sources of data when evaluating program implementation effectiveness under circumstances in which the given institutional sample is not representative. The first part of the discussion centers on the decision-making process based on a mixed methods design multiple case study approach rather than a meta-analysis while the second part of the paper focuses on observation through different instruments allowing for the creation of both qualitative and quantitative data providing evidence of the extent of implementation and the effectiveness of the implemented model. Instruments used to triangulate findings included a time-lapse analysis, an inventory assessment based on characteristics of implemented model, and traditional student surveys. In addition, instruments were supported by interviews of key individuals to determine perspectives of all individuals involved in the program under review in order to determine personal observation of program success and how instructor perception compares to student participant views.

Roundtable Rotation II: How to Design an Evaluation Plan When There is a Small Number of Participants?

Roundtable Presentation 316 to be held in Suwannee 19 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Extension Education Evaluation TIG and the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Amy Meier, University of Nevada, Reno, meiera@unce.unr.edu

Marilyn Smith, University of Nevada, Reno, smithm@unce.unr.edu

Janet Usinger, University of Nevada, Reno, usingerj@unr.edu

Abstract: Program accountability via evaluation has become increasingly important in the current economic situation. However, certain program designs and circumstances make evaluation difficult. Programs in rural communities have low numbers of participants to collect evaluation data from; and programs that designed to be intensive and target a small group of participants over a long period of time also result in low numbers of participants. This session will begin with an introduction to the evaluation design of Bootstraps, an intensive program in a rural community. The evaluation design includes both qualitative and quantitative methods. Although the evaluation shows that there has been significant behavioral change, there is still an issue of small numbers of participants. This session will explore ideas and practices used for evaluating programs with small numbers of participants. Suggestions of how to use qualitative or quantitative methods to produce reputable program evaluations when there are small numbers will be shared.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Practical and Theoretical Concerns: Words From the American Evaluation Association's Minority Serving Institution Faculty Initiative 2008 Cohort

Roundtable Presentation 317 to be held in Suwannee 20 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Multiethnic Issues in Evaluation TIG and the AEA Conference Committee

Presenter(s):

Jose Prado, California State University Dominguez Hills, jprado@csudh.edu

Deborah M Oh, California State University Los Angeles, doh2@calstatela.edu

Chuan Chang, University of Hawaii at Manoa, chuanc@hawaii.edu

Eulynda Toledo, Dine College, toledobenalli@yahoo.com

Leslie Grier, California State University Fullerton, lgrier@fullteron.edu

Katy Pinto, California State University Dominguez Hills, kpinto@csudh.edu

Abstract: We have two concerns. The first is one of definition and description. In other words, what are the defining characteristics of an evaluative practice that bridges itself to our instructional, scholarly, and service interests? Thus, our roundtable addresses issues tied to race, language, geography, and the academy. For example, we address racist professional and institutional practice in higher education and consider how evaluation might challenge or support these. Such interests stimulate our second concern. Generally, racial minority evaluators are presented with numerous challenges and opportunities. Two are: 1) to exercise an apparently neutral evaluative practice, or 2) to practice an evaluation that examines its discursive properties and that is simultaneously aware of its impact on agendas of the communities we serve. We invite our audience to consider the following questions: Is evaluation a neutral endeavor? How might it adequately address the "authentic" interests of the people served by evaluated institutions?

Roundtable Rotation II: All Things to All People: Experiences and Challenges of an Internal Evaluator at an Historically Black College/University (HBCU)

Roundtable Presentation 317 to be held in Suwannee 20 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Multiethnic Issues in Evaluation TIG and the AEA Conference Committee

Presenter(s):

Stella Hargett, Morgan State University, drslhargett@gmail.com

Abstract: This session will explore the experiences and expectations of an internal evaluator at one Historically Black College/University (HBCU). The multiple roles played by the internal evaluator will be explored as well as the challenges encountered in two distict internal evaluations.

Roundtable: Who is Watching the Watchdog

Roundtable Presentation 318 to be held in Suwannee 21 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Presenter(s):

David Bell, University of Pittsburgh, dab34@pitt.edu

Abstract: Nongovernmental organizations (NGOs) that serve to hold policy makers accountable must also be held accountable. As the influence and level of NGO activity has grown, so have the questions about motivation and legitimacy. Of the various accountability mechanisms, voluntary networks represent a potent, voluntary alternative. This paper will explore 'who is watching the 'watchdog'' through a comparative analysis of voluntary NGO accountability networks. A framework for evaluating voluntary networks as an accountability mechanism is introduced to stimulate research. It is argued that a valid construct for evaluating voluntary networks can inform network selection and use by NGOs. This can result in an effective response to challenges of legitimacy, advancing organizational learning, and providing accountability to various stakeholders. As with democracy; however, effectiveness is significantly influenced by the acceptability of the network's product by civil society, as well as civil society (including key funding sources and donors) encouraging NGOs to participate.

Session Title: Instrumentation Strategies for "Digitally" Interactive Populations

Multipaper Session 319 to be held in Wekiwa 3 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Distance Ed. & Other Educational Technologies TIG

Chair(s):

Theresa Murphrey, Texas A&M University, t-murphrey@tamu.edu

Involvement in Virtual Schooling: The Validation of an Instrument to Measure Parental Involvement Mechanisms

Presenter(s):

Feng Liu, University of Florida, martinlf@ufl.edu

Erik Black, University of Florida, erikwblack@gmail.com

James Algina, University of Florida, algina@ufl.edu

Abstract: Parental involvement has been recognized as an important factor for students' achievement in traditional school settings. The lack of research regarding the effect of parental involvement for students' achievement in virtual schooling is, in part, because of the dearth of one valid and reliable instrument to measure this construct. In this study, one Parental Involvement Mechanisms Model including four factors: parent reinforcement, parent modeling, parent encouragement, and parent instruction, 51 indicators/items was analyzed using confirmatory factor analysis and exploratory factor analysis with the data collected from 938 participants (parents) in virtual schooling. The result shows this instrument is overall a valid measurement. The strong relationship among these four factors provides the evidence that the items tending to measure them might be overlap to some degree. Suggestions were given to modify this instrument for the future research.

Incorporating eeLearning Into the Teaching of Evaluation

Presenter(s):

Theresa Murphrey, Texas A&M University, t-murphrey@tamu.edu

Abstract: Teaching courses online has become a common practice. In fact, universities are reporting increasingly higher numbers of courses being delivered online. Teaching evaluation online offers unique challenges and opportunities for engaging students and encouraging constructive dialog that can improve comprehension of the subject matter. Fortunately, technologies are allowing experiential, electronic learning (eeLearning as coined by Trevitte & Eskow, 2007) as never before. This session will present a literature review of the technologies that are currently available to assist in teaching evaluation and a first-hand account of using eelearning technologies to encourage engagement and team-work. The use of both asynchronous (i.e., discussion boards, email, Jing�, etc.) and synchronous (i.e., Centra�, chat, and phone, etc.) technologies will be discussed in the context of both individual and group assignments.

The Unfolding Model: Using Test Validity to Guide Professional Program Evaluation Studies in Distance Education and E-learning

Presenter(s):

Valerie Ruhe, University of British Columbia, valerie.ruhe@ubc.ca

Bruno D Zumbo, University of British Columbia, bruno.zumbo@ubc.ca

Abstract: In this paper, we present Ruhe and Zumbo's (2008) Unfolding Model, a new model of program evaluation based on Messick's (1989) model of test validity. Based on scientific evidence, relevance, cost-benefit, underlying values and unintended consequences, our model unfolds to reveal practical tools and strategies for diverse technology-based contexts. Our approach brings rich, theoretical insights from test validity into program evaluation and distance education. The model is also a practical tool to guide scientific evaluation studies of innovative instructional programs. With this model, we are responding to recurring calls for a more professional approach to the evaluation of innovative instructional programs. Our model has been tested on four authentic post-secondary courses in diverse subject areas, and is especially helpful for novice evaluators. Finally, its adaptive and dynamic quality ensures its relevance for Web 2.0 and beyond.

Session Title: Degrees of Separation: Monitoring and Evaluation in Conflict Zones

Panel Session 320 to be held in Wekiwa 4 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Alice Willard, International Relief & Development, awillard@ird-dc.org

Abstract: M&E in conflict zones has developed a number of "good enough" practices designed to balance data reliability with the safety of both the beneficiaries and the project M&E staff. This type of remote-controlled M&E means that there are fewer external actors implicated with data collection, and that the basic tools of monitoring need additional reinforcement to assure data quality. Remote-controlled M&E means that the methods and safety of the M&E team exist in an uneasy equivalence--methods with the highest validity and reliability are simply impossible when the safety of both the respondents and the staff cannot be guaranteed. The following three panelists provide varying solutions to this difficult balancing act, where the best method is frequently the one that has the most physical distance between the project management and the community; and where the most reliable information is from triangulating proxies.

International Relief and Development (IRD) Community Stabilization Program (CSP) and Community Action Program (CAP) in Iraq: Critical Role of Building National Staff

Mamadou Sidibe, International Relief & Development, msidibe@ird-iq.org

These IRD Iraq programs approach M&E in different ways. In part, this is due to the scale and organizing paradigm of these two large-scale projects, and due to the different environment of operating in Iraq from 2004-2009 (pre and post-surge). Both projects are national in scope. The CSP's activities tend to cluster near urban areas and focus on small-medium enterprise renewal. CAP, by contrast focuses on improving the capacities of smaller communities to begin to affect the enabling environment of civil society. Both projects include a small M&E team of trained staff that report to the head of the project, with staff posted in an uneven ratio to monitor the progress of community mobilizers. This paper examines the different mechanisms in place to monitor progress, and the particular dynamics between the M&E staff and the mobilizers, and the role of an external contractor in supervising M&E for all implementing partners.

International Relief and Development (IRD) Strategic Provincial Roads-Southern and Eastern Afghanistan (SPR-SEA) and Afghanistan Vouchers For Increased Production In Agriculture (AVIPA) Projects in Afghanistan: Critical Role of Training and Working With the Community

Alice Willard, International Relief & Development, awillard@ird-dc.org

IRD currently operates two major projects in Afghanistan: 1) road building and 2) relief seed distribution. As with the IRD Iraq portfolio, these projects include a core of trained M&E staff who report directly to the project on community outreach. These programs are distinguished from the Iraq programs by the role of the technical field staff in data collection and extensive involvement of both the M&E and technical staff in relief distribution and construction. The communities' involvement is critical to both. How the M&E teams manage the data collection varies not only by virtue of the types of information required by the donor, but also the community footprint. The challenge with both is to ensure that the information is reliable no matter the timing, and how to engage community participation without a consistent local presence. This presentation focuses on the patterns for safety and reliability given variable presence and access.

Africare's Longitudinal Food Security Programming in Niger and Chad: Critical Importance of Building Flexible Systems

Delia McMillan Wilson, Independent Consultant, dellamcmillan@aol.com

Africare has been implementing food security and health programs in Chad for 25 years and Niger for the last 39 years. During which, the state of Chad and Niger has varied with both international and regional unrest, making their M&E systems necessarily flexible to keep pace with the variable access to the communities they serve. In some cases this has meant finding ways that community-based volunteers and staff could collect the M&E data and report. Africare has integrated many of its data collection efforts into its model for community capacity building which includes a variety of community self-assessment and planning tools. This presentation focuses on various factors that contributed to and detracted from the utility of these tools in sustaining the project's M&E activities during periods of access and non-access to the project sites. The presentation will also highlight the critical importance of developing longstanding relationship with beneficiaries.

Session Title: How to Do Participatory Impact Pathways Analysis

Skill-Building Workshop 321 to be held in Wekiwa 5 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

Boru Douthwaite, Challenge Program on Water and Food, bdouthwaite@gmail.com

Sophie Alvarez, International Center for Tropical Agriculture, b.sophie.alvarez@gmail.com

Abstract: In this workshop, participants will be introduced to Participatory Impact Pathways Analysis (PIPA) and develop impact pathways for their own program. PIPA is a practical planning and evaluation approach fast being adopted for use with complex programs in the water and food sectors (see http://impactpathways.pbwiki.com). PIPA begins with a participatory workshop where stakeholders make explicit their assumptions about how their program will achieve impact. Participants construct problem trees, carry out a visioning exercise and draw network maps to help them clarify their program theory in the form of 'impact pathways'. Impact pathways describe which actors need to change to achieve the program vision, what are those changes and which strategies are needed to make them happen. PIPA goes beyond the traditional use of logic models and logframes by engaging stakeholders in a structured participatory process, promoting learning and providing a framework for 'action research' on processes of change.

Session Title: Assessment of Emergent Technologies: The Case of Nanotechnology

Multipaper Session 322 to be held in Wekiwa 6 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Research, Technology, and Development Evaluation TIG

Chair(s):

Juan Rogers, Georgia Institute of Technology, jdrogers@gatech.edu

Abstract: The papers in this panel will address several complementary issues in the assessment of nanotechnology as an emerging field. The first paper will present new metrics and maps of the field using new techniques to follow the evolution of its content and links across the scientific community. The second will present a bibliometric approach to track the impact of various funding streams to the field of nanotechnology. The third will present the results of a mixed-methods approach to establish the development path and uses of a specific strand of nanotechnology, namely, active nanostructures. The final paper will present an analysis of the dynamics of distributions of publications and citations in the field to understand its underlying probability structure. Taken together, these papers present a set of new tools to assess emerging fields since the approaches themselves are not limited to this particular field.

Assessing Nanotechnology: Research Metrics and Maps

Alan Porter, Georgia Institute of Technology, alan.porter@isye.gatech.edu

New and Emerging Science & Technology ["NES&T"] research raises special evaluation challenges. Many believe that interdisciplinary processes are essential to advance such areas. However, assessing interdisciplinarity poses conceptual, data, and analytical challenges. We illustrate for selected nanoscience & nanoengineering ["nano"] topics. Our Integration score improves upon alternative metrics (e.g., Shannon & Herfindahl diversity indices), as it addresses all three components of diversity - variety, balance, and disparity. The Integration score is calculated from the distribution of a paper's (or set of papers) cited journals, from which we get associated Subject Categories (Web of Science), and cluster those into macro-disciplines. Further using this information, we compose a family of visualizations that provide multiple perspectives on the nature of the research under scrutiny. These include: intellectual and social research networks , science overlay and citation link maps, and research coherence maps. The metrics and maps provide rich insights into research practices and impacts.

Who is Funding Nanotechnology and How? Evidence From Publication Analysis

Jue Wang, Florida International University, jwang@fiu.edu

The US government has been playing an important role in promoting the development of nanotechnology by setting up R&D programs and providing research funding. An annual budget of $1.5 is allocated to nanotechnology through the National Nanotechnology Initiative (NNI) in recent years. With such a large amount of government investment, it is interesting and also important to learn how the money is spent and what is produced from it. This study seeks to probe the funding pattern of federal agencies and their contribution to scientific development of nanotechnology. Using acknowledgement analysis, this study links publication records to their respective funding agencies. It compares the impact of these funding agencies by investigating the quality of funded research, the amount of research output, its specialization field, recipients and their institutions.

Assessing the Development Path of "Active Nanotechnology"

Vrishali Subramanian, Georgia Institute of Technology, vsubramani6@gatech.edu

Regulating nanotechnology requires understanding whether actual technology developments follow the path intended by funding policies. We focus on the case of "active nanotechnology" as a case a point. The NSF Active Nanostructures and Nanosystems grant solicitation defines an active nanostructure as "An active nanostructure changes or evolves its state during its operation," and gives the following examples: nanoelectromechanical systems, nanomachines, self-healing materials, nanobiodevices, transistors, amplifiers, targeted drugs and chemicals, actuators, molecular machines, light-driven molecular motors, plasmonics, nanoscale fluidics, laser-emitting devices, adaptive nanostructures, energy storage devices, and sensors. With this definition we develop a search strategy to bound active nanotechnology and establish the patterns of development. We perform bibliometric analysis and describe the main publication trends. We chose some novel technology prototypes from these and interviewed scientists in order to understand how they were developed and how they might be commercialized. The results are compared to the given policy rationale.

Publication and Citation Distributions in the Development of Nanotechnology

Juan Rogers, Georgia Institute of Technology, jdrogers@gatech.edu

Nanotechnology has emerged recently as an interdisciplinary field and the question arises as to field specific patterns of its citation and publication distributions. The upper tail of typical distributions of citations per paper and publications per author follow power laws which are scale free and, therefore, self-similar for natural aggregations of papers or authors. In this paper, we will explore the dynamics of these distributions for nanotechnology from 1990 to find any special patterns given the interdisciplinary nature of the field. We will model the citation process as a random graph and the publications per author distribution as an Information Production Process (IPP) to determine the evolution of the distributions in time. The result will be a complete specification of the probability structure of citations and publications in this interdisciplinary field which provides the basis for interpretation and benchmarking of most bibliometric indicators of research performance.

Session Title: Teams, Collaborative Projects and Networks: Evaluating Them and Their Impact

Multipaper Session 323 to be held in Wekiwa 7 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Research, Technology, and Development Evaluation TIG

Chair(s):

Cheryl Oros, Independent Consultant, cheryl.oros@comcast.net

Networking for Results: Assessing Collaborative Projects on Science and Technology

Presenter(s):

Ruslan Rakhmatullin, TCD, School of Business Studies, rakmator@gmail.com

Louis Brennan, Trinity College, brennaml@tcd.ie

Abstract: This paper will attempt to provide insights into advantages and possible consequences of the pre-competitive scientific networking. Although there is some general consensus that increasing levels of collaboration amongst academics produce 'better' results, and which generally mean higher quality, the issue of collaboration and its impact on research outcomes remain an uncertain area with a wide variety of views of what roles such collaboration plays and its general implications for scientific performance . The research paper seeks to address this gap in existing literature by answering the following research question: Can some of the research outputs and outcomes be attributed to researcher's participation in scientific network activities? , or is it just a bureaucratic novelty? This study brings together the fields of evaluation theory and performance management by investigating creation of added value through pre-competitive scientific networking. As such formal scientific networking is touted and perceived as an added value activity, this research attempts to search for evidence that will either support or reject this perception. This research paper consults relevant literature to build a theoretical framework to reflect the objectives of the proposed research, which are to understand whether some of the scientific outputs and outcomes can be attributed to researchers' participation in formal networking projects and whether such participation is reflected in improved performance.

Review of Research Teams in Chinese Academy of Sciences: Identify the Effective and Efficient Team Building

Presenter(s):

Jiaofeng Pan, Chinese Academy of Sciences, pjf@hq.cas.ac.cn

Qiang Li, Chinese Academy of Sciences, lq@casipm.ac.cn

Bing Shi, Chinese Academy of Sciences, bshi@cashq.ac.cn

Abstract: Based on self-evaluation worksheets of research institutes in CAS, this study analyzed the personnel structure of PI teams and their disciplinary layout or research activities and identified four possible types of research teams. This study also reviewed typical possible teams in typical institutes by quantitative assessment, interview with researchers and managerial staff, expert workshop as well as self-cognition of institutes, and achieved rational understanding of team building status in institutes. This study also investigated the effectiveness and efficiency of team building and put forward suggestions on classified administration of different teams.

Applying Network Analysis to Science and Technology Development: Examples From the European Commission (EC Evaluation Part 2)

Presenter(s):

Francis Cunningham, European Commission, frank.cunningham@ec.europa.eu

Franco Malerba, University of Bocconi, franco.malerba@unibocconi.it

Caroline Wagner, SRI International, caroline.wagner@sri.com

Marko Grobelnik, Josef Stefan Institute, marko.grobelnik@ijs.si

Nicholas Vonortas, George Washington University, vonortas@gwu.edu

Abstract: Traditional approaches to evaluating research and deployment programs have focused on inputs, outputs and outcomes and have left untouched the dynamics within the interactions. The European Commission's DG Information Society and Media has been at the forefront in developing methodologies and tools to assess the 'behaviour additionality' of the EU RTD intervention in terms of linkages and networks. Collaborative programs such as the EU's 7th Framework Program ICT research and the ICT Policy Support Program of Competitiveness and Innovation Framework Program aim to strengthen the knowledge bases of research and innovation actors and enhance the prospects for subsequent exploitation and innovation. New methods are needed to represent the impact of these programs: network analysis offers a great deal of promise towards this goal. This panel presents evaluation based on network analysis (2004-2008) sponsored by the EC. It explains how the work is contributing to policy and planning.

Session Title: Discussing a Multi-Site Collaborative Evaluation Model to Assess Academic Pipeline Programs (K-20) Designed to Increase the Numbers of Individuals From Disadvantaged Backgrounds Into Professional Health and Biomedical Sciences Careers

Panel Session 324 to be held in Wekiwa 8 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the College Access Programs TIG

Chair(s):

Lourdes M Rivera, University of California San Diego, l1rivera@ucsd.edu

Discussant(s):

Lawrence Hy Doyle, University of California Los Angeles, hydoyle@ucla.edu

Abstract: Institutions of higher education recognize the need to train and support more students from underrepresented and disadvantaged backgrounds in the Health and Biomedical Sciences. The University of California, San Diego (UCSD), Los Angeles (UCLA) and San Francisco (UCSF Fresno campus) conduct several pipeline programs that address this issue. Evaluation specialists from these universities will discuss the need for a more uniform theory-based model to assess such programs. This discussion will focus on current evaluation methods such as the use of focus groups, surveys, logic models, the Delphi study approach, reflective practices, and intricate student tracking systems. The goal of the discussion is to develop strategies for integrating current approaches to map several pathways to careers in the Health and Biomedical Sciences, thus creating a unified model to determine how programs could best tailor their resources to support vulnerable students in different pathways.

University of California San Francisco's (Fresno Campus) Pipeline Programs: Use of Logic Model and Long-Term Student Tracking to Measure the Impact of the Program

Andrew Alvarado, University of California San Francisco, aalvarado@fresno.ucsf.edu

Katherine Flores, University of California San Francisco, kflores@fresno.ucsf.edu

The Latino Center at UCSF-Fresno has developed and implemented an educational pipeline program spanning from middle school to university designed to increase the numbers of individual from disadvantaged backgrounds into professional health careers. The evaluation of the pipeline utilizes a logic model that illustrates the comprehensive assessment of a series of interventions offered to students throughout each educational level by measuring 1) Inputs; 2) Throughputs or Processes; 3) Intermediate Outcomes; and 4) Sustained Outcomes or Impacts. Each segment of the learning pathway is evaluated using both process (formative) and outcome (summative) measures using qualitative and quantitative methods. Attendees will learn how to comprehensively capture the experience of disadvantaged students and their families while they are in such programs as well as measure the impact of several program components through long-term student tracking.

University of California Los Angeles Pipeline Programs: Use of Reflective Practice to Evaluate Programs That Improve Disadvantaged Student's Skill Levels Significantly Within the Limited Term of the Programs

Lawrence Hy Doyle, University of California Los Angeles, hydoyle@ucla.edu

Sebastian Uijdehaage, University of California Los Angeles, bas@mednet.ucla.edu

University of California Los Angeles' (UCLA) pipeline programs include a range of disadvantaged participants from first year undergraduates to medical students. The pipeline consists of a five-year dual degree program designed to develop the skills of leaders in healthcare for underserved communities; post-baccalaureate programming for students who have not yet been accepted to medical school; and health disparity curricula and small group learning for pre-med and pre-dental undergraduates. Formative evaluations using Delphi studies and reflective practice have allowed practitioners to gain consensus on values and standards to structure the programs, their admissions, and the means of assessment to match the agreed upon standards. Panel attendees will have an opportunity to gain insight into practice of the cutting-edge admissions process, the development of curriculum that takes students from surface learning to more sophisticated learning strategies, and work that improves student skill levels significantly within the length-limited term programs.

University of California San Diego's Efforts to Evaluate Pipeline Programs: Use of Innovative Systematic Tracking to Measure Program Success

Lourdes M Rivera, University of California San Diego, l1rivera@ucsd.edu

Jose Cruz, University of California San Diego, j3cruz@ucsd.edu

Sandra Daley, University of California San Diego, sdaley@ucsd.edu

The University of California, San Diego (UCSD) School of Medicine designs programs with specific performance objectives and an evaluation component that prepares students from disadvantaged backgrounds for careers in the Health and Biomedical Sciences. UCSD pipeline programs target middle/high school, community college, undergraduate and post-baccalaureate students. In order to measure program success, UCSD utilizes logic models to identify short- and log-term outcomes within a theoretical framework as well as an innovative student tracking system that begins during the program and extends to up to 10 years after student participation. The tracking system incorporates student and family orientations, modern forms of communication and social networking technologies, tracking specialists, online tracking tools, and other strategies. Attendees will learn about framing evaluation systems around several programs with similar objectives occurring at different levels of the educational pipeline with a particular focus on long-term outcomes. The need for developing tracking standards will be discussed.

Session Title: Evaluating Energy Efficiency

Multipaper Session 325 to be held in Wekiwa 9 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Environmental Program Evaluation TIG

Chair(s):

William Michaud, SRA International Inc, bill_michaud@sra.com

Evaluation of the United States Environmental Protection Agency's Energy Star Labeling Program and Reported Energy Savings

Presenter(s):

Jerri Dorsey, United States Environmental Protection Agency, dorsey.jerri@epa.gov

Gabrielle Fekete, United States Environmental Protection Agency, fekete.gabrielle@epa.gov

Abstract: The U.S. Environmental Protection Agency's (EPA) ENERGY STAR program is a voluntary energy efficiency program. In 2006, EPA reported that using ENERGY STAR products prevented greenhouse gas emissions equivalent to those from 23 million vehicles, and saved Americans $12 billion on their utility bills. The EPA's Office of Inspector General (OIG) evaluated both how effectively EPA is managing the ENERGY STAR Product Labeling Program, and the validity and accuracy of the overall programs reported energy savings. The OIG found EPA lacks reasonable assurance that the ENERGY STAR self-certification process is effective. There is little oversight in using the program's label in retail stores. Using the label on products that do not meet ENERGY STAR requirements may weaken the label's value and negatively impact the program. The OIG found the program's savings claims were inaccurate. Deficiencies included the lack of a data collection quality review; and reliance on unverified estimates, forecasting, and third-party reporting.

Counting on Market Intelligence: When the Experts are Wrong

Presenter(s):

Anne West, Cadmus Group Inc, anne.west@cadmusgroup.com

Ben Bronfman, Cadmus Group Inc, ben.bronfman@cadmusgroup.com

Shahana Samiullah, Southern California Edison, shahana.samiullah@sce.com

Abstract: Can we believe market experts when structuring energy efficiency program evaluations? How does it impact evaluation efforts if industry experts are wrong about their target market? In actual practice, industry market actors may not really possess the market intelligence necessary to understand the market and true potential of their program offering. Market actors, who should have the intelligence, have been wrong. Proposers may be fascinated with their solution to a technical problem, but may be weak on market characterization and did not find the right business model to showcase their technology. Evaluating programs that rely on industry experts' market assessments requires an evaluation design that can unmask misunderstandings of the situation and point out mistakes to prevent similar mistakes. This paper explores several programs where experts did not fully understand their market, the impact on evaluation efforts, and, suggests an evaluation designed to identify errors early in the program cycle.

Session Title: Thinking Critically About Masters' Education in Evaluation: Experiences From an Education Program in Sweden

Multipaper Session 326 to be held in Wekiwa 10 on Thursday, Nov 12, 1:40 PM to 3:10 PM

Sponsored by the Teaching of Evaluation TIG

Chair(s):

Ove Karlsson Vestman, Malardalen University, ove.karlsson@mdh.se

Abstract: In 2008, we initiated a masters program in evaluation at our two universities - Malardalen and Orebro Universities in Sweden. The program, like many masters' program, is designed to support the development of evaluation practitioners. Our experiences to date have generated a number of issues related to designing a masters-level education in evaluation. These issues include, for example, important prerequisites for students; the nature and role of evaluation theory that should be presented; the order in which theory and practice are presented, and how they can be integrated in the program design; and how best to introduce students to evaluation as a profession. This paper session will engage these issues through reflections on our program design, student perceptions gathered from our first-year students and modest comparisons to other masters' programs in evaluation.

Reflections on the Masters' Program Design

Kari Jess, Malardalen University, kari.jess@mdh.se

Osman Aytar, Malardalen University, osman.aytar@mdh.se

The original design of our masters' program was based on reasoned decisions about what kinds of students would likely be interested in this program and what kinds of knowledge and skills evaluation practitioners need to be credible and to conduct defensible evaluations in contemporary society? After one year of program operation, we are now questioning some of these decisions. The student profile, for example, was not as expected, and a number of students have withdrawn from the program, claiming that it was 'too theoretical.' This presentation will use student feedback data and faculty reflections to share experiences with our program as designed and as implemented. The presentation will concentrate on the most salient program design issues identified in our data from students and our own critical reflections.

Perspectives on Theory and Practice in Master's Education in Evaluation in Relation to Social Work

Lars Oscarsson, Orebro University, lars.oscarsson@oru.se

Christian Kullberg, Orebro University, christian.kullberg@oru.se

Theory-practice tensions are common to all fields of professional practice, and thus to the educational preparation of professionals in that field. Evaluation is no exception. And if you combine evaluation with social work, another professional field characterized by such a tension, you have an educational context including not only theoretical and methodological issues, but also dimensions of ethical and political nature; this since social work practice as well as evaluation includes, and take place within, a societal context of political, professional, and consumer ambitions and expectations. So, in our masters' program in evaluation, we have had to engage with questions like, the following. Should we start our instruction with evaluation theory or practice, or social work practice? What kind of connections between theory and practice in evaluation and in social work, should our program foster, and how can we best do this? What kinds of challenges to the conventional separation of theory and practice are important for students aspiring to be evaluation practitioners in social work in view of the political and ethical aspects of that societal arena and professional field? This paper will raise these kinds of issues about the ongoing theory-practice relationship as relevant to our masters' program design and implementation.

Comparisons of Ideas in Evaluation Masters' Programs

Ove Karlsson Vestman, Malardalen University, ove.karlsson@mdh.se

Georgie Parry-Crook, London Metropolitan University, g.parry-crooke@londonmet.ac.uk

Using information on the web, selected literature and presentations on evaluation masters' program at the 2008 European Evaluation Society conference, this presentation will begin to draw comparisons between the program developed and used at Malardalen University and �rebro University to other programs for master education in evaluation. Some of the question that will be addressed is what target groups are the programs trying to reach? Is it policy and program planners, managers or consultants, teachers, nurses, social workers and other practitioners? What aims for the programs are the most common? Is it to produce graduates who are able to manage assessment and evaluation programs at a high level in the organization or is the aim more towards practitioners who want to learn more about how they can evaluate their own practice? What content and curriculum characterize the program? How are for example the relations between courses in evaluation theory and methods and practical experience handled? Is the program given on line or on campus?

Return to Evaluation 2009 Search Results for All Sessions