Search Results for All Sessions

Return to search form

Session Title: An Evaluation Capacity Building Cafe: Conversations With 17 Evaluators

Think Tank Session 101 to be held in Centennial Section A on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

Presenter(s):

Hallie Preskill, Claremont Graduate University, hpreskill@ca.rr.com

Shanelle Boyle, Claremont Graduate University, shanelle.boyle@gmail.com

Discussant(s):

Luba Botcheva, Children's Health Council Outcomes Research Consulting Service, lubabb@yahoo.com

PeiYao Chen, TCC Group, pchen@tccgrp.com

Iris Smith, Emory University, ismith@sph.emory.edu

Chantell Johnson, TCC Group, cjohnson@tccgrp.com

Allison Crean, TCC Group, allisoncrean@earthlink.net

Sheridan Green, JVA Consulting LLC, sheridan@jvaconsulting.com

Michele Graham, JVA Consulting LLC, magraham@kpmg.com

Tom Grayson, University of Illinois Urbana-Champaign, tgrayson@uiuc.edu

Leslie Goodyear, Education Development Center Inc, lgoodyear@edc.org

Jeehae Ahn, University of Illinois Urbana-Champaign, jahn1@uiuc.edu

Srik Gopalakrishnan, The Ball Foundation, srik@ballfoundation.org

Barbara Iverson, The Ball Foundation, biversen@ballfoundation.org

Ellen Taylor-Powell, University of Wisconsin Extension, ellen.taylor-powell@ces.uwex.edu

Nancy Brooks, University of Wisconsin Extension, nancy.brooks@ces.uwex.edu

Barbara E Merrill, Neighborhood House, bmerrill@neighb.org

Jean King, University of Minnesota, kingx004@umn.edu

Maureen Wilce, Centers for Disease Control and Prevention, mwilce@cdc.gov

Abstract: Evaluation capacity building (ECB) has become a hot topic of conversation, activity, and study. Seeking to enhance stakeholders' understanding of evaluation concepts and practices, and in an effort to create evaluation cultures, many organizations have been designing and implementing a variety of strategies as a means of helping their members learn about and engage in evaluation practices. Participants will have the opportunity to interact and learn from 17 evaluators who have been involved in evaluation capacity building projects and who are engaged in an ECB research study. In this modified 'World Café,' each ECB evaluator will have three 15-minute opportunities to discuss their work with others at round tables. The presenters will provide their table participants with a one-page handout that describes their ECB work; all attendees will receive a handout that summarizes each of the presenters' ECB efforts. Findings from an ECB research study will also be presented.

Session Title: Social Network Analysis for Evaluation of Institutional Culture

Demonstration Session 102 to be held in Centennial Section B on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Presenter(s):

Stephen Lurie, University of Rochester, stephen_lurie@urmc.rochester.edu

Abstract: Using actual data from evaluation of an NIH-funded Clinical Translational Sciences Institute, the presenter will introduce basic concepts of social network analysis, including cohesion, betweenness, brokerage, partitions, vectors, 1- and 2-mode networks, and data layout. The presenter will demonstrate these concepts with Pajek, a freely available software package, for performing these analyses. Thus, attendees will obtain an orientation to this software, which should allow them to perform basic analyses, and also to facilitate further self-study. Finally, the presenter will contrast social network analysis with cluster analysis and multidimensional scaling, and demonstrate how these tools can be used to enrich data presentation obtained from social network analysis.

Session Title: A Portfolio of Federal Evaluation Policy Case Studies

Panel Session 103 to be held in Centennial Section C on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Presidential Strand

Chair(s):

Stephanie Shipman, United States Government Accountability Office, shipmans@gao.gov

Abstract: Despite extensive discussion within AEA of a few highly visible federal evaluation policies, there is no overarching policy that defines what evaluation is or how it should be done in the federal government. Instead, federal program evaluation grew over the past decade in a highly decentralized manner until it was recently spotlighted by OMB's Program Assessment Rating Tool (PART). As program evaluation expanded to agencies and subject areas beyond the social services, agencies developed their own methods and policies to meet government-wide accountability demands as well as their programs' individual needs and circumstances. In this panel, three federal agency officials will describe key aspects of their agencies' evaluation policies, the purposes those policies serve, and how they came about. A fourth panelist will describe the challenges encountered as OMB attempted to create government-wide evaluation policy through the PART.

Evaluating Welfare Reform Demonstrations: Federal, State and Evaluator Partnerships

Mark Fucello, United States Department of Health and Human Services, mfucello@acf.hhs.gov

Investing in evaluation to produce credible information about well-conceived welfare reform and self-sufficiency interventions for families living in and near poverty requires complex, collaborative planning among levels of government and the professional evaluation community. The Administration for Children and Families (ACF) in the Department of Health and Human Services has a history of creating partnerships with human services agencies and research firms to foster rigorous evaluations of policy and program alternatives. Evaluation collaborations begun in the 1980s that led to the Temporary Assistance to Needy Families (TANF) program in 1996 through current initiatives following the reauthorization of TANF in 2005 contribute to a body of rigorous research on a range of family self-sufficiency questions affecting public assistance policies and program strategies. The goal of ACF's welfare reform evaluation agenda has been to increase knowledge about the effectiveness of programs aimed at helping low-income families achieve self-sufficiency.

Planning Evaluation to Improve the Effectiveness of Department of Labor Programs

Richard French, United States Department of Labor, french.richard@dol.gov

As Director of the Program Planning and Results Center, Mr. French is responsible for providing Departmental leadership, direction, policy advice, and technical assistance on all aspects of strategic and performance planning, including managing the Department's compliance with the Government Performance and Results Act of 1993. In this role, he also oversees and manages a multi-million dollar budget that is dedicated to conducting annual evaluations of selected Department of Labor (DOL) programs. Mr. French will discuss the history of these evaluations; the process DOL utilizes in selecting programs for evaluations; and the impact that evaluations have made on improving the effectiveness of DOL programs.

Reflections on Efforts to Meet Executive and Congressional Requirements and Implications for Future Federal Evaluation Policy

Cheryl J Oros, United States Department of Veterans Affairs, cheryl.oros@va.gov

The presentation will provide an overview of evaluation experiences for a federal research agency and its current and planned evaluation policies. Included in the overview will be the history of evaluation practices and policies, targets of evaluation policies, and the purposes they serve for both internal decision making and for external accountability, including the PART and GPRA. Trends and differences in favored evaluation approaches and coordination with other federal science agencies will be noted and discussed. More in depth analysis of these experiences will be provided based on interviews with heads of evaluation units and PART coordinators. Efforts to encourage the development of new evaluation tools and methodologies, especially in collaboration with other federal entities will be discussed. A set of recommended options will be presented for evaluating federal research programs with a specific focus on how programs could benefit from regular evaluations.

OMB's PART as an Effort to Create Government-wide Evaluation Policy

Stephanie Shipman, United States Government Accountability Office, shipmans@gao.gov

OMB created the Program Assessment Rating Tool (PART) to fulfill the goal of GPRA: that program performance should better inform program management and policy making. PART not only represents an effort to bring sustained attention to federal management reforms but also a dramatic increase in expectations for federal evaluation activity: that all programs should have comprehensive effectiveness evaluations. However, several features of federal programs and agencies have stymied OMB efforts to use PART as a standardized approach to conducting a thorough assessment of all federal programs. The diversity and complexity of the federal environment has also stymied efforts to create a government-wide policy for federal agency evaluation. This presentation will discuss issues identified in GAO reviews of the first couple of years' experience with the PART, as well as issues raised in continuing discussions among the federal evaluator community.

Session Title: The Higher Education Assessment Agenda: Focusing on Student Learning Outcomes

Multipaper Session 104 to be held in Mineral Hall Section A on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Assessment in Higher Education TIG

Chair(s):

Elizabeth Kelly, Independent Consultant, ekelly.work@gmail.com

Discussant(s):

William Rickards, Alverno College, william.rickards@alverno.edu

A Systematic Approach to Assessing Student Learning Outcomes

Presenter(s):

Jennifer Reeves, Nova Southeastern University, jennreev@nova.edu

James Pann, Nova Southeastern University, pann@nova.edu

Abstract: Assessing student learning outcomes has become challenging for institutions of higher education, especially with the recent push from accrediting bodies to require academic programs to articulate measurable expected outcomes of student learning, and to document how each outcome will be assessed. NSU established its Academic Review process more than five years ago; since that time the process has continually been refined to meet both the needs of the university and the recommendations of SACS. The approach has six steps: (a) Program Outcomes and Measures Matrix (i.e., curriculum mapping), (b) internal assessment of student learning, (c) an Internal Review Committee report, (d) an external consultant report, (e) the Academic Review Committee report, and (f) action planning by the center dean based on the recommendations. The purpose of this paper is to share with other individuals responsible for assessment at institutions of higher education an effective method to assess student learning outcomes.

Evaluating the Impact of Web Based Resources on Student Learning and Satisfaction: Lessons Learned for Postsecondary Courses

Presenter(s):

Melinda Hess, University of South Florida, mhess@tempest.coedu.usf.edu

Corina Owens, University of South Florida, cowens@coedu.usf.edu

Autar Kaw, University of South Florida, kaw@eng.usf.edu

Abstract: As the call for accountability in education, as well as other fields, continues to grow, instructors at all levels are answering by expanding their pedagogical toolbox. In order to provide enhanced learning environments, many faculty have implemented web-based learning, thus posing new challenges for appropriate and effective evaluation. The unique nature of web-based instruction, whether as a primary or supplemental means of instruction, requires assessment of content knowledge and skills as well as course presentation and delivery. Findings of the study indicate that web-based modules implemented in a Numerical Methods course in a College of Engineering for instruction improved both student performance and satisfaction. This paper focuses on the framework and evaluation methods used in this study, including: 1.) assessment of student performance on a multiple-choice examination as well as a recently developed concept inventory, 2.) student satisfaction with course components and delivery, and 3.) faculty assessment of developed modules.

A Longitudinal, Mixed-Method Design for Assessing College Students’ Civic and Political Engagement

Presenter(s):

Rachael Tan, Schroeder Measurement Technologies, jinbee@yahoo.com

Dawn Terkla, Tufts University, dawn.terkla@tufts.edu

Abstract: Recently, there has been an increased push for colleges and universities to prepare students for lives of active citizenship. This presentation will highlight the mixed-method approach Tufts University developed to measure their success at cultivating “active citizens” through the University College Scholars Program. An annual survey provides an overview of students’ civic and political attitudes and activities, and interviews conducted with select participants during their sophomore and senior years allow for an in-depth examination of how students formed their opinions and why they participated in these activities. The longitudinal design allows for changes in civic engagement to be tracked over participants’ four years of college and two years post- graduation, and enables results from each year to be used in a formative manner to improve the study design and program elements, and provide summative information to stakeholders.

Session Title: Creating a Culture of Process Improvement in the Human Services: South Carolina T.E.A.C.H. Project Experience

Demonstration Session 105 to be held in Mineral Hall Section B on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Human Services Evaluation TIG

Presenter(s):

Joyce Miller, KeyStone Research Corporation, joycem@ksrc.biz

Bruce Carnohan, KeyStone Research Corporation, brucec@kbsc.biz

Tania Bogatova, KeyStone Research Corporation, taniab@ksrc.biz

Abstract: This workshop provides an overview of a process improvement methodology, You Get What You Design (sm) and its impact on the operations of the South Carolina T.E.A.C.H. Early Childhood« Project. The methodology was used to map and analyze work processes to identify areas of waste and to eliminate non-value added work, with focus on achieving the goals of the Project and providing the best service possible to the "super-client" - the child care professionals applying for T.E.A.C.H. scholarships. The session will highlight the specific work processes mapped and analyzed, the action plans established, the improvement implemented, and performance measures to track changes and overall impact of the process. The participants will learn about process improvement methodology, You Get What You Design (sm); practice using this methodology on a sample work process; understand potential for using this process improvement methodology in the human services; see the impact of the changes made in the operation of the SC T.E.A.C.H. program.

Session Title: Online Evaluation Resources: Knowledge Sharing for the Field

Panel Session 106 to be held in Mineral Hall Section C on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Advocacy and Policy Change TIG

Chair(s):

Astrid Hendricks Smith, The California Endowment, ahendricks@calendow.org

Abstract: This session will highlight three novel online resources designed to facilitate the planning and evaluation of policy change and advocacy efforts. These free or low-cost resources help advocates design an evaluation, assess the effect of their work, and strategically communicate their findings. The three presenters will provide live walkthroughs demonstrating the features and capabilities of each resource.

Point K Learning Center

Johanna Gladfelter, Innovation Network Inc, jgladfelter@innonet.org

Innovation Network's Point K Learning Center (www.innonet.org/pointk) makes advocacy evaluation resources available for free to anyone with internet access. Aimed at evaluation novices and experts alike, our collection offers workbooks, tip sheets, reports, articles, links, and more. We have been particularly focused on offering resources from multiple perspectives--evaluators, advocates, and funders. Innovation Network started putting free advocacy evaluation resources online over two years ago. We began with about forty resources, presented on static HTML pages (users had to scroll through to find what they were looking for). Since then, the number of resources in the system has more than tripled. In February 2008 we moved the resources into a searchable database. New community features mean users can search, rate, comment, email resources to each other, and contribute new resources. We offer our own research and publications, but the bulk of available resources are from the broader advocacy evaluation field.

Advocacy Capacity Assessment Tool and Advocacy Evaluation Tool

Susan Hoechstetter, Alliance for Justice, shoech@afj.org

Alliance for Justice developed advocacy capacity assessment and advocacy evaluation tools that work together (or separately) as a response to funders seeking ways to be 'accountable' for their advocacy grantmaking. The tools are pragmatic and easy to use by grantees that fill them out, as well as funders using them to learn more about their grantees' work. They are based on several concepts key to effective advocacy. Among those concepts are: 1) strong advocates are flexible and often respond to outside controls by amending their strategies and projected outcomes; 2) building an organization's advocacy capacity is key to long-term success and should be planned and evaluated similarly to advocacy campaigns; 3) incremental results, or progress, towards outcomes should be included in the evaluation; and 4) advocacy groups need to consider all four 'Avenues of Change' -- legislative, administrative, judicial, and election related - when developing partners and strategies for their campaigns.

Advocacy Progress Planner

David Devlin-Foltz, Aspen Institute, ddf@aspeninstitute.org

The Advocacy Progress Planner (APP), developed by Continuous Progress Strategic Services, offers a menu of options that advocates or their funders can click through, highlighting their type of policy goal, target audiences, assets, tactics, and benchmarks. Clear, readable definitions pop up as users mouse over their options on the APP's visually inviting pages. Questions prompt users to clarify their strategy and develop meaningful, measurable benchmarks. A notes box on each page permits users to offer specific responses. The finished APP offers a simple logic model that can be printed out or shared electronically with colleagues. Coalition partners, funders, and third-party evaluators can have password-protected access to APPs, permitting lively exchange and consensus-building at the outset of an advocacy effort. Easy online access also facilitates periodic check-ins, evaluation and learning as the advocacy progresses. David will highlight examples drawn from dozens of campaigns already using the tool.

Session Title: Picture This: Using Images to Gather and Communicate Evaluation Data

Skill-Building Workshop 107 to be held in Mineral Hall Section D on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Qualitative Methods TIG

Presenter(s):

Emily Hoole, Center for Creative Leadership, hoolee@ccl.org

Kelly Hannum, Center for Creative Leadership, hannumk@ccl.org

Abstract: In this skill-building workshop, facilitators share their experiences and lessons learned using images to gather and communicate data. Participants in this hands-on session will have the opportunity to experience gathering and presenting data using images. Images provide information that can compliment and extend survey and interview data in a manner that more fully engages stakeholders in the process and offers an additional means to gather data not easily collected via other data collection efforts. Use of images can cross language and culture and even the playing field between evaluator and the evaluee.

Session Title: Evaluating the Impact of Spending: A Foundation Case Study in Performance Assessment

Panel Session 108 to be held in Mineral Hall Section E on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Chair(s):

Ann St Claire, ClearWay Minnesota, astclaire@clearwaymn.org

Abstract: ClearWay Minnesota, like many social minded granting organizations has broadened its definition of success from simply evaluating the outcomes of specific projects to assessing the extent to which their organizational vision and mission is being achieved through programmatic activities. ClearWay Minnesota was created in 1998 with a portion of Minnesota's tobacco settlement. With a mission to improve the health of all Minnesotan's by eliminating the harm caused by tobacco and a limited 25 year life span in which to do so, the need to aggressively assess progress in meeting its mission in measurable ways is critical. This panel will present a case study of organizational evaluation at the project, strategic plan, and lifetime goal levels. The overall theory of change that supports the multi-layered organizational evaluation plan will also be discussed. Panelists will describe how these elements evolved and come together into a comprehensive, yet manageable plan for ClearWay Minnesota.

Part I: The Evaluation Structure and Its Strategic Implications

Sara Peterson, LarsonAllen, speterson@larsonallen.com

Sara A. Peterson, JD, of LarsonAllen currently leads the external evaluation team for ClearWay Minnesota. Her evaluation experience includes work with nonprofits, foundations, and state funders. Like her work with ClearWay Minnesota many of the projects required tiered assessment tracking the progress of individual grantees relative to their grant goals while at the same time tracking the collective work of participants toward larger program goals and the strategic goals of the funding organization. In this session, she will introduce the ClearWay Minnesota evaluation structure and its logic model. With this as the starting point for discussion she will discuss strategic-level evaluation for the organization and the long-term outcomes it has developed. Throughout, she will relate this evaluation structure with her experience in other settings.

Part II: Assessing the Grants and Granting Success

Ann St Claire, ClearWay Minnesota, astclaire@clearwaymn.org

- Ann W. St. Claire, MPH, Research Program Manager of ClearWay Minnesota currently oversees the organizational evaluation. Her evaluation experience includes work with individual projects and at the programmatic level. In this session, she will describe the purpose, process, and usefulness of the Report Card evaluation for individual projects. The Report Card system was established to measure the performance of each funded project to achieving the greater organizational goals. This system is used to assess grant making successes and challenges on an annual basis, contributes to measuring strategic plan outcomes, and also informs future grant making priorities.

Discussion: Adapting to Other Contexts

Barbara Schillo, ClearWay Minnesota, bschillo@clearwaymn.org

- Barbara A. Schillo, PhD, Director of Research and Evaluation at ClearWay Minnesota will end the presentation with a brief discussion of how the model and tools discussed in this presentation can be adapted and used in similar organizations. She will facilitate a discussion with the audience to help answer questions and share our collective experiences in similar organizational evaluation efforts.

Session Title: Methods for Assessing Program Impacts on Student Performance

Multipaper Session 109 to be held in Mineral Hall Section F on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Liliana Rodriguez Campos, University of South Florida, lrodriguez@coedu.usf.edu

The Impacts of a Teacher Training Program on Elementary Students’ Performance in Reading and Mathematics

Presenter(s):

Shahpar Modarresi, Montgomery County Public Schools, shahpar_modarresi@mcpsmd.org

Faith Connolly, Naviance, faith.connolly@naviance.com

Abstract: This paper examines the effectiveness of a teacher training program in improving reading and mathematics performance of elementary students. Due to the lack of random assignment of students to teachers, this evaluation used a quasi-experimental design. Both statistical significance tests and effect sizes were used to analyze the evaluation data. The analysis of covariance (ANCOVA) was utilized to test significant differences between the two groups of students (students of trained teachers and students of teachers who had not had the training). Propensity scores (based on students’ pretests, teachers’ qualification status and several other background measures) were computed using logistic regression models. To balance the two groups of students, the propensity scores were divided into five categories and used as covariates in statistical models. On average, no statistical or practical significant differences were found for performances on the reading or mathematics measures between the two groups of elementary students.

Potential Use of the Sheltered Instructional Observation Protocol (SIOP) Model for Program Evaluation

Presenter(s):

Victoria Stewart, Northwest Regional Educational Laboratory, stewartv@nwrel.org

Elizabeth Autio, Northwest Regional Educational Laboratory, autioe@nwrel.org

Abstract: Northwest Regional Educational Laboratory (NWREL) will investigate interrater reliability of the Sheltered Instructional Observation Protocol (SIOP) Model as part of the Washington state legislature’s English Language Learner Demonstration Project. Specifically NWREL field researchers will visit 37 schools across 11 districts in an effort to capture a snapshot of instructional practices in the elementary, middle, and high schools for students who are not proficient in English. Two researchers will observe a class and rate the teacher’s skills with presenting curricula in the core subjects of English, Mathematics, Science, and Social Studies using the well established SIOP rubric. The rubric is comprised of 8 components with 30 items. Each item will be scored using a five-point Likert scale ranging from 0 to 4 and later used to gauge where teachers are with administering appropriate lessons for ELL. Each of the rater's scores will be re-assessed to take stock on the similar and divergent ratings.

Session Title: Collaborating With Clients and Competitors in Education Evaluation

Panel Session 110 to be held in Mineral Hall Section G on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Chair(s):

Jon Price, Intel Education, jon.k.price@intel.com

Discussant(s):

Daniel Light, Education Development Center Inc, dlight@edc.org

Abstract: In 2000, Intel began its world-wide education programs and companion evaluation work. Since that time, both Education Development Center and SRI International as well as evaluation organizations throughout the world have been engaged as evaluators. Why would Intel engage multiple evaluators? What are the challenges, advantages and opportunities of a multi-evaluator approach? The session will include a multi-perspective discussion about how working in a collaboration of competitive organizations and a corporate client has informed the ongoing program implementation and evaluation design. The session will focus on what the presenters have learned through collaboration between and triangulation across organizations, synergistically building larger program learnings. Contributors will explore ways in which evaluators and client approach and implement evaluation within this context of cross-organization collaboration, including lessons learned and guidelines for making such collaborations productive.

Development, Growth and Sustainability: Building Collaboration as an Evaluation Practice

Roshni Menon, Education Development Center Inc, rmenon@edc.org

Scott Strother, Education Development Center Inc, sstrother@edc.org

When the challenge of creating an evaluation strategy for a new program is overtaken by the challenge of sustaining the program on a global scale, effectively managing ongoing evaluation efforts may depend on including collaboration efforts in revised strategies. By referencing the early design components of the Intel Teach teacher professional development program, this portion of the panel discussion will illustrate the challenges associated with maintaining a single agency partnership and the transition that resulted in collaborative efforts with over two dozen agencies worldwide. Transforming education systems and supporting national competitiveness are difficult, long term endeavors. On-going, embedded evaluation can help create policies that support real change.

Degrees of Collaboration with Competitive Evaluation Partners: Working Together and Separately

Ann House, SRI International, ann.house@sri.com

Ruchi Banot, SRI International, ruchi.bhanot@sri.com

A basic tension of collaborating across evaluation organizations is the question of how different evaluators can stay coordinated and consistent to provide findings that collate together, yet separate and distinct enough to explore different facets of a single program. Using Intel's Teach Essentials Online program as an example, this presentation will focus on the different levels of partnership used by SRI and EDC to collaborate (but not replicate) both larger conceptual matters (including research design and forming larger statements about the program) as well as the detailed daily tasks (including instrument design and identifying informants). This discussion will provide a picture of varying degrees of cross-organizational coordination and collaboration at different points in this evaluation work, and reflect on the strategies used to determine coordination levels.

Using Multiple Evaluators as an Evaluation Policy

Jon Price, Intel Education, jon.k.price@intel.com

This portion of the panel session will discuss the strategies associated with an evaluation design utilizing multiple organizations on a global scale, with emphasis on the coordination between Intel Corporation as the Grantor and SRI International/Education Development Center - Center for Children and Technology as collaborating (or primary) grant recipients. Such a strategy begins with the development of clear program goals, indicators and models that can be used to measure impact and ensure consistency and reduce variability in the evaluation designs. A discussion of the challenges encountered and considerations necessary to manage such a broad scale evaluation effort will follow, with reference to available rubrics, instruments, and observation/interview protocols.

Session Title: Measuring Student Achievement: Issues and Implications for Educational Evaluation

Multipaper Session 111 to be held in the Granite Room Section A on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Susan Connors, University of Colorado Denver, susan.connors@cudenver.edu

Critical Dimensions of Authenticity in Assessment

Presenter(s):

Bruce B Frey, University of Kansas, bfrey@ku.edu

Vicki Schmitt, University of Alabama, vschmitt@bamaed.ua.edu

Abby Bowen, University of Kansas, aao1984@ku.edu

Abstract: A well accepted position among educational evaluators is that the best classroom assessments are authentic (e.g. Archbald & Newman, 1988; Bergen, 1993; Gronlund, 2003; Meyer, 1992; Newman, Brandt & Wiggins, 1998; Wiggins, 1989a, 1989b). The term best typically means valid and authentic is usually defined as having something to do with the real world. While most authors speak of authentic in the context of application outside the classroom, some do not and emphasize other aspects of assessments which determine their authenticity. We have completed an initial analysis of the most-cited scholars and advocates and offer a summary of those elements of authenticity which appear to have consensus as being critical. Somewhat surprisingly, those elements include components unrelated to the realism of the assessment.

Development of an Evaluation Rubric for Engineering and Science Skills Assessment Through Student-Maintained Portfolios

Presenter(s):

Rucha Londhe, Goodman Research Group Inc, londhe@grginc.com

Abstract: The proposed paper is based on a pilot study conducted to evaluate an NSF funded Innovative Technology Experiences for Students and Teachers (ITEST) project aimed at using a collaborative on-line environment to facilitate hands-on science and engineering activities for middle and high school students. Along with other goals, the project sought to foster engineering, programming, and work-force skills in student participants. To evaluate this project outcome, students were asked to maintain a portfolio throughout the two weeks of summer camp they attended. These portfolios served as the unit of analysis for an embedded assessment. A rubric was created to evaluate these portfolios. The proposed paper outlines the process of rubric development, explains the scoring based on the rubric, and provides examples from the actual portfolios. In addition, the paper assesses the use of portfolios in the context of broader methodology issues in embedded assessment.

The Impact of Benchmark Assessment on Student Achievement In Middle School Math: Two-Year Post-Implementation

Presenter(s):

Susan Henderson, WestEd, shender@wested.org

Anthony Petrosino, WestEd, apetros@wested.org

Sarah Guckenburg, WestEd, sgucken@wested.ord

Steve Hamilton, WestEd, shamilt@wested.org

Abstract: This study examines whether districts using quarterly benchmark exams in middle school mathematics show greater gains in student achievement than those not employing this practice. The study will examine differences in student achievement as measured by the Massachusetts State Comprehensive Assessment (MCAS) in schools using quarterly benchmark assessments aligned with Massachusetts Curriculum Frameworks Standards for mathematics in grades 8, two years post implementation. This study offers practical insight to evaluators in conducting scientifically based evaluations of educational programs through a quasi-experimental design that does not require randomization to treatment group prior to the implementation of an educational innovation. The results of this study have the potential to provide a solid research base to inform district and school level practices in the use of benchmark assessment to increase student achievement on state education standards.

Examination of Potential Growth Models Using Massachusetts Comprehensive Assessment System Mathematics Scores

Presenter(s):

Susan Henderson, WestEd, shender@wested.org

Natalie Lacireno-Paquet, WestEd, npaquet@wested.org

Craig Hoyle, Education Development Center Inc, choyle@edc.org

Abstract: This study conducted by the federally funded Regional Educational Laboratory of the Northeast and Islands evaluated three potential growth model approaches of student achievement using the MCAS Mathematics test. With the US Department of Education approving states to use growth models as part of school and district accountability under No Child Left Behind, this study used real data to gauge how Adequate Yearly Progress determinations would be different. For Massachusetts, there is interest in the possibility of using growth models as a policy tool to concentrate district attention on groups or individual students over time. Three approaches to growth models were applied to a cohort of students from 4th to 8th grade using MCAS math data. These approaches include a z-score approach, a multi-level modeling approach, and a vertically aligned achievement level approach (transition matrix approach). Results suggest that growth models will not produce a large boost in the number of students counting toward AYP over time but may be a useful tool in assisting districts with focusing limited resources on accelerating student achievement.

Session Title: Out of the Crossfire: Challenges in Evaluating a Hospital-based Violence Intervention Initiative

Think Tank Session 112 to be held in the Granite Room Section B on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Health Evaluation TIG

Chair(s):

Robert LaChausse, California State University at San Bernardino, rlachaus@csusb.edu

Discussant(s):

Robert LaChausse, California State University at San Bernardino, rlachaus@csusb.edu

Abstract: This panel session brings together new evaluators, mid-career evaluators, and recognized evaluation experts to discuss what works in the evaluation of health programs. On panel are 4 speakers with a wide range of evaluation experience sharing their unique perspectives on innovative approaches regarding health evaluation methods, policy, and practices. Questions posed to the panelists include: 1. How should we go about evaluating health and prevention programs? 2. How can evaluators increase the quality of program evaluations? 3. How can evaluators encourage the use of data-driven decision making? 4. How can evaluators include the ethnic and cultural aspects of stakeholders when conducting evaluations? 5. What national or state policies should be considered for the evaluation of community health programs? The panel session will end with a 15-minute question and discussion session. Participants will have an opportunity to ask questions and share their perspectives regarding the evaluation of health programs.

Perspectives from a Not-So-New Evaluator

Jenica Huddleston, University of California at Berkeley, jenhud@berkeley.edu

Jenica Huddleston, MS is currently a doctoral student at the UC Berkeley School of Public Health, where she is focusing on implementation and evaluation of a multi-level intervention aimed at addressing childhood obesity. She is also the owner of Zetetic Associates, an evaluation consulting company working primarily on evaluating topics related to children and community health. Ms. Huddleston believes that a one-size-fits-all perspective does not work in evaluation, health-related or otherwise. Considerations of purpose, goals, stakeholders and utility should be assessed when determining the most appropriate design and methods for an evaluation. Unfortunately this is not always the case when we look at how evaluation often works in the real world.

Thoughts on Health Evaluation from Two Decades of Service

Elizabeth Harris, Evaluation, Management and Training Associates Inc, eharris@emt.org

Dr. Harris is EMT's Vice-President with two decades of experience in evaluation and survey research with federal, state and local agencies. The mission of E.M.T. Associates is to promote and facilitate the use of relevant information to improve social policy and enhance the resolution of social problems. Dr. Harris' areas of expertise include evaluation of public health, mental health and alcohol and other drug prevention, and integrated behavioral health services. Dr. Harris is currently the Project Manager for the Centers for Disease Control and Prevention's consumer response services center evaluation, CDC INFO. She is also serving as the Principal Investigator for Los Angeles County's First 5 LA Connect evaluation of a warm line for families with young children and their service providers. She recently served as Co-Principal Investigator for the National Minority Substance Abuse and HIV Prevention National Evaluation funded by the Center for Substance Abuse Prevention.

Challenges in Evaluating Health Promotion Programs

Eusebio Alvaro, Claremont Graduate University, eusebio.alvaro@cgu.edu

Dr. Alvaro is an Associate Professor in the School of Behavioral and Organizational Sciences at Claremont Graduate University. His evaluation activities involve studying the health promotion and disease prevention programs with a particular focus on the development and testing of media messages targeting health behavior change among Hispanics. He is currently funded by the Health Resources and Services Administration, the National Institute on Drug Abuse, and other organizations. He has also conducted evaluations of tobacco control programs, organ donation programs, and adolescent drug use programs.

Perspectives from the Field of Public Health

Ann Zukoski, Oregon State University, ann.zukoski@oregonstate.edu

Dr. Zukoski is currently an Assistant Professor -Senior Researcher in the Department of Public Health at Oregon State University and a private consultant. Dr. Zukoski is currently a co-investigator on a Centers for Disease Control (CDC) funded study to explore factors that affect the ability of rural Latino women and men to access reproductive health services and protect themselves from HIV/STIs and unintended pregnancy. She believes that the public health field is charged with protecting the health of populations over their lifespan. With this aim, interventions are designed to influence the actions of individuals, families, community groups, organizations and institutions. Within this broad framework, evaluators are asked to determine the worth of public health programs. She feels that health evaluations are being required to emphasize health outcomes but often fail to consider other important measures such as enhancing community capacities, changing institutional environments, and altering health policies.

Session Title: Building a Bridge Between Evaluation and Public Policy Worlds

Panel Session 117 to be held in Room 104 in the Convention Center on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Government Evaluation TIG and the Advocacy and Policy Change TIG

Chair(s):

Rakesh Mohan, Idaho Legislature, rmohan@ope.idaho.gov

Discussant(s):

Melvin Mark, Pennsylvania State University, m5m@psu.edu

Abstract: This year's conference theme forces us to think beyond our familiar evaluation world. We find there is a powerful world of public policy that exerts tremendous influence on us. By building a bridge between the two worlds, we can position ourselves to better manage the influences of those public policies that affect evaluation practice; and at the same time, through our evaluation work, we can influence public policies to bring about social betterment. This session brings together four people who have spent decades working either as experts in the field of evaluation or key players in the public policy world: the executive Director of the National Conference of State Legislatures, a former Speaker of the Idaho Legislature, a former Deputy Inspector General for Evaluation and Inspections at the US Department of Health and Human Services, and a former President of AEA.

Walking Through the Halls of State Capitols: An Inside View of the Policy World

William T Pound, National Conference of State Legislatures, william.pound@ncsl.org

William Pound brings to the panel a unique perspective of the public policymaking world. He has been the Executive Director of the National Conference of State Legislature (NCSL) since 1987. NCSL is a leading bipartisan organization that serves legislators and staffs of the nation's all 50 states, its commonwealths, and territories. NCSL provides research, technical assistance and opportunities for policymakers to exchange ideas on the most pressing state issues, and is an effective and respected advocate for the interests of state governments before Congress and federal agencies. In addition to its headquarters in Denver, NCSL maintains an office in the nation's capital that represents state legislatures on state-federal issues.

Educating Policymakers About Evaluation

Bruce Newcomb, Boise State University, brucenewcomb@boisestate.edu

Bruce Newcomb brings to the panel a unique perspective of the state policymaking world. Mr. Newcomb, the recipient of the National Conference of State Legislatures' Excellence in Leadership Award, recently retired as the Speaker of the Idaho House of Representatives. He served 10 terms in the Idaho Legislature, including 4 terms as the speaker. He is a champion of evaluation use in public policymaking and is an ardent supporter of the independent legislative evaluation function. He was the principal author of the bill that resulted in establishing the Idaho Office of performance Evaluations in 1994.

Identifying the Policymakers and Making the First Contact

George Grob, Center for Public Program Evaluation, georgefgrob@cs.com

George Grob, recipient of AEA's Alva and Gunnar Myrdal Government Award, is currently serving as a consultant to the AEA Evaluation Policy Task Force. He brings more than 30 years of federal evaluation and management experience, including working with a broad range of policymakers such as US Congressmen, agency heads, inspector generals, and executive and legislative staff.

Session Title: Evaluating Private Sector Development in Developing Countries: Factoring in Risk and Multilateral Development Bank 'Additionality'

Multipaper Session 118 to be held in Room 108 in the Convention Center on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Linda Morra-Imas, World Bank Group, lmorra@ifc.org

Discussant(s):

Theodore Moran, Georgetown University, morant@georgetown.edu

Marvin Taylor-Dormond, World Bank Group, mtaylordormond@ifc.org

Abstract: As part of the World Bank Group, the mission of the International Finance Corporation (IFC) is to promote sustainable private sector development in developing countries, and thereby contribute to growth and poverty reduction. It aims to achieve this mission through a range of investment and advisory services activities. In carrying out this mission, IFC faces two fundamental challenges: i) dealing with commercial risk, particularly at the development "frontier" where the private sector is in only a nascent state (which is the case in many low income countries and especially in Africa); ii) ensuring that its activities complement, rather than replace, those of others. This session will examine how, as evaluators, we can i) factor commercial risk into evaluations of private sector development; and ii) estimate the "additionality" - or unique value added - of a Multilateral Development Bank (MDB) such as IFC.

Factoring in Risk in Evaluating Private Sector Development in Developing Countries

Hiroyuki Hatashima, World Bank Group, hhatashima@ifc.org

Dan Crabtree, World Bank Group, dcrabtree@ifc.org

Stoyen Tenev, World Bank Group, stenev@ifc.org

The private sector operations of Multilateral Development Banks such as the International Finance Corporation face intrinsic business risk. These risks are particularly apparent at the development "frontier", where the private sector is in only a nascent state (for example in Sub-Saharan Africa, Middle East and North Africa, and South Asia). There is considerable literature on how commercial risk affects financial results, and accordant metrics, but no formal literature on how development outcomes might be affected by different risk conditions nor how best to assess development performance under different risk conditions. This paper addresses this gap. The paper identifies eight risk factors associated with a project's development and financial success. Overall, the higher the level of risk intensity, other things being equal, the lower the chance of high development outcomes. Of the individual risk factors, sponsor risk, sector risk and country business climate risk appear to have the most influence on development results. Risk profiling of this kind can offer insights into the long-term development and financial performance of projects, beyond conventional credit analysis that focuses on short term business performance, particularly when considered alongside the quality of efforts to mitigate the risks identified.

Estimating Multilateral Development Bank "Additionality" in Facilitating Private Sector Development

Dan Crabtree, World Bank Group, dcrabtree@ifc.org

Hiroyuki Hatashima, World Bank Group, hhatashima@ifc.org

Stoyen Tenev, World Bank Group, stenev@ifc.org

Development institutions are expected to do more than just provide finance and advice to their clients, be they governments or companies. Rather, they are expected to help address the serious market- and non-market institutional failures that can exist in the operation of markets and institutions in developing countries - and which collectively constrain economic growth, development and poverty reduction. 'Additionality' arises when development institutions provide inputs and services over and above those available from existing market- and non-market sources and which also exhibit unique added value. This paper will offer the first detailed exploration of the concept of MDB additionality, including a comparative assessment of how different MDBs think about additionality. Secondly, it will show how this concept can be operationalized in a way that distinguishes between different types and levels of additionality, and builds on existing international good practice standards for the evaluation of MDB private sector operations. Thirdly, it will examine the theoretical and empirical connections between additionality, development impact, and profitability.

Session Title: Towards A Better PART: Re-analysis of Random Assignment National Evaluation of Upward Bound- Bayesian Techniques for Academic Competitiveness and SMART Grant Program; and Regression Discontinuity for Small Grants Evaluation

Multipaper Session 119 to be held in Room 110 in the Convention Center on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Government Evaluation TIG

Chair(s):

David Goodwin, United States Department of Education, david.goodwin@ed.gov

Discussant(s):

Laura Perna, University of Pennsylvania, lperna@gse.upenn.edu

Abstract: The Government Performance Results Act (GPRA) and the Office of Management and Budget (OMB) Program Accountability Rating Tool (PART) reflect increasing demands that federal programs to provide rigorous evidence of effectiveness. Using critical analysis and multiple methods, we explore design, implementation, and measurement issues that arise in three different types of studies with different methods. The session includes an exploration of measurement error and a re-analyses of the controversial random assignment National Evaluation of Upward Bound that leads to a different conclusion than that originally reached in the PART process; an exploration of what Bayesian methods can contribute to the quasi-experimental observational study of the effectiveness of the Academic Competitiveness Grants (ACG) and National SMART Grants (NSG) program; and consideration of the potential of using regression-discontinuity designs for evaluating small national programs. We invite discussion concerning how to use evaluations to increase program effectiveness, Federal accountability, and desired social change.

Exploring Measurement Error in the Random Assignment 1992-93- 2003-04 National Evaluation of Upward Bound--Do Alternative Analyses Change the Conclusions?

Margaret Cahalan, United States Department of Education, margaret.cahalan@ed.gov

We take a critical look at measurement error relative to findings from the National Evaluation of Upward Bound (UB) and explore lessons learned from a policy and methodological perspective. The nationally representative random assignment study followed a multi-grade cohort from 1992-93 to 2003-04. Following reports of lack of overall positive effects, OMB gave UB an ineffective PART rating and budget recommendations in FY05 and FY06 called for zero funding. Major issues include: 1) unequal weighting; 2) treatment-control group equivalency; 3) survey non-response bias; 4) lack of standardization for expected high school graduation year (EHSGY); and 5) service substitution and dropout issues. Our major finding is that when administrative records are used to supplement survey data and outcomes standardized by EHSGY, contrary to previously published reports, the UB program demonstrated statistically significant positive effects on the major goals of the program; postsecondary entrance, application for financial aid; and attainment of postsecondary credentials.

Using the Regression-Discontinuity Design For Measuring the Impact of Federal Discretionary Grant Programs for OMB's PART

Jay Noell, United States Department of Education, jay.noell@ed.gov

Federal programs are required by OMB to complete a PART (Program Assessment Rating Tool) for use in performance budgeting. Programs are given a numerical score (and an associated judgment) based on their PART, and 50 percent of the score depends upon a program's performance results. Many smaller discretionary grant programs are unable to produce credible evidence of effectiveness, and can be judged ineffective, which can result in OMB proposing reducing or eliminating their budgets. This paper describes a way that many of those programs could be evaluated using a regression-discontinuity research design (RDD). The advantage of the RDD is that it provides a basis for making unbiased causal inferences about program effectiveness when evaluations using randomized control trials (RCT) are not possible. A number of discretionary grant programs funded through the U.S. Department of Education and other federal agencies could be evaluated this way. But with what results?

A Comparison of Bayesian and Standard Methods in Evaluating the American Competitiveness (ACG) and National SMART Grant (NSG) Programs

Sharon Stout, United States Department of Education, sharon.stout@ed.gov

What are the challenges of using national data sets and financial aid award data to measure trends and interruptions in trends over time that may be attributed to new legislation? How can Bayesian statistical techniques contribute to these analyses? The ACG and NSG Programs increase Pell Grant awards to provide additional incentives to eligible students to encourage them to take a more rigorous program of study in high school; or as 3rd and 4th year students, to major in mathematics, science, or critical foreign languages. Using national surveys and transcript study data as well as federal student aid Pell award databases, we apply Bayesian statistical techniques and consider how to model the data appropriately to generate efficient and robust inferences making full use of quantitative and structural prior information. These methods and their results are compared to standard methods.

Session Title: Evaluating Public Health Research Centers: Assessing the Value-Added

Think Tank Session 120 to be held in Room 112 in the Convention Center on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Research, Technology, and Development Evaluation TIG

Presenter(s):

Thomas Bartenfeld, Centers for Disease Control and Prevention, tab3@cdc.gov

Howard Kress, Centers for Disease Control and Prevention, hak5@cdc.gov

Discussant(s):

Sue Lin Yee, Centers for Disease Control and Prevention, sby9@cdc.gov

Demia Wright, Centers for Disease Control and Prevention, amy7@cdc.gov

Michele Hoover, Centers for Disease Control and Prevention, mlh5@cdc.gov

Abstract: Assessing the value of research centers requires a focus on the overall research program, not merely the projects that they undertake. The technology, energy, and medical fields have done much work on how to evaluate research programs. In evaluating their own funded research centers, the Centers for Disease Control and Prevention (CDC) can apply many of these methods. However, given the scope and breadth of responsibilities, the disparate activities within and across centers, the modest funding for CDC research programs, and the need for value judgments, CDC evaluators have had to make strategic decisions to get at the 'so what' of its research center programs. Staff from CDC's Prevention Research Centers (PRC), Injury Control Research Centers (ICRC), and Academic Centers for Excellence (ACE) on Youth Violence Prevention will lead a brainstorming and sharing session on evaluating public health research center programs in light of limited funding and evaluation resources.

Session Title: Design: Methods and Issues

Multipaper Session 121 to be held in Room 103 in the Convention Center on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Chair(s):

Dale Berger, Claremont Graduate University, dale.berger@cgu.edu

Maximizing Follow-up Completion Rates in the Face of Real-world Constraints: Successes from a Tobacco Cessation Program Evaluation Project

Presenter(s):

Kay Calendine, University of Arizona, kcalendi@u.arizona.edu

Sue Voelker, University of Arizona, smlarsen@u.arizona.edu

John Daws, University of Arizona, johndaws@email.arizona.edu

Abstract: Programs whose evaluation components require the collection of participant follow-up contact data frequently face challenges with resources, limited time frames, and other real-world constraints, hindering the ability to achieve contact completion rates high enough to produce useful program evaluations. To overcome these issues, the Arizona tobacco cessation program evaluation project identified several barriers on which to focus and developed strategies targeting these challenges. One identified barrier was a callback pool that was too large to be managed effectively with the available resources. The proposed solutions were to decrease the size of the participant follow-up pool by sampling, while focusing more effort toward locating difficult-to-reach participants This paper presentation discusses the two main strategies implemented to address this barrier, the resulting benefits of these strategies, the project’s subsequent success in increasing the completion rate from 40% to over 80%, and suggestions for incorporating these strategies into other follow-up data collection projects.

Developing Effective, Non-Depressing Pre-Tests

Presenter(s):

Linda Heath, Loyola University Chicago, lheath@luc.edu

Jonya Leverett, Loyola University Chicago, jlevere@luc.edu

David Slavsky, Loyola University Chicago, dslavsk@luc.edu

Abstract: Pre-test data are crucial for assessing program effectiveness, but the very act of administering a pre-test can adversely affect the integrity of the research design and the program itself. Most designs use the same or alternate forms of measures for the pre- and post-tests. The measure needs to be at a high enough level to capture program gains, but administering such a high-level measure at the pre-test can lead to demoralization of the treatment group and disappearance of the comparison group. Valuable program time and resources then need to be spent restoring a sense of efficacy to the program participants and seeking post-test data from comparison group members. Grant budgets and schedules often preclude spending time developing less-threatening pre-tests on program participants. In this research, the effectiveness of developing Computerized Adaptive Tests (CAT) on Introductory Psychology students for ultimate use with public school teachers is explored.

Session Title: Moving From Ideas to Action: Measuring Foundation Impact

Think Tank Session 122 to be held in Room 105 in the Convention Center on Wednesday, Nov 5, 4:30 PM to 6:00 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Presenter(s):

Nancy Csuti, The Colorado Trust, nancy@coloradotrust.org

Discussant(s):

Nancy Csuti, The Colorado Trust, nancy@coloradotrust.org

Doug Easterling, Wake Forest University School of Medicine, dveaster@wfubmc.edu

Sigurd Nilsen, Council on Foundations, sigurd.nilsen@cof.org

Abstract: Over the past two decades much as been written calling for more accountability on the part of foundations to demonstrate impact of their work. Many foundations have experimented with assessing impact with varying degrees of success. These attempts have been sporadic, are rarely sustained, often frustrating to all involved parties. However, in the current funding climate as foundations are more closely examined as a possible solution to federal budget cuts, the need to not only examine foundation impact but share the results has increased. This session will focus on what - and who - is necessary to make foundation-focused evaluation relevant - foundation staff, boards and CEOs, and independent evaluators. Questions are: 1. What factors influence a foundation to embark on an impact evaluation? 2. What can independent evaluators do to encourage foundations to evaluate impact? 3. What can be done to improve the odds that a foundation will use the results? What is the role of the foundation versus the evaluators in making this happen?

Session Title: Contrary to Expectations: When Evaluators have Bad News to Share

Think Tank Session 123 to be held in Room 107 in the Convention Center on Wednesday, Nov 5, 4:30 PM to 6:00 PM