Return to search form  

Session Title: You + Graphic Design = Fame, Glory

Demonstration Session 851 to be held in California A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Data Visualization and Reporting TIG

Presenter(s):

Stephanie Evergreen, Evergreen Evaluation, stephanie@evergreenevaluation.com

Abstract: "Death by Powerpoint" won't literally kill your audience. But it will cause them to check their phone messages, flip ahead in the handout, and fall asleep. In this demonstration, attendees will learn the science behind good slideshows and will leave with direct, pointed changes that can be administered to their own evaluation presentations. The demonstration will focus on evidence-based principles of slideshow design that support legibility, comprehension, and retention of our evaluation work in the minds of our clients. Grounded in visual processing theory, the principles will enhance attendees' ability to communication more effectively with peers, colleagues, and clients through a focus on the proper use of color, placement, and type in slideshow presentations.

Session Title: A Conversation With Michael Patton About how Values Undergird the Assessment of Program Effects

Panel Session 852 to be held in California B on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Qualitative Methods TIG

Chair(s):

Charles Reichardt, University of Denver, creichar@du.edu

Abstract: The paradigmatic views and methodological values of a leading proponent of qualitative methods will be probed and challenged in a collegial conversation about assessing program effects. Although the questions that will be asked are meant to confront and prod, the purpose of the conversation is to understand rather than debate - to talk with, rather than past, each other. If qualitative and quantitative researchers are to resolve their longstanding animosities, they must come to understand each other's perspectives and values. The purpose of this panel is to provide a forum for a qualitative researcher to answer the pointed but respectful questions that quantitative researchers need to have answered if they are to understand and appreciate qualitative methods. In return, the answers will challenge and probe quantitative assumptions and perspectives. Ample time will be left for comments and questions from the audience.

Questions for Michael Patton About Assessing Program Effects: From a Quantitative Perspective Charles Reichardt, University of Denver, creichar@du.edu I will pose a series of questions about how qualitative researchers assess program effects. My purpose is to understand the logic, practice, and values of qualitative research, as well as to probe for soft spots. My hope is that quantitative researchers in the audience will come to appreciate and trust qualitative methods by seeing how qualitative methods can withstand rigorous scrutiny from a quantitative perspective. I also hope that qualitative researchers in the audience will benefit in the following way. Over the years, qualitative researchers have challenged the use of quantitative methods. These challenges have helped quantitative researchers better understand their own methods and values, and the challenges have improved the practice of quantitative research. In a parallel fashion, a quantitative challenge to the use of qualitative methods might help enlighten qualitative researchers' understanding of their own methods and values, and thereby improve the practice of qualitative research.

Responses From Michael Patton About Assessing Program Effects: From a Qualitative and Complexity Perspective Michael Patton, Utilization-Focused Evaluation, mqpatton@prodigy.net Chip Reichardt posed some questions to me about my perspective on assessing program effects. While I believe the questions were intended to be open, inviting and neutral, the methodological values revealed by and embedded in the questions offer the opportunity for an in-depth exploration of how methodological values, assumptions, and paradigms influence the very evaluation questions we ask with significant consequences for how we engage in an evaluation inquiry and for evaluative thinking generally. This issue has been the subject of debates, expert lectures, and advocacy presentations. Instead of those formats, we're suggesting a conversation in which colleagues' panel presentations constitute questions generated by and left unanswered in my writings, especially with regard to issues of attribution and assessing program effects. I will respond through the lenses of systems thinking, complexity concepts, qualitative inquiry with a fundamental focus on evaluation use and consequences.

Values Issues and Assumptions in Choosing Evaluation Questions and Evaluative Evidence Melvin Mark, Pennsylvania State University, m5m@psu.edu Some evaluations are designed to estimate the effects of a program on a set of pre-identified outcomes. Keeping with the theme of this session, I will explore a set of values and assumptions related to choosing such an evaluation versus addressing a set of alternative questions that might be addressed in an evaluation. I will also briefly explore values issues and assumptions that are associated with alternative methods that might be used to "estimate program effects." The presentation is intended to try to illuminate aspects of the exchange between Reichardt and Patton in this session, as well as situate that exchange in relation to selected other contemporary conversations or disputes in evaluation.

Session Title: Starting and Succeeding as an Independent Consultant

Panel Session 853 to be held in California C on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Independent Consulting TIG

Chair(s):

Jennifer Williams, Out of the Crossfire Inc, jenniferwilliams.722@gmail.com

Abstract: Independent Consultants will share their professional insights on starting and maintaining an Independent Evaluation Consulting business. Panelists will describe ways of building and maintaining client relationships and share their expertise related to initial business set-up and lessons they have learned. Discussions will include the pros and cons of having an independent consulting business, the various types of business structures, methods of contracting and fee setting, as well as the personal decisions that impact having your own business. Panelists will examine some consequences of evaluation in the context of conducting independent consulting in diverse settings. The session will include ample time for audience members to pose specific questions to the panelists.

Moving (and Shaking): From Employee to Consultant Jennifer Williams, Out of the Crossfire Inc, jenniferwilliams.722@gmail.com Dr. Jennifer E. Williams is President and Lead Consultant of J. E. Williams and Associates, an adjunct professor, licensed counselor, and Independent Consultant. She has extensive experience conducting education, social and market research and program evaluation. She will share her experience of moving from being an employee to a consultant and the impact it has had on her both personally and professionally.

Getting Started: What Questions Do I Need to Ask and Answer? Judah Viola, National-Louis University, judah.viola@nl.edu Dr. Judah J. Viola is an assistant professor of community psychology at National-Louis University in Chicago. He has been working part-time as an independent evaluation consultant for the past seven years while maintaining his ties to academia. He consults with a variety of school systems, museums, and small non-profits in the mid-west. His (2009) wrote the book, "Consulting and evaluation with community based organizations: Tools and strategies to start & build a practice". His presentation will focus on what questions new consultants need to ask and answer for themselves before they start their businesses.

Traveling and Working: International Evaluation Consulting - One Woman's Perspective Tristi Nichols, Manitou Inc, tnichols@manitouinc.com Dr. Tristi Nichols is a program evaluator and owner of a sole proprietorship consulting business. Her work focuses primarily on international issues, which provides a unique lens through which to view independent consulting. Her reflections about consulting, international travel, the types of decisions she makes, and their impacts on her professionally and personally as a wife and mother will be of interest to novice, veteran, or aspiring independent consultants.

Reflections from 30 Years of Evaluation Experience Maryann Scheirer, Scheirer Consulting, maryann@scheirerconsulting.org Dr. Mary Ann Scheirer has been an evaluator for three decades, working in a variety of settings including higher education, government agencies, large consulting firms, and now, independent consulting. Her presentation will focus on how and why she moved into independent consulting and lessons learned from this move. She will provide a contrasting perspective as her move came after many years of service in multiple organizations.

Setting Yourself Apart Amy A Germuth, EvalWorks LLC, amygermuth@evalworks.com Dr. Amy Germuth has started two successful evaluation firms and is currently president of EvalWorks, LLC. She focuses primarily on STEM education research and evaluates local, state, and national programs. She will present a Prezi that discusses critical points related to starting one's own business and things to consider to make it successful. Critically she will point to the need for marketing, consulting, and business skills as part of a successful evaluation consultancy.

Session Title: Diversity Dialogue: Strategies & Stories From the Evaluation Road

Think Tank Session 855 to be held in Pacific A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Presidential Strand

Presenter(s):

Fiona Cram, Katoa Ltd, fionac@katoa.net.nz

Discussant(s):

Kari Greene, Oregon Public Health Division, kari.greene@state.or.us

Maurice Samuels, University of Chicago, mcsamuels@uchicago.edu

Nicole Bowman, Bowman Performance Consulting, nicky@bpcwi.com

Sharon Brisolara, Evaluation Solutions, sharon@evaluationsolutions.net

Fiona Cram, Katoa Ltd, fionac@katoa.net.nz

Abstract: The diverse values of varied stakeholders in evaluation contexts constitute a vital domain for evaluation. Stakeholder values help shape the direction, substance, criteria, and intended uses of the evaluation. An inevitable challenge is to respond to the diversity and plurality of legitimate stakeholder values and interests. And too often, the values and interests of stakeholders who are least well served are quieted by other more powerful voices. In response, democratic evaluation inclusively seeks to provide spaces for all stakeholders, including those least well served, to have a 'say' in the shape of the evaluation through processes of dialogue and deliberation (House & Howe, 1999). This interactive session will highlight different approaches used to engage the challenge of generating meaningful avenues for inclusion of less powerful stakeholders' values, voice, and interests in evaluation. Attendees will receive practical tools and information through presentation, interactive breakout groups, and reflections from a discussant.

Session Title: Reliability: The Beginning of Value

Multipaper Session 857 to be held in Pacific C on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Quantitative Methods: Theory and Design TIG

Chair(s):

Dale Berger,  Claremont Graduate University, dale.berger@cgu.edu

Reducing Rater Bias in Scoring Performance Assessments Presenter(s): Robert Johnson, University of South Carolina, rjohnson@mailbox.sc.edu Min Zhu, University of South Carolina, helen970114@gmail.com Brandon Loudermilk, University of South Carolina, loudermb@mailbox.sc.edu Xiaofang Jae, University of South Carolina, jae2008@gmail.com Ashlee Lewis, University of South Carolina, lewisaa2@mailbox.sc.edu Abstract: This study examines the use of visual representations of types of scoring bias in training raters to score arts assessments. Initial evidence is mixed about whether the quality of raters' scores improved when their training incorporated visual representations of scoring bias versus all verbal descriptions of bias. Two of the three treatment groups (i.e., raters trained with visual representations of bias) displayed closer agreement with validation scores than did the controls; whereas, two of the control groups had higher interrater reliability than the treatment groups. The potential for the utility of visual representations of bias is reflected in that a majority of raters in the treatment group successfully recalled more of the types of bias that were presented visually rather than verbally.

Bootstrap Reliability Presenter(s): Cristian Gugiu, Western Michigan University, crisgugiu@yahoo.com Abstract: All measures are imperfect since no test (survey) is capable of measuring with perfect accuracy or precision. Over the past century, psychometricians have developed a multitude of reliability measures in order to provide researchers with a simple index that quantifies the reliability of a test. Undoubtedly, the most popular index continues to be coefficient (Cronbach's) alpha. However, this reliability estimator is plagued by a number of issues, including that it (a) only provides a lower bound estimate of the true reliability, (b) is only interpretable if the test is unidimensional, (c) has no lower bound (i.e., negative estimates are possible), and (d) is only appropriate for continuous data. This paper will introduce a new method for estimating the reliability of a test based upon the bootstrap method. The bootstrap reliability method is simple to understand and has the versatility to work with continuous, ordinal, and nonlinear data.

Employing Generalizibility Theory to Assess the Reliability of Peer Evaluation Presenter(s): Mihaiela Gugiu, Central Michigan University, gugiu1mr@cmich.edu Abstract: Since the early 1990s, a growing movement in education has advocated for the implementation of peer evaluation in teaching. Naturally, this movement has not gone unchallenged by those who question the reliability and validity of peer evaluation. The purpose of this paper is to demonstrate the applicability of Generalizibility theory in estimating the reliability and validity of student peer evaluation. I draw from my experience in teaching introductory courses in political behavior, where group projects were utilized as a means of enhancing student knowledge. College students were asked to evaluate the group presentations of their peers and the written papers of two other groups. Generalizability theory was used to examine the reliability of peer evaluation of student written group projects and oral presentations. Moreover, the validity of peer evaluations was examined by comparing the grades resulting from this method to those produced by the instructor and a graduate teaching assistant.

Session Title: Changing Our Tune: Reinventing Evaluation While Your Organization Transforms

Panel Session 858 to be held in Pacific D on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Non-profit and Foundations Evaluation TIG

Chair(s):

Lester Baxter, The Pew Charitable Trusts, lbaxter@pewtrusts.org

Abstract: Every organization with an internal evaluation function struggles with questions of that unit's mission, responsibilities, and relationship to the larger organization. This panel will examine what happened to an evaluation department when the private foundation within which it was well-established became a public charity. This change in legal status led to a dramatic transformation of the organization, including the shift to directly operating the majority of its policy reform projects and an expanded infrastructure. Panelists will discuss how the organization's rationale for evaluation (and its complementary roles in planning and knowledge sharing) evolved in response to internal changes. Discussion will focus on the unit's re-envisioning of its role, describing which efforts failed, which are succeeding, and which are still in flux. The panel will be of interest to evaluators working in complex or changing institutional environments, and those interested in the role of evaluation and planning in policy change efforts.

Taking Our Own Medicine: Lessons From Assessing and Reshaping our Evaluation Practice Scott Scrivner, The Pew Charitable Trusts, sscrivner@pewtrusts.org Lester Baxter, The Pew Charitable Trusts, lbaxter@pewtrusts.org As evaluators, we are used to asking and answering questions about what works, what doesn't, and why, and using findings to make sound judgments about how to allocate resources and design and run effective programs. It is perhaps less common to apply our evaluative lens to our own practice. This presentation will describe how the Planning and Evaluation department at the Pew Charitable Trusts responded to large shifts in the organization's dominant business model, staffing, and geographic focus-focusing on what we've learned from assessing our approach to policy change and advocacy evaluation. Discussion topics will include our efforts to: reconsider major aspects of our evaluation model; balance learning and accountability to meet the needs of internal audiences; respond to demands for more timely mid-course evaluations; and expand stakeholders to include new audiences, such as external funders.

The Evolution of a Planning and Evaluation Unit's Role in Planning for Policy Change Campaigns Nicole Trentacoste, The Pew Charitable Trusts, ntrentacoste@pewtrusts.org Scott Scrivner, The Pew Charitable Trusts, sscrivner@pewtrusts.org Pew's shift in 2004 from a private foundation to a public charity resulted in significant changes to the Planning and Evaluation (P&E) unit's approach to program planning (including designing evaluable projects), largely due to Pew's new focus on directly operating its programs, ability to engage in a broader range of advocacy tactics, and growth in demand for planning assistance. As a foundation, P&E's primary planning role was to oversee the development of grantmaking strategies. Since Pew's change in legal status, our planning practice has evolved to meet the changing needs of the organization. We have found that in an organization that directly manages advocacy campaigns, planning must be flexible and timely, and planning does not stop once a strategy has been designed. Discussants will describe the ways in which P&E now incorporates tools and processes to help with the effective design, implementation and management of programs.

Tensions and Ties Between Evaluation and Program Planning in a Changing Organization Glee Holton, The Pew Charitable Trusts, gholton@pewtrusts.org Nicole Trentacoste, The Pew Charitable Trusts, ntrentacoste@pewtrusts.org Lester Baxter, The Pew Charitable Trusts, lbaxter@pewtrusts.org It's not uncommon for private foundations to have "Planning & Evaluation" departments, and evaluators recognize the important linkages and overlaps between these two roles. At Pew, the tensions and ties between Planning and Evaluation were put to the test when the organization transformed from a private foundation making grants to a public charity that now largely directly operates programs and engages in a broad range of advocacy tactics. This presentation will discuss how the organization's change in status and operating model affected the linkages between planning and evaluation, and in particular how we've adapted and developed evaluative tools to use in planning for advocacy and policy change campaigns. We will also discuss our approach to knowledge sharing and using evaluation findings in planning, particularly in light of our new need to evaluate in-house campaigns rather than the work of external grantees.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Evaluation Challenges of Built Environment Policy-System-Environment (PSE) Changes

Roundtable Presentation 859 to be held in Conference Room 1 on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

Laurie Ringaert, Seattle King County Public Health, laurie.ringaert@kingcounty.gov

Jim Krieger, Seattle King County Public Health, james.krieger@kingcounty.gov

Nadine Chan, Seattle King County Public Health, Nadine.chan@kingcounty.gov

Kadie Bell, Seattle King County Public Health, Kadie.bell@kingcounty.gov

Ryan Kellogg, Seattle King County Public Health, Ryan.Kellogg@kingcounty.gov

Abstract: The Public Health - Seattle & King County was awarded two highly-competitive federal stimulus grants to address the leading causes of death in our region as part of the CDC Communities Putting Prevention to Work (CPPW). This presentation focuses on the evaluation of the seven local government grantees involving a participatory, developmental approach and a focus on creating policy, systems and environment (PSE) changes that would produce healthier built/food environments. PSE evaluation methodology and the introduction of health concepts into planning are both relatively new. As a result, creative evaluation plans and processes and specific tools were developed to capture baseline and changes over time. Team role challenges will be discussed. The evaluation takes into account what real world changes are possible in an atmosphere of economic downturns multiple stakeholder interests in policy development. The presenter will discuss the outcomes, challenges and lessons learned from this evaluation approach.

Roundtable Rotation II: Examining the Impact of a Community Partnership to Increase Capacity to Train more Nurses and Provide Better Care for Older Adults

Roundtable Presentation 859 to be held in Conference Room 1 on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

Paula Rowland, Independent consultant, paula@global-concerns.com

Afsaneh Rahimian, Independent Consultant, rahimianafsaneh@yahoo.com

Abstract: We used a mixed-method approach to evaluate the impact of a diverse community collaboration implementing multi-layered strategies to address the shortage of nurses specializing in elder care. Pacific Lutheran University's School of Nursing worked with the partners to refocus the nursing school curriculum, hire faculty, award scholarships, expand in- patient clinical placements and create innovative community based clinical opportunities to increase students' exposure to gerontology and influence their career choice. We measured the strength of the collaboration through a focus group with the community partners, phone interviews with nursing school faculty and community stakeholders, focus group with the scholarship recipients, and an online survey of all PLU nursing students enrolled in the study year. Findings suggest that the strength of this collaboration is at the very core of the project's successful achievement of its short and mid-term outcomes and that the partnership is on its way to becoming self- sustaining.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Improving Evaluation Practice With Youth: A Checklist for Developmentally Sensitive Program Evaluation

Roundtable Presentation 860 to be held in Conference Room 12 on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Non-profit and Foundations Evaluation TIG

Presenter(s):

Silvana Bialosiewicz, Claremont Graduate University, silvana@cgu.edu

Miriam Jacobson, Claremont Graduate University, jacobson.miriam@gmail.com

Tiffany Berry, Claremont Graduate University, tiffany.berry@cgu.edu

Abstract: The evaluation of programs that serve youth can be complex given the multifaceted nature of child and adolescent development. These evaluations require a developmentally sensitive approach, which includes thoughtful consideration of the characteristics this unique population brings to the evaluation. In this Round Table we will describe a developmental sensitivity checklist framed within the Centers for Disease Control's six-phase framework. Our goal is to document the pertinent considerations for typical as well as atypical youth across each stage of the evaluation process. This checklist was developed through an extensive literature review of best practices in applied research with youth and was validated by an expert panel of developmental psychologists and veteran youth-program evaluators. In this Round Table we will introduce our tool, receive feedback to refine the tool, as well as engage evaluators in a discussion about how we can continually improve the quality of program evaluations targeting youth.

Roundtable Rotation II: Assessing the Systems of Supports and Opportunities for Youth in Six Detroit Neighborhoods as a Building Block for Development

Roundtable Presentation 860 to be held in Conference Room 12 on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Non-profit and Foundations Evaluation TIG

Presenter(s):

Della M Hughes, Brandeis University, dhughes@brandeis.edu

Brian Dates, Southwest Counseling Solutions, bdates@swsol.org

Sara Plachta Elliott, The Skillman Foundation, selliott@skillman.org

Abstract: What does it take at a neighborhood level to ensure young people can be safe, healthy, well educated, and prepared for adulthood? The Skillman Foundation in Detroit, Michigan is investing $100 M over ten years in six neighborhoods to (among other neighborhood capacities) create systems of supports and opportunities (SOSO) with an array of youth development, volunteer, college and career exposure and access, and youth employment preparation and placement programs. Brandeis University assessed the SOSOs to provide data for planning and decision making. Southwest Counseling Solutions is a Skillman grantee charged with SOSO development and management in two of the six neighborhoods. Participants will address how the assessment took place and what kind of results it produced, the practical applications of having a database for planning and system development, how the Foundation and a community-based organization are managing the data going forward, and whether the data really makes a difference.

Session Title: Organizational Learning and Approaches in Calling for, Conducting, and Using Evaluations

Multipaper Session 861 to be held in Conference Room 13 on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Jim Rugh,  RealWorld Evaluation, jimrugh@mindspring.com

Strategies for Improving International Development Evaluation Terms of Reference (TORs) or Requests for Proposals (RFP) Presenter(s): Anne Cullen, Western Michigan University, anne.cullen@wmich.edu Daniela Schroeter, Western Michigan University, daniela.schroeter@wmich.edu Kelly Robertson, Western Michigan University, kelly.robertson@wmich.edu Michele Tarsilla, Western Michigan University, michele.tarsilla@wmich.edu Pedro Mateu, Western Michigan University, pedro.f.mateu@wmich.edu Abstract: A 2010 study on TORs and RFPs issued by international development organizations found that rigid evaluation terms of reference (TORs) or requests for proposals (RFP) limit evaluators' ability to determine (i) the best approach and methodology for given evaluation problem and (ii) the most feasible procedures for implementing the evaluation under given (and often unknown) timelines and budgets. On the other extreme, TORs are often vague and provide no budgetary guidelines leaving evaluators guessing as to what evaluation commissioners require. This paper presents real world strategies for international development commissioners to improve the quality of their TORs and RFPs by overcoming common mistakes such as short turnaround time for proposals, rushed start dates, short evaluation timeline, strict familiarity/experience requirements, and rigid guidelines for evaluation conduct. Session participants will be encouraged to share their suggestions for successful evaluation TORs and RFPs.

Mission Metrics: One Agency's Effort to Capture Mission Level Results Presenter(s): Barbara Willett, Mercy Corps, bwillett@mercycorps.org Gretchen Shanks, Mercy Corps, gshanks@mercycorps.org Abstract: For many years Mercy Corps has struggled with a lack of information available that spoke to agency-level performance, not just a collection of independent programs. Efforts to improve M&E helped, but there was still something missing that would elevate information to a higher level, to provide meaning as well as utility to the agency. Mission Metrics is one agency's effort to answer the question that keeps us up at night: How do we know how we are doing? This system aligns a tremendously diverse set of programs with the agency's Mission through a specially designed framework of themes and broad indicators. The framework was developed collaboratively based on the ideas and values implied by Mercy Corps' Mission, and given meaningful and measurable form by Mercy Corps' people. This paper describes the 3-year journey taken to answer a critical question, and some of the things it has learned along the way.

Reflections on the value of self-evaluation of programs in the African Development Bank Presenter(s): Foday Turay, African Development Bank, f.turay@afdb.org James Edwin, African Development Bank, j.edwin@afdb.org Mampuzhasseril Madhusoodhanan, African Development Bank, m.mampuzhasseril@afdb.org Mohamed Manai, African Development Bank, m.manai@adb.org Abstract: In the African Development Bank (AfDB), completed programs are self-evaluated by the Operational Departments for feedback on the project results and to draw lessons for management. The resulting self-evaluation reports (SERs) are reviewed for quality by the AfDB's 'independent' Evaluation Department. This paper presents evaluative reflections on the issue of SER value in the AfDB. Relying on value factors from the literature, it develops an analytical framework which includes SER audience, purpose, timing, quality, format and context. It draws on perceptions on SER value of staff of the AfDB's operational and evaluation departments, and on the results of a review of 149 SERs prepared during 2009-10. Individual interviews are held with all AfDB operational and evaluation staffs who were involved in the 2009-10 SERs. The emerging findings reflect differences in perceived values and point to the coherence with some of the evaluation principles especially credibility and usefulness.

Definitions and Dashboards: Data Quality in an International Non-Profit Education Organization Presenter(s): Michael Wallace, Room to Read, michael.wallace@roomtoread.org Rebecca Dorman, Independent Consultant, rebeccashayne@hotmail.com Wally Abrazaldo, Room to Read, wally.abrazaldo@roomtoread.org Abstract: Since 2008, Room to Read's M&E system has included the collection of information on our Global Indicators (GIs)—a combination of quantitative program accomplishments and program performance measures on all active projects that show progress towards our program objectives. This paper describes our experience in collecting, storing, analyzing, and reporting this information during the past three years: 2008: Developing a system for collecting, entering, cleaning, and analyzing indicator data; using multiple channels of communication between our headquarters and field offices. 2009: Getting definitions right; improving field-level ownership of data; streamlining communication with a single headquarters communication channel; explaining GI trends. 2010: Developing dashboards (online tools that show real-time performance and progress on key indicators) for communication of data quality issues; improving accountability for data timeliness and accuracy; comparing our internal program GIs with external data sources. The paper concludes with lessons learned and challenges going forward.

Becoming Learning Organizations: Value and Usability of Evaluations in Bilateral Donor Agencies Presenter(s): Winston Allen, United States Agency for International Development, wallen@usaid.gov Abstract: The value of evaluation as a source of learning has gained recognition among bilateral donor agencies. New evaluation policies have been adopted and structural changes made to enhance the role and function of evaluation. This trend has been fueled by political demand, and a corollary interest to rigorously demonstrate the effectiveness, and impact of development programs. Agencies that have established evaluation policies in the last decade include USAID, NORAD, AUSAID, DANIDA, and DFID. Strengthening organizational evaluation capacity is not an end in itself. The value of evaluation as a learning tool lies in its use to make strategic and evidence-based decisions that will maximize development program effectiveness. This paper presents an analysis of evaluation policies of five bilateral agencies, from the perspective of the value of evaluation as a source of learning. The results demonstrate that credibility of evaluations is an important value for evaluation use.

Session Title: Engaging Youth in School-based Youth Participatory Evaluation

Panel Session 862 to be held in Conference Room 14 on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Chair(s):

Robert Shumer, University of Minnesota, rshumer@umn.edu

Abstract: A lot has occurred since the early beginnings of youth led evaluation. Ever since Kim Sabo asked Brad Cousins why there weren't youth involve as evaluators, in school participatory evaluation models, a field has been growing. From Massachusetts to Michigan, from Minnesota to California, youth led evaluation has gone into high gear. Many programs are operating and expanding. However, most youth participatory evaluations are being developed through after-school programs and initiatives. Few courses exist in K-12 schools that teach youth how to be civically engaged citizens and conduct youth participatory evaluations. In this session we learn about a few programs that engage students in evaluation and then see and hear about two California programs that engage students through actual in-school efforts to evaluate service-learning and character education. programs.

Framing the Work of Teaching Evaluation in Schools Robert Shumer, University of Minnesota, rshumer@umn.edu In this presentation Dr . Shumer will provide a brief history of youth participatory evaluation and cite two examples of studies conducted in school settings. The goal of this presentation will be to discuss some of the guidelines and challenges of doing participatory evaluation with school age youth. Dr. Shumer has been doing youth led evaluation for many years throughout the US. He has authored a book on Youth Led Evaluation, which was used as a text for one of the courses.

Teaching Participatory Evaluation in a High School to Evaluate Service-Learning Programs Susan Ward Roncelli, Eagle Rock High School, swrerhs@aol.com In this presentation Susan will explain how she developed the evaluation course in a Los Angeles Unified School District high school and what were the successes and challenges in creating such a class. She will then explain, along with one or two of her students, how the evaluation is progressing and what major learning outcomes are occurring from the work. Ms. Ward-Roncelli has been teaching in Los Angeles for many years and is currently the GEAR UP coordinator at Eagle Rock High School. She has years of experience doing service-learning and is well versed in the challenges of trying to evaluation such programs.

Evaluating Service-Learning Programs: A High School Student Perspective Eagle Rock Students, Eagle Rock High School,  Two students from Eagle Rock High School will discuss how they are developing plans for their evaluation class and how the class activities are helping them to learn about both service-learning and participatory evaluation. These students will be selected from the introductory class at Eagle Rock High School. Since they are involved in developing this first effort class at their school, they will provide special insight into what is involved in actually trying to initiate evaluation as an actual high school course and how it helps to teach academic knowledge and skills. Youth perspectives are important to help us better understand engaging young people in participatory evaluation processes. Hearing directly from the youth involved should give us a better idea of how viable and practical youth-led evaluation is in a school-based context.

The Character of Evaluation: Measuring Respect, Responsibility, and Caring Lara Shumer, Valley Center Elementary School, mslary@yahoo.com Ms. Shumer will frame the evaluation project and explain how it fit with the school's character education program. She will describe how the evaluation work fit with her regular school activities and explain what role(s) she played in guiding the youth through the evaluation process. Lara Shumer has been teaching elementary school for the past nine years. She has included youth led evaluation as an important learning activity for several years. In one project her students actually got the school district to change to biodiesel fuels for their buses and all fuel uses.

Session Title: When Monetary Quantification Is Not Sufficient: Other Factors That Are Useful for Determining Program Success

Multipaper Session 863 to be held in Avila A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Costs, Effectiveness, Benefits, and Economics TIG

Chair(s):

Nadini Persaud,  University of the West Indies, npersaud07@yahoo.com

The Evaluation of Different Types of Cancer Treatment and Quality of Life (QOL) by Using UW QOL Scale (University of Washington QOL Scale) in Bangladesh Presenter(s): Housne Ara Begum, University of Dhaka, drhousne@gmail.com Ferdous Ara Islam, Bangladesh Institute of Health Sciences, nasrul_saiful@yahoo.com Abstract: As the cost of cancer treatment is very high so the Quality of life after having different cancer treatment should be known. Studies regarding this were not found in Bangladesh. This study aimed to evaluate the Quality of Life changes by different types of cancer treatment. Respondents were 301. QOL of most variables were improved except loss of household asset. The best scores improved in the indicators before & after Radiotherapy such as: pain from 27.77% to 30.15%, ability to self care 75.39% to 84.12%, ability to play role activity in family & society 69.84% to 75.39%, can take part in family and leisure activities 74.60% to 79.36% and same trend was found for both Radiotherapy+Chemotherapy and Chemotherapy alone and statistically significant. However, the Best scores deteriorated in case of Chemotherapy for loss of household asset was 70.42 % to 42.25% and the lowest compare to Radiotherapy and Radiotherapy+Chemotherapy together.

The New Feasibility Standards: An Economic Analysis Presenter(s): Julio Cesar Hernandez-Correa, Western Michigan University, julio.c.hernandez-correa@wmich.edu Abstract: The Program Evaluation Standards (1981, 1994, 2011) have identified particular problems or deficiencies in evaluation, given a detailed guide as to carrying out an effective and realistic evaluation, and offered general procedures that allow conducting a transparent and ethical evaluation. Recently, the Joint Committee on Standards for Educational Evaluation revised The Program Evaluation Standards (2011) and delivered the third edition. The Feasibility Standards (F standards) were one of the general categories of standards that were significantly revised. This paper studies the economic implications to an evaluation as described in two new standards in the 2011 edition of The Program Evaluation Standards: Project Management (F1) and Resources Use (F4). These new standards were compared with Cost Effectiveness (F3) in the 1994 edition. This paper explores limitations and the improvements of these standards regarding concepts such as cost-benefit, efficiency, and effectiveness.

Impact of Cognitive Behavioral Therapy on Labor Market Outcomes of Ex-Offenders Presenter(s): Mustafa Karakus, Westat, mustafakarakus@westat.com Allison Roeser, Westat, allisonroeser@westat.com Abstract: The REEOP and New Start programs are set up to provide support and services related to job creation, training, and job placement efforts for people who leave prison in Los Angeles. Both programs were designed to assess skills and employment history and match participants with a job. For those individuals needing additional skills or job training, linkages to the appropriate resources and services are made through cross-referral within the project. The two programs were very similar in all aspects except that the New Start program also had a cognitive behavioral therapy (CBT) component which created a natural experiment to evaluate the effectiveness of CBT. This study uses propensity score matching method to examine the impact of CBT on employment outcomes. Results indicate significant improvements in job placement. Results are examined in light of costs and benefits analysis to asses the value of adding CBT component to employment programs for ex-offenders.

From Program Effect to Cost Savings: Valuing the Benefits of Educational Innovation Using Vertically Scaled Test Scores And Instructional Expenditure Data Presenter(s): Valeriy Lazarev, Empirical Education, vlazarev@empiricaleducation.com Denis Newman, Empirical Education, dn@empiricaleducation.com Abstract: State and local educational administrations need to determine if the expected benefits of a program being considered for adoption outweigh the total cost of the program implementation and support. While costs can be estimated directly, derivation of a comparable estimate of program benefits is a complex problem. We develop an approach to the evaluation of educational program benefits which we term instructional time equivalent. This approach is applicable when program effects are evaluated using outcomes measured on growth scales and interprets a positive program effect as equivalent to an increase in the overall instructional time. We demonstrate our methodology and the challenges that its development entails using the results of several experimental and quasi-experimental studies that we conducted over the past few years. We produce ex post assessments of whether program adoption was economically justified using total cost estimates for each program.

Session Title: Isn't Just Talk Talk Talk: How Systems Approaches Can Work in the Real World of Boundaries and Power

Think Tank Session 864 to be held in Avila B on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Systems in Evaluation TIG

Presenter(s):

Mary McEathron, University of Minnesota, mceat001@umn.edu

Erin Watson, Michigan State University, droegeer@msu.edu

Discussant(s):

Martin Reynolds, The Open University, m.d.reynolds@open.acgçï.uk

Vidhya Shanker, University of Minnesota, shan0133@umn.edu

Abstract: Evaluators have long known, discussed and written about power imbalances evident in decisions made about how programs are funded, implemented, and evaluated. While a number of systems-thinking evaluation approaches identify multiple stakeholder perspectives and define program boundaries, two approaches - Soft Systems Methodology and Critical Systems Heuristic- can be used to address issues of power imbalances in human systems. In this interactive session, presenters share practice-based vignettes of these approaches to introduce questions of power relations and boundary judgments. Participants engage in small group discussions to explore: (1) What issues of power and boundaries are addressed using these approaches? (2) Who and/or what needs to be present to move awareness into skillful action? (3) How can these approaches address historical imbalances of power due to race, class, gender, and culture? When full group reconvenes, discussants Martin Reynolds and Vidhya Shanker reflect on group responses and with participants create recommendations for practice

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Evaluating K-12 Professional Development: English as a Second Language (ESL) Coaching and Inclusion

Roundtable Presentation 865 to be held in Balboa A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Michelle Bakerson, Indiana University South Bend, mmbakerson@yahoo.com

Abstract: Evaluators are often contracted by school districts or organizations receiving grants to develop and facilitate programs to benefit the school or organization. One such school district in Northern Indiana is a district whose K-12 teachers received professional development in coaching English as a Second Language (ESL). The evaluation was conducted to determine teacher attitudes and perceptions toward full ESL inclusion in which ESL teachers work as coaches in conjunction with classroom teachers to provide both indirect and direct services in an inclusive setting, as proposed by the school district. The evaluation was designed to be a learning tool for facilitating the improvement of the professional development provided at this school. Accordingly, a collaborative evaluation approach was utilized to actively engage the school and the teachers during the whole process. The steps, advantages, and obstacles of this evaluation will be discussed.

Roundtable Rotation II: Improving Educational Leadership Through the Development of Professional Learning Communities

Roundtable Presentation 865 to be held in Balboa A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Annie Woo, Oregon Department of Education, dranniewoo@gmail.com

Abstract: This evaluation study provided administrators of Professional Learning Communities (PLC) within eight school districts across (32 individual schools) in four states with formative and summative data to assist them in making program changes as they addressed the challenges of high school reform. We focused our efforts on the identification of the factors that contribute to the success of schools in achieving Adequate Yearly Progress through the implementation of PLC, with the aim of providing teachers with a more supportive teaching environment. Achievement data review, surveys, and interviews were conducted for the purposes of measuring the effectiveness of PLC in: a) developing effective professional learning communities in schools; b) providing teachers with both an intellectually challenging and emotionally supportive professional environment; and c) increasing student achievement. Findings pertain to PLC implementation, improvisation, challenges to evaluation, and lessons learned with relevance to high school reform.

Session Title: Examining and Understanding the Power and Impact of the Robert Wood Johnson Foundation's Community Health Leaders Programs

Panel Session 866 to be held in Balboa C on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Non-profit and Foundations Evaluation TIG

Chair(s):

Laura Leviton, The Robert Wood Johnson Foundation, llevito@rwjf.org

Discussant(s):

Claire ReineIt, Leadership Learning Community, claire@leadershiplearning.org

Abstract: This session presents the evaluation findings of three interrelated community health leadership programs supported by the Robert Wood Johnson Foundation (RWJF). The first two presentations will focus on the RWJF Community Health Leaders (CHL) Award Program which has recognized hundreds of innovative leaders who have made extraordinary contributions to increasing access to quality health care and improving health outcomes at the community level. The third presentation focuses on findings from the evaluation of the Ladder to Leadership: Developing the Next Generation of Community Health Leaders which is a collaborative initiative of RWJF and the Center for Creative Leadership. The initiative focuses on developing critical leadership competencies of early- to mid-career professionals. Key questions to be addressed are: (1) What factors are most influential in determining leaders' pathways? (3) What are the most significant commonalities among the leaders? (4) What are the key challenges and lessons learned and their implications?

Critical Leadership Pathways Among the Robert Wood Johnson Foundation's Rural Community Health Leaders Alberto Cardelle, East Stroudsburg University, acardelle@po-box.esu.edu The paper presents the results of an evaluation of the Robert Wood Johnson Foundation's Community Health Leaders (CHL) program. Specifically it identifies the critical life pathways taken by the leaders working on rural health issues. The evaluation used a mixed method approach. It used social network analysis to identify the paths that the leaders took to their positions and it developed and analyzed in-depth cases studies of 12 leaders. The evaluation integrated both analyzes to identify the pathways and the common factors across the different leadership experiences. The results show that the critical components of rural leaders' pathways included: early exposure to social injustice and an activist family member; a passion for an ideal; experience with networks, that gave them exposure to government entities and other leaders in their community; skills that allowed them to be problem solvers; and a history of seeking positions of leadership.

The Powerful Pathways of Diverse San Francisco Bay Area Community Health Leaders Hanh Cao Yu, Social Policy Research Associates, hanh_cao_yu@spra.com This presentation highlights the results of a qualitative study that drew from leadership dialogue circles, and biographical interviews of 9 San Francisco Community Health Leaders (CHLs). The evaluation analyzed the influential forces and defining events that shaped CHLs' paths to leadership and employed a framework for analysis that incorporated "critical life maps." CHLs' identities and philosophies were shaped by their families, living abroad, immigration to the USA, various mentors, coming of age during the Civil Rights Movement, and catalytic experiences to overcome adversity and find inner strength. CHLs' achieved breakthroughs despite struggles to staying true to self and community needs; maintaining organizational focus and cutting edge; collaborating effectively, and negotiating political processes. The leaders' arrival into legitimate power and greatness was meaningfully punctuated by the RWJF leadership award which further impacted their leadership trajectory, supported their networking, and enabled them to leverage the award in organizational development and programmatic work.

Evaluation Findings from Ladder to Leadership: An Emerging Community Health Leaders Program Heather Champion, Center for Creative Leadership, championh@ccl.org Tracy Patterson, Center for Creative Leadership, pattersont@ccl.org Ladder to Leadership is a 16-month leadership development program aimed at increasing leadership skills and professional networks for emerging, non-profit, community health leaders from eight communities across the US. Each cohort of up to 30 Fellows participated in three multi-day, face-to-face leadership development sessions; action learning projects; professional coaching; and developmental goal setting. A comprehensive, longitudinal evaluation was designed to measure the impact of the program on the Fellows, their organizations, and their community. Findings from the immediate post-program measures of the first three cohorts show that over 80% of Fellows and their 360né¦ Observers reported that Fellows demonstrated increased confidence, leadership effectiveness, readiness for leadership responsibilities, and readiness for promotion. Increases in their professional networks were also reported. In addition to the impact on the individuals and their organizations, impact on the Fellows communities and the sustainability of impact one year post-program will also be discussed.

Session Title: Theory and Practice: Putting It All Together

Multipaper Session 867 to be held in Capistrano A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Organizational Learning and Evaluation Capacity Building

Chair(s):

Erika Fulmer,  Centers for Disease Control and Prevention, efulmer@cdc.gov

Theoretical and Practical Implications of Evaluation Capacity Building in Taiwanese K-9 School Settings Presenter(s): Shu-Huei Cheng, National Taiwan Normal University, shcheng@ntnu.edu.tw Abstract: Numerous empirical studies have been conducted on evaluation capacity building (ECB). However, one key gap is a lack of research related to evaluation capacity (EC) at an organizational level, especially in non-American schools. Consequently, the purpose of this study was to explore the essential dimensions of EC in K-9 schools in Taiwan, along with the approaches to ECB. The study included semi-structured interviews with evaluation experts in universities as well as practitioners in K-9 schools. Consistent with the existing literature, this study identified the main dimensions of EC as supportive leadership, culture, structures, and resources. The sub-dimensions, however, took different shapes in Taiwanese schools and revealed a relationship between EC and contexts. Also, the governments, evaluation experts, and schools all played significant roles in ECB. This study concluded with implications for theory and practice.

Advancing the Value of Evaluation in a Medical School Presenter(s): Derek Wilson, University of British Columbia, derek.wilson@ubc.ca Karen Joughin, University of British Columbia, karen.joughin@ubc.ca Leonie Croydon, University of British Columbia, leonie.croydon@ubc.ca Chris Lovato, University of British Columbia, chris.lovato@ubc.ca Abstract: In medical schools across Canada, program evaluation has historically had a very limited, circumscribed presence and role within the larger organization. At the University of British Columbia, a formal evaluation unit was established in 2004 when the medical school expanded to a multi-site program model. The mandate of the unit includes evaluation of the MD undergraduate and postgraduate programs, as well as long-term studies of the impact of program expansion. Within this broad mandate, a key focus of efforts to date has been on the evaluation of the undergraduate curriculum. This paper will describe recent and historical strategies employed to build a strong, utilization-focused curriculum evaluation agenda, and to advance the “value” of evaluation by building the culture and capacity for evaluation. These strategies include broadening the scope of evaluation efforts, enhancing the utility of products, promoting engagement/participatory approaches, supporting self-directed evaluation (e.g., E-CLIPS), and establishing infrastructure.

Knowledge Creation in Healthcare Organizations as a Result of Individuals' Participation in the Executive Training in the Use of Evidence Informed Decision Making Presenter(s): Francois Champagne, University of Montreal, francois.champagne@umontreal.ca Louise Lemieux-Charles, University of Toronto, l.lemieux.charles@utoronto.ca Gail MacKean, University of Calgary, glmackea@ucalgary.c Trish Reay, University of Alberta, trish.reay@ualberta.ca Jose Carlos Surez Herrera, University of Montreal, joseko70@hotmail.com Malcolm Anderson, Queens University, andersnm@post.queensu.ca Nathalie Dubois, Montreal Public Health, ndubois@santepub-mtl.qc.ca Abstract: Evaluations of attemps to improve the use of evidence informed decision making in healthcare organizations have focused on impact on individual skills and knowledge and failed to grasp how learning processes occur in the organizations. We conducted a research project on the organizational impact of two programs aiming at developing capacity and leadership to optimize the use of evidence in decision making . To guide our evaluation, we developed a logic model based on Nonaka Dynamic Theory of Organizational Knowledge Creation. We used multiple cases studies using embedded units of analysis and relying on a triple comparative design. In each case, we collected data through interviews and documentation. Our results showed an impact on the immediate work environment of the trainees . The findings emphasized the importance of multiple pattern of interactions within the organization and of focusing on identifying contextual conditions that faciliates and impedes the programs' impact.

Training Program Staff in the Use of Action Research: An Internal Evaluator-External Evaluator Collaborative Adventure Presenter(s): Georgia Kioukis, EducationWorks, gkioukis@educationworks.org Vonda Johnson, Paragon Applied Research and Evaluation, vjohnson@paragon-are.net Abstract: EducationWorks, an education non-profit organization, runs three after-school programs located in Camden, New Jersey. When their funder mandated that an action research method be employed as part of the program evaluation, the internal evaluator and the external evaluator partnered to ensure that a participatory and collaborative action research framework was instituted. McNiff and Whitehead (2006) describe action research as 'a form of enquiry that enables practitioners everywhere to investigate and evaluate their own work.' Action research was embraced as a means to build evaluation capacity for these program sites. This paper will present details regarding the training, staff reactions to the training, staff progress using the action research approach, and perspectives on an internal-external evaluator partnership. McNiff, Jean and Jack Whitehead (2006) All You Need to Know about Action Research. Sage Publications: Thousand Oaks, CA.

Session Title: Demonstrating the Use of Evaluation to the Social Work Student, Teacher, and Practitioner

Multipaper Session 868 to be held in Capistrano B on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Social Work TIG

Chair(s):

Sarita Davis,  Georgia State University, saritadavis@gsu.edu

What was the Outcome: Using a Logic Model for Social Welfare Policy Evaluation Presenter(s): William Cabin, The Richard Stockton College, williamcabin@yahoo.com Abstract: There has been significant use of logic models in evaluation (Frechtling, 2007; Knowlton and Phillips, 2008). However, evaluation logic models have focused almost exclusively on program evaluation and program planning. This paper introduces a logic model for evaluating social welfare policy. The logic and decision of the model are discussed and specific examples presented using TANF and The Food Stamp Program.

An Evaluation of a Trauma-Informed Human Rights Perspective Curriculum for Social Work Students Presenter(s): Tom Nochajski, University at Buffalo, thn@buffalo.edu Bincy Wilson, University at Buffalo, bincywil@buffalo.edu Abstract: A trauma-informed and human rights (TI-HR) curriculum was recently integrated into the University at Buffalo School of Social Work Master's level program. The current project considers the impact of this change on students in the first and second year. An instrument was developed to evaluate knowledge, attitudes, self-efficacy, and behavioral intentions with regard to TI-HR. Foundation (n=139) and advanced year (n=181) students completed the assessment at the beginning and end of the fall 2010 semester. Initially, advanced year students showed significantly greater: knowledge, positive attitudes, levels of self-efficacy, and likelihood to engage in TI-HR behaviors than foundation year students. By the end of the semester, foundation year students were equivalent with advanced year students on all measures. Results suggest the foundation year curriculum provides sufficient background in TI-HR to allow for significant gains across all areas assessed. However, integration of TI_HR within the advanced year courses needs further assessment.

Session Title: Design, Cost, and Initial Program Model Findings From the Substance Abuse and Mental Health Services Administration (SAMHSA) Grants for the Benefit of Homeless Individuals National Cross-Site Evaluation

Panel Session 869 to be held in Carmel on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG

Chair(s):

Nahama Broner, RTI International, nbroner@rti.org

Abstract: SAMHSA's Grants for the Benefit of Homeless Individuals (GBHI) program funds grantees to provide services to expand and strengthen treatment to those with substance use and, or co-occurring mental disorders and link treatment services with housing with the goals of abstinence, housing stability and decreased homelessness. In operation since 2001, this U.S. federal program has not previously been evaluated. This panel presents the evaluation framework, a socioecological model, the design, and data collection and analysis methodology for each of the study's components-structure, process, outcome and cost--developed to address evaluation questions. We also present initial findings from the 25-site impact study regarding service models, intervention costs and leveraging of resources to develop integrative strategies for the provision of evidence-based treatment, wrap-around services, and housing. Implications for multi-site evaluation design of heterogeneous programs will be discussed in the context of assessing the effectiveness of interventions on abstinence, homelessness and housing.

Grants for the Benefit of Homeless Individuals (GBHI) Cross-Site Evaluation Framework, Design and Methods Jim Trudeau, RTI International, trudeau@rti.org Nahama Broner, RTI International, nbroner@rti.org Karl Maxwell, United States Department of Health and Human Services, maxwell@samhsa.hhs.gov, The SAMHSA GBHI cross-site evaluation framework assumes four interrelated propositions of a socioecological model in which first, clients are located within the local grantee project site, which is located within the local treatment system and the broader community context. Second, there are interactive relationships among these ecological levels. Third, each level contains appropriate evaluation components (i.e. structure, process, outcome, and cost elements). Fourth, grantees are located not only in their local communities but also in the "virtual community" of the federal program that directly influences grantees and indirectly influences their clients and treatment systems in which they are embedded. The cross-site evaluation includes 137 programs, with an impact study of 25 programs. The presentation describes the goals of the evaluation, research questions, cross-site design, and data collection and analysis methodology developed in consultation with a national expert panel.

Services and Housing Model Approaches: A Cross-site Evaluation of the Grants for the Benefit of Homeless Individuals (GBHI) Nahama Broner, RTI International, nbroner@rti.org Jim Trudeau, RTI International, trudeau@rti.org Alexander Cowell, RTI International, acowell@rti.org Arnie Aldridge, RTI International, aaldridge@rti.org Karl Maxwell, United States Department of Health and Human Services, maxwell@samhsa.hhs.gov, This presentation describes the approaches to integrating evidence-based treatment, wrap-around services and housing developed by 25 programs funded in 2009 through the SAMHSA GBHI program serving veterans, women and families, youth and those involved in the criminal justice system with substance use and, or co-occurring mental disorders. Based on structured interviews, document extraction, focus groups, site visit observation, and client self-report data, we will present the primary models that serve to organize grantee approaches to substance use, services and housing. We will describe a number of model elements including setting, clinical population characteristics, treatment and housing criteria, evidence-based practices, supports and financing that were used to develop these models across sites. Lessons learned and innovations with regard to differential implementation of evidence-based treatment and approaches by subpopulation will also be delineated. This presentation will conclude with the planned use of these models in analyses and the implications of this approach.

Costs of Service Provision for Participants in Grants for the Benefit of Homeless Individuals Program: Approach and Preliminary Results Arnie Aldridge, RTI International, aaldridge@rti.org Alexander Cowell, RTI International, acowell@rti.org Nahama Broner, RTI International, nbroner@rti.org Carolina Barbosa, RTI International, cbarbosa@rti.org The presentation reports on the approach and preliminary findings of the economic component of the SAMHSA GBHI cross-site evaluation. We describe the methods of obtaining the costs of services directly covered by the grantee as well as the correlated services that the grantee leverages. These correlated services are from within the grantee agency, from partner providers and from the existing social support system. The costs of services provided under the grant are estimated using a protocol that includes a structured instrument completed by the evaluator and the grantee in tandem. That protocol also gathers qualitative data on key service areas to which clients are connected, in which clients may not have otherwise received services, but that are not covered directly by the SAMHSA grant. Emphasis is placed on the scope, challenges, and alternative approaches for collecting the data. Preliminary cost estimates are presented.

Taxonomy of Approaches to Transportation for Grants for the Benefit of Homeless Individuals (GBHI) Program Participants Arnie Aldridge, RTI International, aaldridge@rti.org Alexander Cowell, RTI International, acowell@rti.org Nahama Broner, RTI International, nbroner@rti.org In this presentation we focus on the role of transportation models in the 25 GBHI programs. Transportation is an oft-noted barrier for clients accessing substance abuse and mental health treatment. As emphasis on recovery support and wrap-around services grows, transportation emerges as an even more important factor in a program's success. Differences in a program's geographic location, its approach to integrating services and housing, its organizational structure, and its target population all can influence the program's ultimate transportation services, which can themselves influence the programs' service delivery models. In this study we provide a taxonomy of the approaches to transportation services and analyze why they were chosen. We describe the pros and cons for each approach in terms of service model integration, fidelity to evidence-based practices, program outputs, costs, efficiency and synergy within the program and the grantee organization, client experiences and outcomes, and implications for program sustainability.

Session Title: Add Usability Testing to Your Evaluation Toolbox

Demonstration Session 870 to be held in Coronado on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Distance Ed. & Other Educational Technologies TIG

Presenter(s):

Christine Paulsen, Concord Evaluation Group, cpaulsen@concordevaluation.com

Abstract: Many of the programs and initiatives that we evaluate today are technology-based. It is not uncommon for initiatives to provide information to their target audiences via websites, while other interventions are delivered with software applications to mobile, handheld or other devices. To properly evaluate such initiatives, the evaluator must consider the usability (user-friendliness and accessibility) of the technology components. This demonstration will provide participants with an overview of the most common usability method--the one-on-one usability session, including the think aloud procedure. Participants will learn how to develop a usability testing script, how to recruit participants, how to run a usability session, how to analyze the data. Video examples of actual testing sessions will be included.

Session Title: Evaluating Complex Collaborative Science Initiatives: Utilization of Logic Models in Four Clinical Translational Science Institutes

Panel Session 871 to be held in El Capitan A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Health Evaluation TIG

Chair(s):

Kevin Wooten, University of Houston, Clear Lake, wooten@uhcl.edu

Discussant(s):

Nick Smith, Syracuse University, lsmith@syr.edu

Abstract: This panel will address the use of logic models in environments that are highly complex, non-linear, process driven, and institutionally political. Drawing upon a representative sample of academic institutions receiving Clinical and Translational Science Awards (CTSA) from the National Institute of Health, four presenters will address the role of evaluation in CTSA settings that engender controversy and inquiry. Value conflicts such as the need for scientific rigor versus emergent design, outcome versus process evaluation, and empirical methods versus action research are all difficult to address using a logic model approach. Our presentations will involve using logic models to: 1) simultaneously evaluate outcomes and development; 2) integrate multiple constituencies for collaboration and a common vision; 3) develop and evaluate multidisciplinary teams in accordance with effective team science processes; and 4) utilize stakeholders and constituencies in the real-time design of evaluation tools to address relevance and dynamic institutional cultures.

Linear Logic Modeling in the Non-Linear World of Translational Research Infrastructure Janice Hogle, University of Wisconsin, jhogle@wisc.edu D Paul Moberg, University of Wisconsin, Madison, dpmoberg@wisc.edu Christina Hower, University of Wisconsin, Madison, cjhower@wisc.edu Bobbi Bradley, University of Wisconsin, Madison, bradley.bobbi@mcrf.mfldclin.edu At the UW Institute for Clinical and Translational Research, embedded evaluators use nested logic models to: summarize complex layers of intent, assist with program improvement, encourage an evaluative perspective, communicate program achievements, and identify evaluation tasks and metrics. We think we have a handle on merit and worth, many processes and some outcomes, and formative and summative assessments. However, we continue to struggle with evaluating impacts within complex CTSA environments. CTSAs are interventions in research processes where fuzzy definitions and boundaries overlap with unpredictable emergent changes in program scope. We are not working in a simple environment of high certainty and high agreement about what to do, how to do it, and what impact to expect. Logic models, which work best in situations of static fixed cause and effect models, have become prevalent among CTSA evaluators. We address the utility of logic models within a developmental evaluation of CTSA implementation.

Implications of Logic Modeling in Large-scale Capacity-building Initiatives That Indicate the Need for Emergent Evaluation Designs Deborah Fournier, Boston University, fournier@bu.edu Evaluators studying collaborative research networks grapple with the inherent complexity and highly adaptive nature of Clinical and Translational Science Institutes (CTSI). Many evaluators across the national network of 55 CTSIs use logic models prospectively to communicate a common vision across multiple projects and sites, draw formative and summative evaluation plans based on specified metrics, guide operational decision-making, and report collaborative scholarly products to the funding agency. The author will present how logic models are used at Boston University's CTSI and examine a critical question: "To what extent are logic models well-suited to evaluating the profusion of moving parts involved in such complex infrastructure development and capacity-building?" Using a systems perspective, the case shows the merit and constraints of models related to emergent evaluation designs. The paper has relevance to evaluation practice regarding complex, highly contextualized initiatives and, more specifically, the NIH nation-wide initiative on advancing Clinical and Translational Science.

Utilizing Logic Models in Developing and Evaluating Multidisciplinary Translational Research Teams Kevin Wooten, University of Houston, Clear Lake, wooten@uhcl.edu Jean Freeman, University of Texas Medical Branch, jfreeman@utmb.edu Much recent interest has been generated around team based science. The discipline surrounding the evaluation of team science is in its infancy, yet holds much promise to unlocking the keys to collaboration, innovation, and scientific breakthroughs. The CTSA award at the University of Texas Medical Branch is completely focused on team based science. The authors will present how logic models have been used to both evaluate team based science, as well as to evaluate team processes along suggested criteria (e.g., NIH Fieldbook on Team Science and Collaboration). Here, the use of logic models to both evaluate the science of multidisciplinary translational research teams as well as the development and evaluation of team processes purported to be of significance, will be depicted. The presentation will focus on the use of hybrid forms of logic models designed to evaluate the dynamic, non-linear, developmental sequence and processes required of effective scientific teams.

Using an Interactive Logic Model to Evaluate a Clinical Translation Science Award Program Julie Rainwater, University of California, Davis, julie.rainwater@ucdmc.ucdavis.edu Stuart Henderson, University of California, Davis, stuart.henderson@ucdmc.ucdavis.edu In program-driven theory evaluations, logic models articulate a program's direction and goals. However, traditional logic models can be limiting for complex programs with multiple objectives and stakeholders. Constructing a model that actively engages a range of stakeholders, illustrates a comprehensive program story, and serves as a fluid blueprint throughout the full evaluation process can reduce some of these limitations. This presentation will highlight our development of an interactive logic model, constructed with DoView software, for the UC Davis Clinical and Translational Science Center. We will show how an interactive model allows us to 1) capture the Center's high-level and program specific outcomes, 2) link multiple logic models from Center programs, and 3) build a model in real time with stakeholder input. We will discuss benefits and challenges of using interactive logic models, including the importance of empowerment and inclusion and the difficulty of creating a consistent narrative.

Session Title: A Seven Year External Evaluation of an International Aid Program in 25 Countries

Panel Session 872 to be held in El Capitan B on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Michael Scriven, Claremont Graduate University, mjscriv1@gmail.com

Abstract: Four of the key participants will describe their experiences, approaches, and lessons learned, in the course of a external impact evaluation that went on for seven years and involved putting evaluation teams into villages and homes in 25 countries from all the non-polar continents. Some issues and achievements that may be of general interest include: selecting and supervising interpreters and local researchers; facilitating open communications; developing a complex model that was both program specific and readily adaptable to other programs; finding a way to establish causation beyond reasonable doubt without using control groups; retaining reasonable independence despite a long and amiable relationship with the client (including contracts that were always limited to one year at a time); getting at values and attitude changes as well as the directly observable housing, nutrition, and economic changes.

The Role of the Evaluation Manager Thomaz Chianca, COMEA Evaluation Ltd, thomaz.chianca@gmail.com “The Role of the Evaluation Manager” will review some of the statistics on number of interviews at different levels, and the complex logistics involved, as well as lessons learned about handling resistance to evaluation, or to having the occasional all-female teams, and some of the interesting issues about how to help teams when the country they were in ran into typhoons or revolutions.

The Client’s Point of View Rienzzie Kern, Heifer International, rkern@heifer.org “The Client’s Point of View” will cover his take on the Heifer management support for an external impact evaluation; his own hopes and conclusions about it; and internal acceptance of the reports and recommendations.

Design and Methodology Issues Michael Scriven, Claremont Graduate University, mjscriv1@gmail.com “Design and Methodology Issues” will look in particular at (i) the management of the 20+ values that Heifer wanted impact on, and how to avoid a merely goal-based approach; (ii) the development of a systematic alternative to the RCT approach (namely, the GEM model, standing for General Elimination Methodology) and its defense against the usual criticisms plus the more serious but rarely mentioned ‘upswing terminator’ problem; and (iii) the interesting case of a Heifer middle manager who decided he would use threats of non-cooperation in order to get changes made in our design.

Paul Clements, Western Michigan University, paul.clements@wmu.edu Refining the Quantitative Approach to Cost Analysis

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: How to Train Evaluators to Interview Deaf and Hard-of-Hearing Participants: Strategies for an Effective Interview

Roundtable Presentation 873 to be held in Exec. Board Room on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Lesbian, Gay, Bisexual, Transgender Issues TIG and the Teaching of Evaluation TIG

Presenter(s):

Jennifer Morrow, University of Tennessee, jamorrow@utk.edu

Ann Cisney-Booth, University of Tennessee, acisneybooth@utk.edu

Lisa Rimmell, University of Tennessee, lrimmell@utk.edu

Abstract: In this roundtable we will discuss our experiences interviewing deaf and hard-of-hearing participants in our evaluation projects. We will discuss the various ways that deaf and hard-of-hearing individuals communicate (e.g., American Sign Language, Signed English, Cued Speech, Auditory-Oral Method). We will review how evaluators can best prepare beforehand (i.e., interview protocol, room arrangements) to interview an individual who is deaf or hard-of-hearing. Lastly, we will spend most of the time leading a discussion with the audience members on strategies for conducting an effective interview with participants who are deaf or hard-of-hearing.

Roundtable Rotation II: Don't Forget Us! Standardizing Methods of Data Collection That are Inclusive of Transgender and Gender Non-Conforming Individuals

Roundtable Presentation 873 to be held in Exec. Board Room on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Lesbian, Gay, Bisexual, Transgender Issues TIG and the Teaching of Evaluation TIG

Presenter(s):

Loretta Worthington, Rainbow Health Initiative, loretta.worthington@rainbowhealth.org

Rachel Fletcher, Rainbow Health Initiative, rachel.fletcher@rainbowhealth.org

Abstract: Historically, local and national health data collection and evaluation efforts have mostly excluded LGBTQ people. Consequently, there is currently no set of standardized questions to collect sexual minority and gender identity information. Gender non-conforming individuals are often lost in data collection efforts. Evaluation methods must develop appropriate and standardized methods of asking about gender identity that result in data collection of relevant information to better serve the needs of gender non-conforming populations in the health, social justice, and social policy fields. Rainbow Health Initiative will discuss health assessment data collection over a 3-year period, including the survey instrument design, complications, and final evaluation questions leading to broad data collection with regards to sexual minorities and gender identity representation. If these questions become standardized, it could significantly increase research data on gender non-conforming populations and provide the means for more research, programs, and services.

Session Title: The World Is Not Flat

Multipaper Session 874 to be held in Huntington A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Integrating Technology Into Evaluation

Chair(s):

Paul Lorton Jr,  University of San Francisco, lorton@usfca.edu

Discussant(s):

Matthew Galen,  Claremont Graduate University, matthew.galen@cgu.edu

Creation of Public Use Files: Lessons Learned from the Comparative Effectiveness Research Public Use Files Data Pilot Project Presenter(s): Erkan Erdem, IMPAQ International LLC, eerdem@impaqint.com Sergio Prada, IMPAQ International LLC, sprada@impaqint.com Abstract: We describe the lessons learned from the creation of Basic Stand Alone (BSA) Public Use Files (PUFs) for the Comparative Effectiveness Research Public Use Files Data Pilot Project (CER-PUF). CER-PUF is aimed at increasing access to CMS claims data sets through the creation of public use files that: do not require user fees and DUAs, have been de-identified, and provide analytic utility to researchers. We describe the steps taken in the project to strike the right balance between data utility and privacy protection. We draw lessons learned from three tasks: (i) the creation of each PUF involving design of the sample data, analysis of variables, analysis of de-identification strategies, risk analysis, and documentation, (ii) environmental scan including stake-holder interviews, case-studies of de-identified individual level public use data, and literature review and legal analysis, and (iii) review of the needs of researchers and statistical de-identification methods that are acceptable to them.

The World is Not Flat: When to Use Relational Databases in Evaluation and Research Presenter(s): Todd Ruitman, Cobblestone Applied Research & Evaluation Inc, todd.ruitman@cobblestoneeval.com Rebecca Eddy, Cobblestone Applied Research & Evaluation Inc, rebecca.eddy@cobblestoneeval.com Namrata Mahajan, Cobblestone Applied Research & Evaluation Inc, namrata.mahajan@cobblestoneeval.com Abstract: The problem: Some evaluators try to store their data in programs based on comfort level rather than appropriateness. We intend to tackle one aspect of the problem by sharing our organization's evolution from using flat file programs like Microsoft Excel to using a relational database (i.e., Microsoft Access) to store some of our evaluation information or data. For example, the evaluation of educational programs can require multiple levels of information that needs to be stored for quick access. District level information (e.g., address, superintendent), school level information (e.g., teacher names, standardized testing schedule), teacher level information (e.g., years teaching, email address), and student level information (e.g., assessment score, gender). Combing all of this information in one flat file is inefficient. We will discuss strategies for organizations to integrate relational databases into their data management systems and tips on choosing the right programs to store study data.

Session Title: The Role of Government Evaluation Policies and How It Affects Quality of Services

Think Tank Session 875 to be held in Huntington B on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Government Evaluation TIG

Presenter(s):

Stanley Capela, HeartShare Human Services, stan.capela@heartshare.org

Discussant(s):

Kristin Kaylor Richardson, Western Michigan University, kkayrich@comcast.net

Sathi Dasgupta, SONA Consulting Inc, sathi@sonaconsulting.net

Gabriel M Della-Piana, Independent Consultant, dellapiana@aol.com

Connie Kubo Della-Piana, National Science Foundation, cdellapi@nsf.gov

Abstract: The purpose of this think tank is to raise several questions. We would begin by asking whether or not government policy at the federal, state or local level affects the quality of government-sponsored services and service delivery systems or programs? If we agree, the group would then explore several other questions. First, what is the role of government evaluation policies in affecting quality of services? Second, are there concrete examples where government policy on evaluation had a positive effect on ensuring quality of services? Third, can one conclude that one way government policy can ensure quality of services is to require programs to seek out accreditation as a way to evaluate its' services?

Session Title: A NASA Approach to Program Evaluation: Use of Social Science Methods to Engineer Education Projects in NASA Education's Portfolio

Multipaper Session 876 to be held in Huntington C on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Government Evaluation TIG

Chair(s):

Brian Yoder, National Aeronautics and Space Administration, brian.yoder@nasa.gov

Abstract: NASA Office of Education has collaborated with a team of external evaluators to develop a comprehensive plan for evaluating its portfolio of education programs and using the findings from evaluations for program improvement. This multipaper session examines how NASA Education is using evaluation to develop knowledge about its programs for the purpose of decision making, and provides examples drawn from two national project evaluations. The first paper outlines NASA's approach to evaluation and utilization of findings. The second and third papers describe the evaluations of two national projects within NASA's Elementary and Secondary Education Program. The fourth paper shares the perspective of the two national project managers. The panel will conclude with an audience discussion of the aptness of the described stakeholder and evaluation utilization.

Overview: A NASA Approach to Program Evaluation

Brian Yoder, National Aeronautics and Space Administration, brian.yoder@nasa.gov

One of NASA's approaches to program evaluation aims at integrating program evaluation with project development so that education projects at NASA have a good chance of showing a measurable impact after a few years of refinement. This presentation provides an overview of some important considerations that informed this approach. These considerations include: developing a program evaluation process that reflects NASA's engineering culture and emphasizes team work and innovation; adheres to NASA's project planning template known as 7120.7; and aligns with current federal program evaluation guidance. This presentation will also highlight some less obvious intended goals of this evaluation approach like merging researcher knowledge and practitioner knowledge to better understand how program activities contribute to intended outcomes.

Evaluation of NASA Explorer Schools: The Formative Stage

Alina Martinez, Abt Associates Inc, alina_martinez@abtassoc.com

Sarah Sahni, Abt Associates Inc, Sarah_Sahni@abtassoc.com

Responding to recommendations from the National Research Council committee that reviewed NASA's elementary and secondary education projects, (1) NASA embarked on a redesign of the NASA Explorer Schools (NES) project in 2008. At each stage of the redesign, NASA has integrated evaluation activities and incorporated findings for program improvement. As part of the pilot activities (Spring 2010), NES gathered data from teachers and students to identify ways to improve the project's performance, and the NES project incorporated these lessons into the project for its September 2010 launch. The design of the formative evaluation has involved stakeholders and will lead to program modifications. The evaluation efforts ultimately will lead to an outcomes evaluation that investigates intended program outcomes, as laid out in the program logic model. (1) National Research Council. (2008). NASA's Elementary and Secondary Education Program: Review and Critique. Committee for the Review and Evaluation of NASA's Precollege Education Program, Helen R. Quinn, Heidi A. Schweingruber, and Michael A. Feder, Editors. Board on Science Education, Center for Education. Division of Behavioral and Social Sciences Education. Washington, D.C. The National Academies Press.

Evaluation of NASA's Summer of Innovation Project

Hilary Rhodes, Abt Associates Inc, hilary_rhodes@abtassoc.com

Kristen Neishi, Abt Associates Inc, kristen_neishi@abtassoc.com

In 2010, NASA's Office of Education launched Summer of Innovation, a NASA-infused summer experience for middle school students who underperform, are underrepresented, and underserved in science, technology, engineering, and math (STEM) fields. Since inception, process and outcomes evaluation has been an integral part of the program's development. By collecting planning and implementation data from awardees through interviews and reporting forms, and outcomes data from participating student and teachers through surveys, the evaluation has codified the lessons learned over the course of its pilot, producing actionable insight that has supported NASA's modification of the program for summer 2011. Formative evaluation efforts are continuing to support the program's continued implementation, to identify promising practices and models meriting more rigorous outcomes evaluation, understand how the awardees meet NASA requirements and the feasibility of these expectations, and continue to generate lessons learned for future implementations of SoI and of NASA's education activities more broadly.

Evaluation Utilization from NASA Project Managers' Perspectives

Rob LaSalvia, National Aeronautics and Space Administration, robert.f.lasalvia@nasa.gov

Rick Gilmore, National Aeronautics and Space Administration, richard.l.gilmore@nasa.gov

The NASA Explorer Schools (NES) and NASA Summer of Innovation (SoI) programs have integrated evaluation into the design, development, and refinement activities of the program. National project managers will discuss how building evaluation at the ground level of the program has differed from work on previous projects, and how evaluation has informed their thinking about the national projects and the individual sites. They will also provide reflections on the process of working closely with evaluators beginning at the early stages of the projects and this evaluation, describing what has worked well as well as what has been challenging.

Session Title: Building a Community to Set a Direction for Science Technology Engineering and Mathematics (STEM) Evaluation: What Can We Learn From Each Other and What Will Best Support Collaboration?

Think Tank Session 877 to be held in La Jolla on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Jack Mills, Independent Consultant, jackmillsphd@aol.com

Discussant(s):

Veronica Smith, data2insight LLC, veronicasmith@data2insight.com

Kathleen Haynie, Haynie Research and Evaluation, kchaynie@stanfordalumni.org

Tom McKlin, The Findings Group LLC, tom@thefindingsgroup.com

Abstract: This session will foster a conversation among evaluators of programs aimed at increasing the quality of science, technology, engineering and math (STEM) education. STEM program aims include: increasing precollege student achievement, increasing scientific literacy among the public, and broadening participation of underrepresented groups in STEM careers. Four experienced STEM evaluators began an on-going discussion following AEA-2010, hoping to elevate the theory and practice of STEM evaluation by sharing insights, concepts, tools and tools. The think tank will be organized around following questions: 1. Should we form a group devoted solely to STEM Evaluation? How shall we organize ourselves? 2. What barriers prevent freely sharing approaches, methods, instruments, reports, and broad findings among STEM evaluators? How can we remove these barriers? 3. How might the quality of our work improve by being a part of a STEM evaluators' community? 4. What are next steps for building this community?

Session Title: Evaluation Of Learning Processes As Dialogue Research

Multipaper Session 878 to be held in Laguna A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Research on Evaluation

Chair(s):

Annette Rasmussen, Aalborg University, anra@learning.aau.dk

Abstract: Recent years have seen quantitative evaluation research achieve a dominant position on the agenda of both social scientists and decision makers. This is a result of the widespread interest in evidence-based knowledge. The result is that qualitative dialogue research is being placed in a more marginal position when it comes to government funded research projects. However, there is a great scientific potential in dialogue research when it comes to conceptualizing meaning, social processes and contexts of learning. This multi-paper session addresses the issues, basic assumptions and contextual background of such dialogue-oriented approaches in education research from different social science perspectives. The first contribution discusses organizational aspects of conducting qualitative dialogue research. The second and third contribution look at research projects based on dialogue research with special regard to the methodological and ethical dilemmas that arise from this approach and need to be handled by the researcher.

Dilemmas of Just-in-Time Evaluation Research

Palle Rasmussen, Aalborg University, palleras@learning.aau.dk

This paper draws on experiences from a 3-year combined development and research project in general adult education done in three Danish regions. The aim of the project was to link courses in general adult education closer to workplaces and local communities. In the development part experimental course designs were formulated and tried out, while in the research part the conditions for and impact of the course designs were assessed through a number of case studies. The research design was open; the research tasks were decided in continuous negotiation between project managers and researchers. This assured that research resources were directed at tasks relevant to the overriding development objectives, but it also revealed differences between developers and researchers in the perception of relevance and validity of knowledge. The papers will discuss risks and potentials in this kind of collaboration between developers and researchers.

Dilemmas of Sense-making Evaluation and Assessment Research

Nanna Friche, Aalborg University, nanna@learning.aau.dk

This paper draws on experiences from a PhD-study of evaluation and assessment practices in a Danish community college. The aim of the study was to investigate how teachers and students make sense of non-everyday phenomenon like assessment and evaluation. With inspiration from user participating evaluation models (The BIKVA Model) the study was based on an explorative approach and the use of qualitative methods applied step-by-step starting with observation studies, followed by focus group interviews and completed by individual interviews. This approach revealed several methodological and ethical dilemmas of asking questions on sense-making processes. In addressing the questions; "How do people make sense of assessment?" and "What is credible evidence in assessment according to people?" this paper will discuss dilemmas of sense-making (and dialogue-oriented) assessment and evaluation research.

Dialogue Research Focused on Educational Frameworks

Annette Rasmussen, Aalborg University, anra@learning.aau.dk

This paper will describe the social science basis and background for a dialogue research approach in education. It takes as point of departure the evaluation requirements given in education development projects that are led by specific aims and policies. Evaluations following a dialogue approach are interested in, not only summing up the outcome of processes, but in understanding the learning processes during the initiative, including both the underlying interests of the participants and their understandings of the development initiative in question. So the approach is characterized by its orientation towards processes as well as participants, towards both structure and agency. In the paper these dimensions will be further outlined, related to case studies on education, and discussed with special regard to methodology, research methods and ethical considerations.

Session Title: Tools That Value Vulnerable Populations

Multipaper Session 879 to be held in Laguna B on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Disabilities and Other Vulnerable Populations

Chair(s):

Frank Martin,  Mathematica Policy Research, fmartin@mathematica-mpr.com

Valuing the Consumer: Logic Model Development for Programs Serving Marginalized Populations Presenter(s): Dianna Newman, Universtity at Albany, SUNY, dnewman@uamail.albany.edu Anna Lobosco, New York State Developmental Disabilities Planning Council, anna.lobosco@ddpc.ny.gov Abstract: Evaluation of systemic change endeavors has shown participatory methods of documenting and evaluating systems change to be more effective than top-down/management decision-making approaches. To determine if a program, policy, or legislative endeavor has changed the system, the voices and values of marginalized stakeholders are needed; however, direct inclusion of vulnerable consumers frequently is overlooked in the development of logic models. This is especially true for models related to systems change efforts. These models, reflecting policy change and legislation, generally include input from program staff and advocates speaking for program consumers, but actual consumer voices and values may not be directly included. Instead, programmatic values of time, resources, and prior experience frequently outweigh inclusion. This paper addresses how to: 1) value disadvantaged voices, not just represent them, 2) include these voices in systemic change logic models, and 3) provides examples (successful and not so successful) of inclusion and lessons learned.

Evaluating Research-to-Practice in Disability: A Knowledge Value Mapping Approach Presenter(s): Frank Martin, Mathematica Policy Research, fmartin@mathematica-mpr.com Juan Rogers, Georgia Institute of Technology, jdrogers@gatech.edu Abstract: This presentation will describe the use of knowledge value mapping (KVM) for evaluating knowledge translation (KT) initiatives in the disability arena. KT has emerged recently in the health science community as a means to address perceived gaps in the application of the best research to treatment of disease. Specifically, in the area of disability and rehabilitation research, federal policymakers have identified KT as an area for critical evaluation and outcome achievement. This presentation analyzes some of the issues raised by the notion of KT. First, it puts KT in the broader context of the evaluation of knowledge flow problems. Second, it introduces the knowledge value mapping framework as an avenue for addressing the fundamental issues that KT raises for research-to-practice evaluation. Third, it illustrates the application of the framework with a KVM case study of accessible currency.

Vulnerable Confidentiality: How to Balance Confidentiality with Data Collection of Vulnerable Populations Presenter(s): Sarah Chrestman, Louisiana Public Health Institute, schrestman@lphi.org Susan Bergson, Louisiana Public Health Institute, sbergson@lphi.org Michael Robinson, Louisiana Public Health Institute, mrobinson@lphi.org Jack Carrel, Louisiana Office of Public Health, jack.carrel@la.gov DeAnn Gruber, Louisiana Office of Public Health, deann.gruber@la.gov Snigdha Mukherjee, Louisiana Public Health Institute, smukherjee@lphi.org Abstract: The issue of protecting participants' confidentiality from an evaluation standpoint becomes increasingly more complex when working with vulnerable populations. Louisiana Positive Charge (LA PC) is a multi-component, statewide intervention to link HIV+ persons (including those incarcerated) into medical care. These clients are of a low socioeconomic status and mostly uninsured with multiple unmet needs who tend to move from clinic to clinic. While many partners are trained in HIPAA guidelines, most of the interventionists have never been trained in human subjects and ethics. Confidentiality issues around how data is collected, transferred, and shared as well as how to effectively and discreetly conduct outreach to locate clients who are out of care and link them into medical care arose during the implementation of LA PC. Topics to be discussed include challenges to maintaining confidentiality and potential solutions for improving the capacity of maintaining confidentiality with partners.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: The Role of the Evaluator Reconsidered

Roundtable Presentation 880 to be held in Lido A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the International and Cross-cultural Evaluation TIG

Presenter(s):

Ruofei Tian, Shenyang Normal University, rftian@163.com

Abstract: Education is needed by human being for two basic functions: social and individual. In nowadays China however, too many social values have been placed on the outcome of education under the disguise of 'All for the children'. When much is talked about gender differences, cultural influence, social classification, policy/decision making or public interests in an evaluation study, the interests of students involved which should indeed be given priority are mostly neglected. The role of the evaluator is therefore to serve as the spokesperson of the students whose voices are in many cases overwhelmed in the negotiations among different stakeholders. To do so, a shift of perspective in research is needed to view students as self-organized psychological systems exchanging information with their environment.

Roundtable Rotation II: Building Monitoring and Evaluation System in Georgia

Roundtable Presentation 880 to be held in Lido A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the International and Cross-cultural Evaluation TIG

Presenter(s):

Ketevan Chomakhidze, Georgian Evaluation Association, kchomakhidze@evaluation.org.ge

Abstract: Georgian Evaluation Association (GEA) is a pioneer and the very first and only formal National Association among Former Soviet Union Countries. This paper presents what has been done by the GEA in various directions to achieve its major goal - form a branch of evaluation that will meet international standards. GEA start with analyze the existing situation in evaluation field and based on these results designed its strategic goals. The major achievement of GEA is developing of Institutional Model of M&E in Adjara Autonomies Republic of Georgia. GEA maintains the growth of qualified evaluator cadres and enhances their professional skills. For this purpose, GEA in association with IPEN (International Project Evaluation Network) is organizing the international conference 27-30 September, to be held in Batumi Georgia. Conference will be preceded by pre conference workshops that will be led by world-wide evaluation experts.

Session Title: Evaluating Quality Improvement Projects in Public Health Departments: Lessons From the Robert Wood Johnson Foundation's Quality Improvement Collaborative

Panel Session 881 to be held in Lido C on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Health Evaluation TIG

Chair(s):

William Riley, University of Minnesota, riley0011@umn.edu

Discussant(s):

Brenda Henry, Robert Wood Johnson Foundation, bhenry@rwjf.org

Elizabeth Lownik, University of Minnesota, beth.lownik@gmail.com

Abstract: Quality improvement (QI) is an increasingly important competency for local health departments, and multiple initiatives are currently taking place to implement QI projects health departments around the country. As this work continues, evaluation of these QI projects is an essential component to assess success and elucidate lessons for QI implementation and scaling up in public health departments around the country. The Robert Wood Johnson Foundation funded a collaborative of 13 public health departments across the country to implement QI projects and to hire evaluators to begin the important work of assess the projects and compiling information for dissemination. This session brings evaluators from several of the projects in this collaborative to present their findings regarding how to evaluate Quality Improvement projects in local health departments.

Evaluating a Family Planning Quality Improvement Project: Lessons Learned Kelli Kenison, University of South Carolina, kenison@mailbox.sc.edu Doug Taylor, South Carolina Department of Health and Environmental Control, taylordj@dhec.sc.gov The South Carolina Department of Health and Environmental Control (SC DHEC) conducted a quality improvement (QI) project in two of its local health department family planning clinics. The goals were to increase caseload, improve customer satisfaction, increase staff satisfaction, through the use of a QI training and support used in combination with a CDC software-supported system called WinPFA. Quality improvement consultants provided training and technical assistance to staff in the two intervention family planning clinics. The evaluation was designed to document delivery, reach, and impact of the Quality Improvement Intervention and included two comparison family planning clinic sites. In this presentation, the evaluators will share the qualitative and quantitative evaluation measures used and the numerous lessons learned about the process of evaluating Quality Improvement projects in health department clinic settings.

Improving the Efficiency of Local Public Health Using Quality Improvement Processes: Small Change Yields Significant Success in Berrien County, Mi Theresa Green, Berrien County Health Department, tgreen@bchdmi.org Angela Martin, Michigan Department of Corrections, martina3@michigan.gov In collaboration with Robert Wood Johnson Foundation, the Berrien County Health Department (BCHD) completed an eighteen month initiative to explore whether purposeful continuous quality improvement would increase efficiency and quality in public health delivery. Specific focus areas were the Children Special Health Care Services (CSHCS) and Environmental Health Food Inspections Service. BCHD employed the Institute for Healthcare Improvement's Model for Improvement as the framework for our CQI intervention. The evaluation has assessed both processes and outcomes using static analysis including comparison measures but also dynamic analysis including trend lines and run charts to examine process variation. Findings show that initiating continuous quality improvement interventions yielded significant improvements in both efficiency and quality in two unique service areas that far exceeded expectations. Through CQI efforts, health departments are better equipped to serve their clients, and to keep people safe from health threats thus improving the health of a community.

The Engineer, the Evaluator, and the Epidemiologist: An Evaluation of a Six Sigma Intervention in a Public Health Clinic Sarah Stawiski, Center for Creative Leadership, stawiskis@ccl.org Tracy Patterson, Center for Creative Leadership, pattersont@ccl.org Emily Hoole, Center for Creative Leadership, hoolee@ccl.org In 2009, The Center for Creative Leadership, the Guilford County Department of Public Health and NC A&T University partnered on a project to evaluate the effectiveness of Six Sigma for process improvement within the clinics. Of particular interest to the evaluation was how leadership culture within a public health setting may help or hinder successful implementation of Six Sigma. Additionally, the evaluation provided real-time data to the team including key stakeholders' attitudes and knowledge about QI, perceptions of health department leadership, and patient and employee satisfaction. In addition, the evaluation documented broader successes and challenges of implementing a QI effort in a public health setting. This project provides insight about the effectiveness of implementing Six Sigma in a healthcare setting, leadership culture and its relationship to QI success. This session would benefit anyone interested in better understanding and measuring context as a key variable in process improvement efforts in healthcare.

Implement Performance What?: Strategies for Launching a Quality Improvement Learning Collaborative in a Large Urban Health Department Dawn Jacobson, County of Los Angeles Department of Public Health, djacobson@ph.lacounty.gov Debra Lotstein, RAND Corporation, lotstein@rand.org The Los Angeles County Public Health Department established an internal learning collaborative to implement multiple ongoing quality improvement (QI) projects. Teams from eight divisions worked together for ten months to apply QI methods to a priority issue unique to their division. A participant self-assessment (n=25) showed 90% of participants are likely/very likely to use QI methods in future work and nearly 80% believe their team was successful/very successful in achieving their project aims. One quarter of participants wanted more training and coaching and no participants thought there was too much training or coaching. These results confirm the feasibility of introducing a formal QI approach in a collaborative setting for a diverse group of public health staff through interactive group learning sessions and team specific coaching sessions. A higher intensity of coaching may be needed initially, which may decrease over time as department staff becomes familiar with QI methods.

Summative Evaluation Results for a Local Public Health QI Initiative to Improve Immunization Rates William Livingood, Georgia Southern University, william_livingood@doh.state.fl.us Radwan Sabbagh, Duval County Health Department, radwan_sabbagh@doh.state.fl.us A Plan-Do-Study-Act (PDSA) approach to Quality Improvement (QI) was applied to a county health department (CHD) with particularly low 4/3/1/3/3/1 series immunization rates (75%) for children, 19 to 35 months old. The Duval CHD launched a RWJF supported QI evaluation to demonstrate the benefits and lessons learned from implementing QI processes in June, 2009. Our evaluation employed a Mixed Method Participatory Design involving: 1) quantitative data on immunizations and factors influencing immunizations, and 2) qualitative data including observations of staff engaged in QI activities and interviews of staff. A pilot clinic, the lowest performing CHD clinic (74% rate) at the start of the project, exceeded over-all CHD performance within four months and exceeded the target (90%) in less than 18 months. The overall CHD rate exceeded 85% within 18 months and is on target to exceed 90% by 2 year anniversary. Lessons learned and insights will be discussed.

Evaluation of a QI Initiative in an Urban Health Department: Challenges and Lessons Learned Courtenay Kessler, University of Wisconsin, Madison, courtenay.kessler@aurora.org Geoffrey Swain, City of Milwaukee Health Department, gswain@milwaukee.gov In collaboration with the Robert Wood Johnson Foundation, the City of Milwaukee Health Department and the Center for Urban Population Health have partnered to implement and evaluate a quality improvement initiative focused on improving the outcomes and effectiveness of school-based immunization clinics. The project focused on standardizing procedures through process mapping and developing common data collection tools, and implementing and evaluating several interventions aimed at increasing parental consent rates through rapid Plan-Do-Study-Act (PDSA) cycles. Evaluation embedded within the PDSA cycles ("Study") examined processes and outcomes related to the specific interventions. More broadly, the evaluation team evaluated changes in staff attitudes and beliefs about QI, as well as challenges and lessons learned related to implementing and integrating a QI approach within a local health department. The presentation will review the roles of both embedded, rapid evaluation ("Study") and the broader (systemic) evaluation of this QI initiative.

Session Title: Tests of Two Frameworks for Evaluating the Impact of Health Research

Multipaper Session 882 to be held in Malibu on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Research, Technology, and Development Evaluation TIG

Chair(s):

Mary Beth Hughes,  Science and Technology Policy Institute, m.hughes@gmail.com

Evaluating the Impacts of Health Research: Revisiting the Canadian Institutes of Health Research Impact Assessment Framework Presenter(s): Nicola Lauzon, Canadian Institutes of Health Research, nicola.lauzon@cihr-irsc.gc.ca Marc Turcotte, Canadian Institutes of Health Research, marc.turcotte@cihr-irsc.gc.ca Laura McAuley, Canadian Institutes of Health Research, laura.mcauley@cihr-irsc.gc.ca Abstract: The Canadian Institutes of Health Research (CIHR) has a mandate to excel in the creation and translation of knowledge into improved health and a strengthened health care system. To ensure CIHR remains accountable to the public, the measurement of progress towards this ambitious mandate is necessary. Research evaluations are challenging because research outcomes are not directly attributable to research funding, often occur after many years, and are difficult to measure. CIHR's Impact Assessment Framework aims to bridge this divide by considering different spheres of influence and by using a contribution, rather than attribution, approach. As a learning organization, CIHR continually seeks to improve. This paper describes challenges encountered while applying the Framework to the evaluation of health research impacts in SARS, Commercialization, Obesity, and Cardiovascular disease as well as assessments prepared by each of our 13 Institutes. Lessons learned and refinements to the framework will be discussed.

Evaluating the Commercialization of Technologies Using the Canadian Institutes of Health Research Impact Assessment Framework Presenter(s): Marc Turcotte, Canadian Institutes of Health Research, marc.turcotte@cihr-irsc.gc.ca Laura McAuley, Canadian Institutes of Health Research, laura.mcauley@cihr-irsc.gc.ca Nicola Lauzon, Canadian Institutes of Health Research, nicola.lauzon@cihr-irsc.gc.ca Abstract: The Canadian Institutes of Health Research (CIHR) is Canada's federal health research funding agency. CIHR's ambitious mandate goes beyond the funding of research excellence to include the translation of research knowledge into health and health system benefits for Canadians. Through CIHR's Impact Assessment Framework, we are investigating the contribution of health research to broader impacts in the areas of health and economics. This evaluation explores the translation of research through commercialization of technologies. Two important steps in the commercialization process is patenting of innovative technologies and products, and the creation of spin-off companies. Using data from the United States Patent and Trademark Office together with administrative and interview data we evaluate CIHR's contributions to the process of moving research from an academic setting to the marketplace. This evaluative study will improve our understanding of facilitators, barriers and drivers of commercialization from a health research funder perspective.

Putting a Value on Biomedical Research Center Programs: Adapting the Research Payback Framework for Application in the United States Presenter(s): Jack E Scott, The Madrillon Group Incorporated, jscott@madrillongroup.com Margaret Blasinsky, The Madrillon Group Incorporated, mblasinsky@madrillongroup.com Mary C Dufour, The Madrillon Group Inc, mdufour@madrillongroup.com Rachel Mandal, National Institutes of Health, mandalr@mail.nih.gov Stephan Philogene, National Institutes of Health, philogenes@mail.nih.gov Abstract: Research center programs are an important tool used by the National Institutes of Health (NIH) to stimulate the growth of research investigation and infrastructure in emerging fields of scientific inquiry. Many evaluations of center programs focus on program outputs (publications, trainees trained, etc.) while neglecting outcomes. Few conceptual models exist to guide outcome assessment for these programs. One exception is the Research Payback Framework (RPF), an outcome-oriented case-study-based approach applied successfully in evaluations of biomedical and health research programs in Europe, Canada, and Australia. Since the RPF incorporates economic and non-economic outcomes, it can encompass the multiple objectives of NIH research centers. To the best of our knowledge, our study is the first application of the RPF framework with a US research center program. We describe its application with an NIH-funded research center program, including the conceptual and methodological enhancements we made and lessons learned to inform future applications.

Session Title: Values in Action

Demonstration Session 883 to be held in Manhattan on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Indigenous Peoples in Evaluation

Presenter(s):

Joan LaFrance, Mekinak Consulting, lafrancejl@gmailcom

Richard Nichols, Colyer Nichols Inc, colyrnickl@cybermesa.com

Karen E Kirkhart, Syracuse University, kirkhart@syr.edu

Abstract: This session will provide an experience of "values in action" by engaging those attending in activities that contribute to entering an Indigenous world and building community. Come prepared to experience ceremony protocol, feasting, gifting, and sharing your lineage. These are actions that link to relationship, family and community -- critical values that permeate throughout Indigenous communities. Through experiencing the expression of these values in mock activities, participants will have an opportunity to learn and discuss how values mediate entry into a community and influence evaluator credibility. After the "values in action" segment, the facilitators will discuss ways in which Indigenous values inform program quality and the evaluation process. Examples will be provided regarding how the process of identifying shared or common values among Indigenous evaluation stakeholders contributes a sense of ownership of the evaluation process.

Session Title: Lessons Learned From Conducting Evaluations

Multipaper Session 884 to be held in Monterey on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Graduate Student and New Evaluator TIG

Chair(s):

Jason Burkhardt,  Western MIchigan University, jason.t.burkhardt@wmich.edu

Adding Value to Writing Center Evaluations through Evaluation Theory Presenter(s): Maran Subramain, Western Michigan University, maran.subramain@wmich.edu Carl Westine, Western Michigan University, carl.d.westine@wmich.edu Abstract: Writing centers are viewed as an important medium which can facilitate better writing skills. As writing centers evolved, questions arose regarding the effects of writing tutoring on student learning and writing performance. Today, writing centers continue to grapple with the challenges of conducting meaningful program evaluation given limited resources (Bell, 2000). On a large scale, writing center evaluation seems to be untapped by practicing evaluators. The main goal of this presentation is to discuss the supports that evaluation theory and practice can offer to the evaluation of writing centers. In particular, based on their own experiences, the presenters will draw upon theory to emphasize using logic models under a theory-driven inspired evaluation framework to build writing center evaluation capacity through conceptual and process use. Reference: Bell, J. H. (2000). When hard questions are asked: evaluating writing centers. The Writing Center Journal, 21, 7-28.

Program Evaluation of an International Street Fair: Lessons Learned by Graduate Student Evaluators Presenter(s): Thomas Long, Virginia Tech, tomlong@vt.edu Janice Austin, Virginia Tech, jema@vt.edu Penny Burge, Virginia Tech, burge@vt.edu Abstract: The International Street Fair at Virginia Tech, in its 52nd consecutive year, brings together students and community members in a world marketplace atmosphere to learn about cultures in different countries through the sampling of exotic dishes, native performers and craft booths. The purpose of this study is to describe a program evaluation of the International Street Fair conducted by two doctoral students in an evaluation course. The program evaluation was conducted using participant surveys to measure learning outcomes. The results provide a description of how the International Street Fair contributes to meeting the learning objectives of the university's international student center. An emphasis is placed on the experience of and lessons learned in conducting one's first program evaluation. Future considerations for graduate students and new evaluators are discussed.

Practical Considerations for Implementing Longitudinal School-based Evaluations Presenter(s): Janet Lee, University of California, Los Angeles, jslee905@yahoo.com Nicole Gerardi, University of California, Los Angeles, gerardi_nicole@yahoo.com Abstract: Increasingly policy makers and program developers recognize that real social change (whether it be to increase academic achievement, change health habits, etc.) often takes multiple years of program participation and implementation to achieve. Correspondingly, making judgments of these programs often require evaluators to also incorporate a multi-year approach to the evaluation design. In this presentation, the authors will discuss case examples of school-based evaluations that utilize a longitudinal/multi-year perspective. Specifically, we will discuss the practical considerations and decisions that were made and examine various successes, challenges, unintended consequences, and lessons learned from implementing longitudinal evaluations. The authors also address the extent to which various stakeholder group values enter the evaluation process and influence decisions. Evaluation decisions made regarding multi-year program evaluations directly impacts the summative judgments made based on available findings. Lessons learned from conducting multi-year school based evaluations can offer important insights for the planning of future evaluations.

Session Title: If Stakeholders Matter, Which Stakeholders do We Listen to First?

Panel Session 885 to be held in Oceanside on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the AEA Conference Committee

Chair(s):

Sanjeev Sridharan, University of Toronto, sridharans@smh.ca

Abstract: While working with different stakeholders across evaluations, there are consistent questions about evaluation use driving the discussion regarding the value of the product and to the stakeholders: - To whom should evaluations be useful? - How do we get evaluations that meet the needs of all audiences? - Is this evaluation to be useful for learning, accountability or both? While responses differ, evaluators worldwide are united in their exploration of these challenges. In Canada, within the context of a renewed federal evaluation policy, preliminary research with representatives from a cross-section of departments suggests a disconnect between evaluation criteria and stakeholder beliefs. In this panel, we will explore the legitimacy of different stakeholder viewpoints in making judgments of quality, merit or worth of an initiative. A combination of realistic evaluation and contribution analysis offer stakeholders an option by which to explore "what works for whom in what context under what conditions?"

A Canadian Federal Government Evaluator Perspective Mary Kay Lamarche, Natural Resources Canada, mklamarche@rogers.com As a member of the panel, Mary Kay Lamarche provides the perspective of evaluation from a Canadian federal government perspective. Mary Kay Lamarche is an Evaluation Manager with Natural Resources Canada (NRCan). Having first learned evaluation on the job with Performance Management Network (PMN), she has now been involved with evaluation, performance measurement, strategic/results-based planning in both the public and private sectors for nearly ten years (seven with the public sector). In October 2008, Mary Kay was part of the first cohort graduating from the Graduate Certificate in Program Evaluation at the University of Ottawa, where her research often focused on realist synthesis. Mary Kay is a member of the Canadian Evaluation Society-National Capital Chapter (CES-NCC) Board and serves as this Chapter's representative on the CES National Council. She is also a member of the American Evaluation Association.

A Pacific Coast, Independent Consultant Perspective Lisa O'Reilly, Independent Consultant, enquiries@lisaoreilly.ca As a member of the panel, Lisa O'Reilly provides the perspective of evaluation from a west coast, independent consultant perspective. Lisa O'Reilly has worked in public sector policy analysis, strategic planning and evaluation for ten years. Lisa's clients include federal government departments, not-profit organizations and international donor organizations - all with their own language and context. Lisa completed her Masters in Public Administration this year with research on evaluation practices of a minority language group in a peace precarious area. Lisa O'Reilly has been a member of the CES BC and Yukon Chapter executive for four years. Currently, she is the BC and Yukon representative to CES National Council. Lisa is also a member of the European Evaluation Society and the American Evaluation Association.

An Academia Perspective Sanjeev Sridharan, University of Toronto, sridharans@smh.ca Panel member Sanjeev Sridharan provides international and academic perspectives on evaluation. He is Director of St Michael's Hospital, Research Scientist at the Centre for Research on Inner City Health Evaluation Centre (Li Ka Shing Knowledge Institute) and Associate Professor (Management and Evaluation) at the Department of Health Policy, University of Toronto. Previously, Sanjeev was the Head of the Evaluation Programme and Senior Research Fellow at the Medical School of the University of Edinburgh. Sanjeev has a strongly interdisciplinary background with a Doctorate in Social Ecology from the University of California at Irvine, Master's in Public Policy from Purdue University, and a Bachelor of Technology degree in Civil Engineering from the Indian Institute of Technology. His evaluation projects include health improvement, delinquency prevention initiatives, substance abuse treatment programmes, and justice systems reform interventions. He is on the Board of the Canadian Journal of Program Evaluation and the Evaluation and Program Planning.

An Internationally Educated Independent Consultant Perspective Jane Whynot, Whynot & Associates, jwhynot@magma.ca As a member of the panel, Jane Whynot provides the perspective of an internationally educated, independent evaluation consultant perspective, based in central Canada, Miss Whynot is a consultant specializing in performance with a background spanning over a decade in independent management for both the public and private sectors. Miss Whynot was an inaugural class member of the graduate diploma in Policy and Program Evaluation at Carleton University. She has continued her evaluation studies at the University of Melbourne in Australia in their Masters of Assessment and Evaluation program with an expected convocation date of December 2010. Miss Whynot is one of the founding members of the Ottawa area Lunch and Learn for Evaluators in addition to a core member of the Mentoring Working Group. She is also an active volunteer member of the Annual Learning Event Planning Committee of the National Capital Chapter of the CES.

Session Title: Framing Public Value, Building Identity, and Enhancing Learning Experiences: A Sampler of Visitor Studies in Zoos, Aquariums, and Natural History Museums

Panel Session 886 to be held in Palisades on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Evaluating the Arts and Culture TIG

Chair(s):

Kathleen Tinworth, Denver Museum of Nature and Science, kathleen.tinworth@dmns.org

Abstract: How can public value and learning be measured in informal settings like a natural history museum, zoo, or aquarium? What role does value play in these environments? What kind of technology is employed to answer value and experience questions? And who does this work? For a third year, members of the Visitor Studies Association (VSA), an international network of professionals committed to understanding and enhancing visitor experiences in informal settings through research, evaluation and dialogue, will present a showcase of studies in informal environments. This year, we focus on frameworks and technology used to measure public value of zoos and natural history museums as well as the ways in which interacting with these settings develops individual's identity as environmental stewards.

Assessing Learning in Zoological Settings Diane Sweeney, Vision Education, dsweeney9@pacbell.net Education is central to the mission of most zoos and aquariums. Participants who interacted with dolphins at three U.S. zoological facilities gained knowledge and appeared to develop stronger identities as environmentally-caring and responsible individuals who take stewardship action. Visitors who directly interacted with dolphins in the water, spectators who watched the interactions from nearby vantage points, and the dolphin trainers who led the interactions were interviewed (n=31), and past-visitors completed an online questionnaire (n=933), as part of a doctoral research investigation of learning in these settings. Within a sociocultural framework and a Community of Practice model of learning, data were coded, categorized, and analyzed based on frameworks for informal science education from the National Science Foundation and the National Research Council. The presentation will discuss the findings, the potential mediators of learning in the physical, social, and personal realms, and the representations and cultural expectations in such experiences.

Examining the Use of Technology by Visitors in Informal Science Settings Victor Yocco, Institute for Learning Innovation, yocco@ilinet.org Many informal science education settings such as zoos, parks, and nature centers have invested in programming that centers on visitors' use of technology. For example: touch screen computer kiosks, Smartphone applications, and cell phone guided audio tours. In this presentation Dr. Victor S. Yocco, Research Associate at the Institute for Learning Innovation, will provide an overview and brief summary of findings from two projects implemented in zoos, which utilized technology in an attempt to enhance visitors' experiences and learning outcomes. Multiple evaluations occurred on the various components comprising each of these projects. Examining the findings of these studies led to the creation of a conceptual model for how visitors engage with technology in zoos. The presentation will conclude with a description of the components making up the Visitors' Technology Use in Zoos model.

Visitor's Values of Zoos in Ohio: A Comparative Collaboration Joe Heimlich, Ohio State University, heimlich@ilinet.org As funding systems and public support for institutions wane, museums of all types are critically considering the hows and whys of public value of the institution. Using a framework developed by Carol Scott, Yocco created a measure of three domains of public value for art museums (Yocco, Heimlich, Meyers, & Edwards, 2008). This instrument was adapted for zoos and used at six zoos in Ohio in an attempt to discern if public value toward zoos varied across sites. The participating zoos collaborated in the research design and did all data collection and entry. The evaluators did statistical analysis, and the collaboration of zoo education directors and evaluators collectively made meaning of the values study. The process for moving toward collectively understanding of differences of values created a rich forum for dialogue about similarities and differences and the implications between and among the institutions.

Measuring Public Value at the Smithsonian's National Museum of Natural History Bill Watson, Smithsonian's National Museum of Natural History, watsonb@si.edu At the 2010 AEA meeting, the Smithsonian's National Museum of Natural History (NMNH) presented the process through which senior staff and a team of consultants developed a framework, metrics, and protocols for evaluating the public value of the museum's products, services, and programs. The framework is designed to collect and aggregate data about the museum's success for four key, measurable attributes of public value: Trust, Scale, Engagement, and Influence. We will present the highlights of the museum's first full-scale pilot test and implementation of the framework, which was conducted in the context of outreach related to the Human Origins Program. The outreach includes a 15,000 square foot exhibition, a multimedia website, and a slate of onsite public programs. The pilot test included data collection from over 4,500 visitors to the museum and its websites. The aggregation of seemingly disparate data points into clear statements of impact will be emphasized.

Collaborative Stakeholder Evaluation: Enhancing Exhibit Evaluation Through Greater Stakeholder Involvement Jennifer Borland, Rockman Et Al, jennifer@rockman.com In 2010, Rockman Et Al worked with the American Museum of Natural History to conduct a Collaborative Stakeholder Evaluation of its Traveling the Silk Road Exhibition. Overall the collaborative evaluation experience seemed to be both positive and productive. Staff and volunteers gained new skills and made new cross-departmental connections within the museum. Furthermore, the evaluation team, aided by the museum stakeholders, was successfully able to establish, implement and refine a process for collaboratively evaluating a major exhibition, using innovative technology products to coordinate the evaluation effort remotely. The end result this effort was a rich, multifaceted set of data and findings that help all stakeholders better understand the impact and outcomes of the exhibition and the skills and experiences gained during this collaborative evaluation process will likely have a lasting impact on everyone who participated.

Session Title: Weaving Networks, Weaving Change: Practical Uses and Experiences of Inter-Organizational Network Analysis Findings

Panel Session 887 to be held in Palos Verdes A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Social Network Analysis TIG

Chair(s):

Raul Martinez, Harder+Company Community Research, rmartinez@harderco.com

Discussant(s):

Paul Harder, Harder+Company Community Research, pharder@harderco.com

Abstract: Increasingly, evaluations of funders supporting local service providers focus on measuring improvements to the larger systems of care. Based on panelist's experiences conducting evaluations of First 5 funded programs in several counties in California, this panel will discuss the implications of quantifying the measurement of systems of care from the perspectives of evaluators and stakeholders. Evaluators will present findings from a number of different First 5 County evaluations all of which used the same method to assess inter-agency collaboration and mapping the networks. Stakeholders from these First 5 Counties discuss their experiences and raise issues regarding the appropriate uses, implementation and meaning from network findings, including how network analysis can assess change over time and how to best use findings for different stakeholders and evaluation goals. The discussant will summarize the main themes from each paper to describe how inter-organizational network analysis can improve evaluation capacity.

Change Over Time in Networks of Care Providers in Monterey County and Interpretations of Systems Change Raul Martinez, Harder+Company Community Research, rmartinez@harderco.com David Dobrowski, First 5 Monterey County, david@first5monterey.org In an evaluation of First 5 services to families and children in Monterey County, an inter-organizational network mapping procedure was employed at two time periods to identify changes in levels of coordination and collaboration among service providers who receive funds from First 5 Monterey County. Across a one year period, there was an increase in the number of interactions between funded partner agencies, with the increase in ties occurring at the Networking and Coordination levels, but with slightly fewer ties at the Collaboration level. The results raise questions about how best to engage participating organizations and other stakeholders in increasing levels of collaboration, when is the right time to measure changes in systems, and whether an increase in collaboration is a necessary or sufficient condition to determine improvement in systems change outcomes.

Measuring Inter-Agency Collaboration and Coordination in First 5 Fresno County: Experiences of Evaluators and Stakeholders in Understanding Systems Change Over Time Gary Resnick, Harder+Company Community Research, gresnick@harderco.com Olivia Arnold, First 5 Fresno County, oarnold@first5fresno.org Improvement in the system of care in Fresno County is expected to occur as a result of the funding provided by First 5 Fresno under CA Proposition 10 for services for families and their children from birth through five years of age. To assess systems improvement, a web-based survey for staff and administrators from all funded agencies was conducted over two years. Front line and administrator staff (at least two) from each agency receiving First 5 funds completed Frey's Levels of Collaboration Scale. Year one results showed that service providers are highly connected to each other, but mainly at the level of networking. Some agencies have long-standing relationships with other agencies at the higher levels of coordination and collaboration but other agencies are relatively isolated. Year two results and change over time will also be presented as implications for interpreting these results to assist programs in enhancing collaborations are discussed.

Social Network Analysis as a Tool for Engaging Communities in the Assessment of Inter-Organizational Collaboration Sae Lee, Harder+Company Community Research, slee@harderco.com Inter-organizational network analysis is being used in several evaluations of First 5 county programs in California to measure changes in systems of care towards greater collaboration among agencies receiving First 5 funds. Another use of the methodology is presented in this paper, involving an evaluation of a participatory community assessment process as part of the comprehensive placed-based initiative developed by First 5 Los Angeles. To assess the current state of collaboration among community agencies within each of First 5 LA's 14 communities, an inter-organizational survey was conducted which included Frey's Levels of Collaboration Scale. The network analysis findings will be discussed in terms of how community partnerships can use the findings to understand the strengths and weaknesses of their relationships with other agencies in the target communities, and how multiple stakeholders can be engaged in bolstering community-based networks as part of a larger community building process.

Session Title: Voices From the Field: A Dialogue on Incorporating Cultural Responsiveness Into Evaluation Training and Practice

Think Tank Session 888 to be held in Palos Verdes B on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Multiethnic Issues in Evaluation TIG

Presenter(s):

Krystal Tomlin, The Robert Wood Johnson Foundation, tomlinkrystal@gmail.com

Discussant(s):

Adina Wright, The Robert Wood Johnson Foundation, amwright777@gmail.com

Lauren Ramsey, The Robert Wood Johnson Foundation, laurenmeta@gmail.com

Jose Reyes, The Robert Wood Johnson Foundation, mr.josereyes@hotmail.com

Sudha Sivaram, National Institutes of Health, sudha.sivaram@nih.gov

Eric Wat, Special Services for Groups, ewat@ssgmain.org

Valerie Williams, University Corporation for Atmospheric Research, vwilliam@ucar.edu

Howard Walters, OMG Center for Collaborative Learning, howard@omg.org

Erica Lizano, University of Southern California, erica.lizano@gmail.com

Monica Getahun, OMG Center for Collaborative Learning, monica@omgcenter.org

Abstract: In response to the growth and cultural diversity of organizations and communities, evaluators have begun to embrace the practice of Culturally Responsive Evaluation (CRE) which focuses on underlying factors such as race and culture. While the practice is still in its infant stage, there are several champions that are leading the way including the Robert Wood Johnson Foundation (RWJF) who has developed several fellowships that nurture professionals by providing them with the necessary skills and experience in CRE. In this case, the RWJF evaluation fellows propose a discourse that focuses on the application of CRE in the "real world" by reflecting on their field placement experience and on the application of the Race Matters Toolkit, which assesses the cultural responsiveness of organizations. In addition to the fellow's experiences, several seasoned professionals will share their personal insight of incorporating the CRE values and approaches in their practice.

Session Title: Evaluating the 3 Rs: Remediation, Retention, and Roadblocks

Multipaper Session 890 to be held in Salinas on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Assessment in Higher Education TIG

Chair(s):

Katherine Beck,  Westwood College, kbeck@westwood.edu

An Evaluation of Two Developmental Assessments Use in Three Community Colleges Presenter(s): Xin Liang, University of Akron, liang@uakron.edu Abstract: Parsad, Lewis, & Greene (2003) found that 92 percent of two-year institutions use the resulting scores for placement into remedial education. Accuplacer and Compass are two tests that dominate the current assessment market for community colleges. However, there is a gap between specific scores and expected proficiency requirement needed for different level of developmental courses in preparing more students to successfully enroll in and complete certificate and degree programs in community colleges. The purpose of this study is to examine the characteristics of Accuplacer and Compass and how the two tests can best be used to inform placement and encourage learning. The data was based on the researcher's experiences with the ACCUPLACER and the COMPASS tests to assess academic proficiency in three community colleges in an evaluation project. The results indicated that the use of ACCUPLACER with 4 levels of proficiency: referral, 2-level below college, 1-level below college, and college level was more sensitive to capture student progress in learning developmental mathematics accurately than COMPASS.

College Preparation and the Need for Remedial Instruction: Evaluating the Foundational Coursework Program Presenter(s): Katherine Beck, Westwood College, kbeck@westwood.edu Abstract: In Colorado, one-third of the high school graduating class of 2009 entered a state university or college needing remedial help in at least one basic skills class: reading, writing or math. For Westwood College, headquartered in Denver, this is a critical statistic. For the average Westwood College student, there are already other difficult circumstances threatening to derail them from academic success. Therefore, we wanted to evaluate our institution's ability to first identify, and ultimately meet the needs of this large proportion of our population who are not prepared for college. This session focuses on how we conducted an evaluation of our Foundational Coursework Program, the challenges we faced, and our unexpected conclusions. Both the identification and the placement of these students will be discussed. Additionally, discussion will focus on the many difficulties campus staff and administration face when identifying and placing students that add complexity to this type of evaluation.

Evaluation Approach of and Lessons Learned from an Evaluation of a Community College Retention Program Presenter(s): Charyl Yarbrough, The Heldrich Center for Workforce Development, cyarbrou@ejb.rutgers.edu Bill Mabe, The Heldrich Center for Workforce Development, billmabe@ejb.rutgers.edu Abstract: Community colleges play a vital role in America's educational system, offering low-income and disadvantaged students the skills they need for economic success. Sadly, most community college students never graduate and half dropout in the first year. Many colleges are working to improve retention by offering support services designed to strengthen non-cognitive abilities and help students overcome barriers. Our paper presents findings and lessons learned from a two-year evaluation of a retention program for black males at an urban community college where the graduation rate for black males is 5%. The purpose of the evaluation was to assess outcomes and provide recommendations for program development. We analyzed outcome data using propensity scoring and used qualitative data collection approaches to learn about program processes. This paper is a resource for evaluators interested in engaging in a discussion of research methodologies that can be applied to outcome based studies of higher education initiatives.

Session Title: Performance Measurement for Policy Advocacy: Insights From Embedded MEL Practitioners

Panel Session 891 to be held in San Clemente on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Advocacy and Policy Change TIG

Chair(s):

Gabrielle Watson, Oxfam America, gwatson@oxfamamerica.org

Abstract: Policy advocacy work has gained increasing recognition by Boards and the philanthropic supporters of the non-profit world as an effective way to achieve systemic change benefiting large numbers of people. Along with this growing support comes growing demand to demonstrate results. This has spawned significant innovation and development within the field of policy advocacy evaluation and a specific interest in tools used by the private sector, such as performance measurement. This panel presents the experiences of two organizations that have been using performance measurement of their policy advocacy work. Presenters, each an internal MEL staffer, explore issues such as the tension between strategic learning and accountability and the search for meaningful and manageable indicators. Finally, presenters will explore how performance measurement systems can be used as a way to prompt teams to engage in meaningful conversations that result in collective learning and corrective actions.

Performance Measures for Advocacy Campaigns: The Search for Usefulness and Practicality Lisa Hilt, Oxfam America, lhilt@oxfamamerica.org Oxfam America embarked on an ambitious effort to create annual policy advocacy campaign plans and institute a quarterly performance measurement system in 2011. The performance measurement system seeks to identify a short set of meaningful and measurable indicators that are relevant across five issue areas, and that can drive strategic conversations and behavior. Lisa Hilt, Campaigns Coordinator at Oxfam America, describes the process of identifying and constructing indicators, designing the quarterly review process, and the lessons about indicators and process gained through this experience. Lisa explores the critical challenge of developing measures that are feasible within the context of a fast-paced and tightly-resourced campaign environment, describing the practical implications of various indicator selections.

Delivering on Demands for Learning and Accountability Kimberly Bowman, Oxfam GB, kbowman@oxfam.org.uk Robust monitoring and evaluation systems can be expected to deliver a lot of things, for many different stakeholders. Management, individuals or project teams might be looking for information to inform or asses their day-to-day work, to provide evidence of progress or to generate the next big strategic insight or program innovation. Meanwhile, donors and marketers have their own information needs. How do we balance the sometimes competing demands for learning and accountability? How do we prioritize which stakeholder gets which needs met, over others? Kimberly Bowman has worked with development organizations in Canada and England, and serves as a Learning & Accountability Advisor to Oxfam GB's UK Campaigns team. In this presentation, she will discuss some of the challenges she has faced in attempting to deliver on the at-times competing demands and expectations for learning and accountability.

From Theory of Change to Performance Measures: Two Paths, One Taken Gabrielle Watson, Oxfam America, gwatson@oxfamamerica.org Oxfam America embarked on an ambitious effort to create annual policy advocacy campaign plans and institute a quarterly performance measurement system in 2011. With three campaigns and five 'issue areas', there is a diversity of issues, policy environments, targets, approaches, and indeed, theories of change, underpinning the work. The aim of the performance measurement system is to identify a short set of meaningful and measurable indicators relevant across the five areas, and that will drive strategic conversations and strategic behavior. Gabrielle Watson, Senior Campaign Evaluation Advisor at Oxfam America, describes how one team navigated the tensions between demands for a common set of indicators and the desire for nuanced qualitative analysis of political context and emergent evidence of political access, policy relevance and influence.

Session Title: From Positive Youth Development to Full Potential: Rubrics That Shift Practice and Evaluation

Panel Session 892 to be held in San Simeon A on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Chair(s):

Kim Sabo Flores, Thrive Foundation for Youth, kim@thrivefoundation.org

Abstract: To move the aim for the positive youth development field, from youth competence to full potential, Thrive Foundation for Youth has made deep investments in the area of thriving. Thriving is an idealized, dynamic state between the person and his or her context. The notion of thriving has serious implications for policy and programming, as it targets practices that grow a young person's pursuit of full potential. Additionally, the focus necessitates new measures that move away from a stagnant snap shot of a young person's status at any given moment, toward methods that capture ongoing growth and strategy. This panel will share Thrive Foundation For Youth's innovative use of rubrics to influence the practice and measurement of thriving. Panelists will discuss the early lessons learned from 5 grantees over a six-month period. Examples will be drawn from youth development programs that include youth-led social change programs and volunteer mentoring programs.

On the Road to Thriving: Implications for Research, Practice, and Evaluation Carol Gray, Thrive Foundation For Youth, carol@thrivefoundation.org Carol Gray, Thrive Foundation's executive director and architect of its theory of change, will present how values and a body of research led to an assumption that: "If Adults Guides support youth to-identify their inner passions or "sparks"; understand and apply a growth mindset; reflect on thriving indicators and risk factors, visualize personal growth and develop goal management skills to build indicators of thriving-then, youth will be on a path to reaching their full potential." From that theory, she will share how Thrive's belief led to a landmark investment in rubrics, developed to serve two simultaneous purposes: #1: Grow explicit language and reflection at the exact point of program delivery. #2: Measure impact. Carol will then facilitate reflection on this rubric innovation with the panel, asking: "How have Thrive's beliefs about rubrics played out?"

Working at the Intersections: Rubrics That Measure and Drive Thriving in Youth Kim Sabo Flores, Thrive Foundation, kim@thrivefoundation.org There are many misconceptions about student self-assessments: The first is that youth will not accurately rate themselves and merely give themselves high marks. The second, is that even upon reflection, young people won't revise their work. However, Carol Dweck's (2006) seminal studies on mindset demonstrate that when young people develop a growth mindset, they are able to accurately self assess and regulate. Additional studies show, when youth are introduced to rubrics and taught how to use them as valuable self assessment tools; share their adult guide's understanding of quality (Sadler, 1989); and have the support needed to improve their work; they will accurately self-assess and effectively revise their strategies (Andrade 2008). In other words, rubrics are powerful, when implemented well. This presentation will focus on the lessons learned in using Thrive Foundation's rubrics in five different types of organizations and the accelerators and challenges in promoting thriving outcomes for youth.

Girls for a Change: Learning From the Field Carrie Ellett, Girls For A Change, ellettca@gmail.com Girls For A Change (GFC) is a national organization that empowers girls to create social change. GFC invites young women to design, lead, fund and implement social change projects that tackle issues girls face in their own neighborhoods. As one of the first organizations to integrate the Step-It-Up-2-Thrive Theory of Change and pilot the rubrics, GFC has had an opportunity to learn from the 10 mentors and 75 adolescents using the tools. During this presentation, a staff member, a youth, and an evaluator will discuss some of the potentials and challenges, as GFC integrates rubrics into their existing curricula, programs, training, and framework.

Integrating Thrive Rubrics With Other Youth Development Evaluation Strategies Melanie Moore, See Change Evaluation, melanie@seechangeevaluation.com The evaluation of youth development programs has become quite sophisticated, with common practices, such as program quality assessment, use of logic models, and use of research-based conceptual frameworks guiding applied research with community-based organizations. The research-based rubrics developed by the Thrive Foundation for Youth are a promising innovation in evaluation of youth development programs. As these rubrics are implemented in youth-serving organizations, it is important for evaluators to understand the ways in which the Thrive tools can complement and integrate with more typical approaches. This presentation will illustrate how the rubrics were used alongside a logic model-driven survey and interview study of a nonprofit youth development program, called Girls for a Change, and this case study will suggest lessons for the field of youth development evaluation.

Session Title: Human Services Initiatives in K-12 School Settings

Multipaper Session 893 to be held in San Simeon B on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Pre-K - 12 Educational Evaluation TIG and the Human Services Evaluation TIG

Chair(s):

Javan Ridge,  Colorado Springs School District 11, ridgejb@d11.org

Discussant(s):

Lisa Garbrecht,  EVALCORP Research & Consulting, lgarbrecht@evalcorp.com

The Relationship Between Discipline Infractions and Perceptions of Discipline Practices Presenter(s): Javan Ridge, Colorado Springs School District 11, ridgejb@d11.org Abstract: Two years of discipline and climate survey data from nine middle schools and five high schools were analyzed to determine if a relationship exists between incidents of disciplinary infractions and respondents' (student, parent and staff) perceptions of the disciplinary practices at their school. Disciplinary infractions were collected by type, and seriousness of infraction. Climate surveys provided information on perceptions of general safety, occurrences of bullying, perceptions of fairness in enforcing discipline rules, perceptions of consistency in rule enforcement, and perceptions of equity of punishment for infractions. Data were compared by school level (middle, and high school) to determine if a relationship exists between perceptions about practices and the level of infractions. Results have implications for training, policy, and supervision.

Structural, Professional, and Organizational Impacts on School Mental Health Integration Presenter(s): Oliver Tom Massey, University of South Florida, massey@usf.edu Donna Burton, University of South Florida, dburton@health.usf.edu Abstract: Research suggests that the schools are the defacto provider of mental health services for children (Burns, 1995). Unfortunately, the provision of mental health services in schools presents complex problems due both to the difficulty of implementing evidence based practices (EBPs) into workable programs in the field (Proctor, et. al., 2009) and because of the often unique circumstances in schools where two organizational silos exist, one which emphasizes academic/educational success, while the other contends with behavioral and mental health services. This presentation describes the results of a survey of student services staff (school psychologists, social workers, guidance counselors), school resource officers, teachers, and principals regarding readiness to adopt EBPs, roles and responsibilities for the provision of mental health services, and perceived barriers and facilitators to mental health service integration. Multilevel factor analysis is used to assess professional group perspectives on roles, responsibilities, and readiness to adopt EBPs.

Evaluation Interrupted: Retaining Long-term Relevance in Shifting Policy Environments Presenter(s): Sheila A Arens, Mid-continent Research for Education and Learning, sarens@mcrel.org Andrea Beesley, Mid-continent Research for Education and Learning, abeesley@mcrel.org Jane Barker, Mid-continent Research for Education and Learning, jbarker@mcrel.org Abstract: One of the advantages of long-term evaluations of systems is certainly the opportunity for evaluators to build relationships with stakeholders, to learn intricacies of interventions, to collect data regarding sustained outcomes, and to understand how a system's reporting structures intersect with evaluation use. However, longer-term efforts can be complicated to manage. In this session, evaluators use their experience evaluating an intervention that modified a school system's counseling program to describe opportunities and complications associated with long-term evaluations. Presenters will summarize the intervention history and background, and consider how state, district and high school context impacted program implementation over time. Following this overview, evaluators will examine their successes and failures in absorbing and responding to policy changes that significantly altered the original evaluation objectives.

Go, Slow, Whoa!: Evaluation of a Nutrition Education Program in a K-5 Setting Presenter(s): Mya Martin-Glenn, Aurora Public Schools, mlmartin-glenn@aps.k12.co.us Abstract: The Go, Slow, Whoa (GSW) nutritional education program was piloted in the spring of 2010 at one elementary school by the Nutrition Services Department in Aurora Public Schools as one way to address the childhood obesity problem. Cafeteria foods and monthly breakfast and lunch menus were labeled as Go, Slow or Whoa foods. Parents attended informational breakfast meetings while classroom nutrition education was provided to students through the USDA's Integrated Nutrition Education Program (INEP). An evaluation was conducted to measure the preliminary effectiveness of GSW on student behavior and nutritional knowledge. Data collected included pre/post student surveys, parent focus groups, and counts of students purchasing school lunch and the number of 'go' food selections offered at lunch. Preliminary results were encouraging, prompting a larger implementation in the 2010-2011 school year. Evaluation results from both the preliminary evaluation and challenges from the second year of larger implementation will be discussed.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: The Good, the Bad and the Unanticipated: Exploring the Consequences of Changing an Evaluation Plan Midstream

Roundtable Presentation 894 to be held in Santa Barbara on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Evaluation Use TIG

Presenter(s):

Kate Golden, University of Nebraska, kgolden@unmc.edu

Jill Kumke, University of Nebraska, jkumke@unmc.edu

Megan Borer, University of Nebraska, mborer@unmc.edu

Lisa St Clair, University of Nebraska, lstclair@unmc.edu

Abstract: When a program struggles with the quality and implementation of its model, evaluators weigh the risk and reward of modifying evaluation designs. The decision to change, even slightly, can present significant challenges but may also yield surprising benefits. We faced this when evaluating an early childhood coaching program for childcare centers. We responded by adding a qualitative component to a primarily objectives focused quantitative design. Reflective of an evaluation approach that incorporates continuous improvement principles, this change provided a forum for reviewing the coaching model and considering improvements. This simple, yet effective adjustment provided a surprising range of benefits and raised new challenges. Impacts are noted on funders, program staff and the evaluation team. This roundtable presentation will encourage discussion around the unexpected effects of modifying evaluation designs. Questions presented will weigh the roles of context, evaluator/client relationship and resources when considering whether to alter the evaluation design.

Roundtable Rotation II: Perspectives on Conflicts Between Policy Makers' and Evaluator's Values When Using Evaluation Research on Mentoring Programs

Roundtable Presentation 894 to be held in Santa Barbara on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Evaluation Use TIG

Presenter(s):

Laura Lunsford, University of Arizona, lglunsfo@email.arizona.edu

Abstract: This roundtable proposes to examine the conflict in values that arises when an intervention, championed by a highly visible policy maker, is at odds with evaluation research. A case study of an undergraduate mentoring program for low-income students will be presented to frame the discussion. Mentoring is a valued activity, but research suggests that mentoring is a voluntary relationship, that it does not benefit all students, and negative outcomes are possible. However, many programs require mentoring, assume mentoring will benefit all students, and have few or no controls for managing bad relationships. Staff may be unwilling to 'rock the boat' with program champions to make needed changes. What is the role and professional obligation of the evaluator when these value conflicts occur? The audience will share thoughts about value conflicts between evaluators and program administrators and suggest ways to resolve such conflicts.

Session Title: Surprise in Evaluation: Values and Valuing as Expressed in Political Ideology, Program Theory, Metrics, and Methodology

Think Tank Session 895 to be held in Santa Monica on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Theories of Evaluation TIG

Presenter(s):

Jonathan Morell, Fulcrum Corporation, jamorell@jamorell.com

Discussant(s):

Tarek Azzam, Claremont Graduate University, tarek.azzam@cgu.edu

Joanne Farley, University of Kentucky, joanne.farley@uky.edu

Abstract: How does political ideology affect program theories, methodologies, and metrics? Participants will be randomly assigned to groups, and asked to sketch an evaluation based on one of three positions. 1) Government has an obligation to alleviate social inequities and thereby promote the public good. 2) Government's role is to uphold civil order so people to pursue their own goals, with the consequences of their actions being their own personal responsibility. In general, less government is better. 3) The family is the primary unit of social cohesion, and there resides the locus of decisions about issues such as health and education. Government can be active or passive, as long as it supports the centrality of the family as the locus of moral authority and daily living. During report backs and we will compare how the evaluation designs differ with respect to program theory, metrics, and methodology.

Session Title: Strengthening Values for Child Welfare Through Participatory Evaluation: Service Commitment, Job Expectations and Goals

Multipaper Session 896 to be held in Sunset on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Human Services Evaluation TIG

Chair(s):

Christine Mathias, University of California Berkeley, cmathias@berkeley.edu

Discussant(s):

Barrett Johnson, University of California Berkeley, barrettj@berkeley.edu

Abstract: Developing and implementing first, a statewide graduate school MSW program and, then a core curriculum for newly hired staff in the values-laden field of public child welfare services involved multi-level, multi-organizational and cross-institutional arrangements and many diverse stakeholders. The State legislation and federal regulations about child welfare training require program evaluation but are non-specific. Participatory evaluation was the method of choice for this evaluation. At first each part of the project was evaluated separately, but integrating the core evaluation with the title IV-E graduate school evaluation has improved our ability to systematically examine the effects of training and education. This session describes three aspects of the evaluation in detail: why we do it; how we involve stakeholders in the statewide evaluation of a standardized core curriculum; and building a chain of evidence, using the example of the examination of IV-E graduates' retention factors.

Developing a Participatory Training Evaluation for Child Welfare

Leslie Zeitler, University of California Berkeley, lzeitler@berkeley.edu

As part of a coordinated strategic plan to revise and improve standardized core training, active participation by a variety of university and agency stakeholders is necessary. This paper will show how regional training academy staff, trainers, county staff developers, curriculum writers, subject matter experts, trainees, and evaluation consultants all contribute to the improvement of standardized in-service training. Among ways these stakeholders participate are training pilots, quality assurance efforts, analyses of test data, use of trainer forums, and brief trainee focus groups. The design and decision-making roles of the Macro-Evaluation Committee will be presented. Consistent with an overall evaluation framework, trainee demographic and test data is analyzed for possible test item bias, aggregate performance, and potential demographic differences. We will describe the challenging process of coordinating participation by stakeholders in the curriculum in the test development/revision process.

Valuing Workers' Values: Helping Social Workers Stay in Child Welfare Service

Sherrill Clark, University of California Berkeley, sjclark@berkeley.edu

Research on the retention of public child welfare workers suggests that a service commitment is influential in attracting persons to the field. Thus, supporting this value may help to facilitate their long term worker retention in the profession. A qualitative data analysis of responses to a recent IV-E MSW graduate survey confirmed that helping others or making a difference in others' lives is a salient aspect of the job for new public child welfare social workers. This theme was present across all six years the survey was administered. In order to understand how to better foster the relationship between having a commitment to service and remaining in the child welfare field, the results of this analysis helped inform the focus of future evaluation of the factors that retain professional child welfare social workers.

Child Welfare Workers' Values: A Longitudinal Assessment of Job Expectations and Career Goals

Susan Jacquet, University of California, Berkeley, sjcaquet@berkeley.edu

Sherrill Clark, University of California Berkeley, sjclark@berkeley.edu

To evaluate the effects of educational and training programs on Title IV-E child welfare workers' values and career goals relative to their retention in the field, we have created a framework of questions focusing on service commitment, job/career expectations and goals which we ask at multiple points in the workers' careers. Cohorts are surveyed when newly graduated/hired, at 3 years post-graduation, and 5-years post-graduation. Telephone interviews involve graduates in the evaluation development by soliciting new focus areas to examine. The sequence of surveys occurs annually; each cohort is followed for at least five years. Comparisons between IV-E and non-IV-E new hires indicate there are differences between IV-E graduates and others, which we will report in this session. The information gathered in this evaluation along with the workers' retention status informs the evaluation of education and training programs as well as the agencies that hire them.

Session Title: Evaluating Teacher Professional Development in STEM: Examining Teacher Learning and Perceptions

Multipaper Session 897 to be held in Ventura on Saturday, Nov 5, 9:50 AM to 11:20 AM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

John Gargani,  Gargani + Company Inc, john@gcoinc.com

Discussant(s):

John Gargani,  Gargani + Company Inc, john@gcoinc.com

Knowledge, Pedagogy, Practice or Student Achievement: Evaluating a Middle School Math MEd Professional Development Program Presenter(s): David Ensminger, Loyola University, Chicago, densmin@luc.edu Jim Breunlin, Loyola University, Chicago, rbreunl@luc.edu Lindsay Viellieu, Loyola University Chicago, lviellieu@luc.edu Abstract: This paper discusses the outcomes evaluation of a University's Middle School Math M.Ed. degree program. The program was initiated through a State-funded grant, in collaboration with a local urban school district and the University. The paper describes how the evaluation team and University's stakeholders stressed examining the more proximal outcomes, (i.e. knowledge gains, changes in pedagogical perceptions and implementation of new skills in the classroom) by participants, while the State-level stakeholders emphasized (and required) that student achievement measures were most important. In addition, the paper describes the experiences of the evaluation team and the changes made to the evaluation plan as a result of lower than expected program participation: changes in participants' role in their school, reduced funding, the withdrawal of the school district's participation in the grant and the change in the program director of the grant.

Evaluating Impact of Teaching Workshops in Science, Technology, Engineering, and Math (STEM) Using Embedded Electronic Assessment Presenter(s): Gina Weisblat, Cleveland State University, boo500@aol.com Micah Arafah, Cleveland State University, m.arafah@csuohio.edu Abstract: It has been well-documented that the general public knows little about engineering, and the K-12 teacher population is no exception. Educators in the K-12 education system are increasingly expected to insert engineering into their teaching, as national and governmental pressure to include or infuse engineering into standard curricula continues to rise. This evaluation uses Bransford's learning theory as a backdrop to understanding how teachers learn and translate this knowledge. We will compare data of program evaluations from two years using a new mechanism for improving learning, and study the impact of the embedded electronic assessment tools on the teachers' learning and perception.

Side by Side: Two Math and Science Partnership (MSP) Teacher Institutes and How They Grew Presenter(s): Paul Lamphear, Hanssen Consulting LLC, paull@hanssenconsulting.com Abstract: The Math and Science Partnership (MSP) program responds to a growing national concern -- the lackluster performance of U.S. children in mathematics and science, and significant gaps between different demographics. Schools have responded with attempts to increase teacher competence. Measuring effectiveness of K-12 professional development has been difficult and controversial. This paper will compare and contrast two longitudinal studies of DOE Title IIB funded Teacher Institute programs that provide post graduate University credits to participating elementary and middle school teachers in the Milwaukee Public School District. We will explore how values in education, state testing for school performance, and differences between math education and science education influence both the initial program design and implementation, and the evaluation design. We will examine how those differences affected how we triangulated information from program and classroom observations, teacher attitude and knowledge changes, student performance changes, and ultimately judged the merit of the programs.

Return to Evaluation 2011