Return to search form  

Session Title: Applications of Evaluation Approaches to Natural Systems Management

Multipaper Session 386 to be held in Capitol Ballroom Section 1 on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Environmental Program Evaluation TIG

Chair(s):

Annelise Carleton-Hug,  Trillium Associates,  annelise@trilliumassociates.com

Begin at the Beginning: Evaluation’s Role in Bridging Knowledge Production and Knowledge Use…or How Do We Determine Best Available Science in Watershed Management Programs? Presenter(s): Mary McEathron,  University of Minnesota,  mceat001@umn.edu Abstract: Because of the daunting scientific complexity of river systems, river managers and stakeholders have come to rely on reviews by independent scientists to assure that the best available science is being used to inform decision making. Independent science review (ISR), a specific type of evaluative activity occurring at the beginning of projects, is a much needed counterpoint to the insistent call for outcome-based evaluations at the end of projects. However, the increase in number of ISRs underscores the importance of carefully considering when and why an ISR should be conducted. This paper presents the results of a retrospective study of three ISRs conducted by the Sustainable Ecosystems Institute (SEI) on the Columbia and Missouri Rivers. The discussion will focus on the importance of examining the research/scientific foundation of programs, the challenges of bridging the worlds of academic research and applied management, and the issue of ISR or evaluation use.

Evaluation and National Park Service Education Programs: a Responsive Utilization Approach Presenter(s): Ana Houseal,  University of Illinois Urbana-Champaign,  houseal2@uiuc.edu Ellen Petrick,  National Park Service,  ellen_petrick@nps.gov Bob Fuhrmann,  National Park Service,  bob_fuhrmann@nps.gov Abstract: Large federally funded agencies such as the National Park Service (NPS) are seeing a need for accountability in education programs. Conducted at the crossroads of the call for and implementation of a new educational evaluation policy outlined in the NPS Interpretation and Education Renaissance Plan, this reflective case study describes positive impacts of a responsive-utilitarian evaluation for stakeholders of a residential environmental education program in Yellowstone National Park. Important components identified as effective in this evaluation include involvement of stakeholders, data collection methods, and use of results by stakeholders within the park management system. Implications for use of this type of evaluation methodology within the broader context of the national Park System and other environmental educational program evaluations are discussed.

National Heritage Areas: Building a Cyclic Program Theory Model from Qualitative Data and Practitioner Knowledge Presenter(s): Jennifer Jewiss,  University of Vermont,  jennifer.jewiss@uvm.edu Daniel Laven,  National Park Service Conservation Study Institute,  daniel_laven@nps.gov Nora Mitchell,  National Park Service Conservation Study Institute,  nora_mitchell@nps.gov Abstract: National Heritage Areas (NHAs) represent a recent model for protected areas in the U.S. NHAs operate through partnerships with local communities, government agencies, and other organizations. Since 1984, 37 NHAs have been established by Congress, making this one of the fastest growing programs affiliated with the National Park Service and raising important national policy questions. Qualitative research conducted at three NHAs identified several common themes despite wide variations in their geographic, social, economic, and political contexts. This paper reflects on a project that engaged practitioners and other stakeholders in building a collective understanding of NHAs. The group developed a common program theory model, which represents the cyclic and long-term nature of heritage stewardship and is designed to serve as the foundation for an evaluation strategy applicable to all NHAs. The presentation will consider the approaches used, the challenges encountered, and the high stakes policy and management issues entailed.

Session Title: International and Cross-Cultural TIG Business Meeting and Presentation: Incorporating an International and Cross-Cultural Perspective in Setting Evaluation Priorities

Business Meeting Session 387 to be held in Capitol Ballroom Section 2 on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the International and Cross-cultural Evaluation TIG

TIG Leader(s):

Nino Saakashvili,  Horizonti Foundation,  ninosaak@yahoo.com

Tessie Catsambas,  EnCompass LLC,  tcatsambas@encompassworld.com

Gwen M Willems,  University of Minnesota Extension,  wille002@umn.edu

Abstract: Bringing an international perspective to evaluation has been an increasingly top priority of AEA. This highly interactive session will explore the perspectives of participants using a world café process and invite their questions, hopes, concerns, and ideas about further incorporating an international perspective in evaluation. The session will be using an appreciative process and become structured around two topics: (a) an international lens in evaluation topic selection; (b) increasing collaboration between nations and cultures in evaluation. Gathered around round tables, participants will be given 2-3 key questions to consider in groups. Through sharing and guided discussion, participants will explore what it means to be an “international evaluator” and what it means to be working in cross-cultural settings. Contributions from all participants will be encouraged through varied facilitation techniques (visioning, cards, forms, movement, group work, plenary discussion). This session will not only discuss but model strategies of working effectively in cross-cultural settings. The International and Cross-Cultural TIG Business Meeting will follow.

Session Title: Lessons Learned From Multi-Site Cost Studies: What to do for, and How to Cope With Resistance to, Cost-Inclusive Evaluation

Panel Session 388 to be held in Capitol Ballroom Section 3 on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Costs, Effectiveness, Benefits, and Economics TIG

Chair(s):

Brian Yates,  American University,  brian.yates@mac.com

Abstract: Evaluating the effectiveness of human services is demanding enough when conducted for one program. Evaluating effectiveness of services offered at multiple sites in a country is a formidable challenge. Evaluating costs as well as effectiveness of services at multiple sites can be exceedingly complex. Add cost-effectiveness and cost-benefit analysis, and evaluation can be near-impossible. For three different multi-sites, we describe lessons learned when including costs and monetary outcomes (benefit) in evaluation research. The first panelist details a successful framework for estimating costs of providing behavioral health and family strengthening services to educational and health care organizations. The second panelist elucidates seven steps that proved useful for a cost analysis over 8 implementations of a standardized program for preventing child neglect. The third presentation examines the resistance that was encountered, and (partially) countered, in a cost study of eight consumer-operated mental health services.

A Completed Cost Study for the National Multi-Site Evaluation of the Starting Early Starting Smart initiative (SESS) Elizabeth Harris,  Evaluation, Management and Training Associates Inc,  eharris@emt.org This paper describes a completed cost study for the national multi-site evaluation of the Starting Early Starting Smart initiative (SESS). SESS programs integrated behavioral health and family strengthening services into early childhood education or primary health care organizations. While these programs implemented a common approach to working with families of children in need, each was adapted to its unique environment with its own mix of specific services and delivery method (e.g., home-based or center-based services). The process recommended here provides a framework adaptable to the specific circumstances of each case. This cost documentation process provides a systematic and comparable estimation of the costs of providing services to participating families included in each of the programs during the core study period. We summarize the purposes and parameters of the framework, outline a step-by-step procedure for arriving at estimated costs, and provide reporting formats.

Lessons Learned from the Multi-Site Cost Analysis of the Family Connections Neglect Prevention Program Melissa Brodowski,  United States Department of Health and Human Services,  melissa.brodowski@acf.hhs.gov Jill Filene,  James Bell Associates,  filene@jbassoc.com Phaedra Corso,  University of Georgia,  pcorso@uga.edu Although there has been a growing emphasis on the importance of economic evaluation for the human services, there have been only a limited number of child maltreatment prevention programs that have incorporated cost analysis into their overall evaluation plans. The presentation will highlight the seven essential steps that were used to plan and implement a comprehensive cross-site cost analysis for a set of eight ongoing Federal grants that are replicating and rigorously evaluating Family Connections, a demonstrated effective neglect prevention program. Lessons learned and facilitating factors include the active engagement of site-level program implementers, use of standardized templates, prospective data collection, ongoing technical assistance throughout data collection process, and communication and dialogue about data inconsistencies. Finally, ways in which cost analysis and can be used to facilitate ongoing learning and bridging the knowledge gap between researchers and practitioners will be shared.

Lessons learned from a Multi-Site Evaluation of Costs, Benefits, Cost-Effectiveness, and Cost-Benefit of Consumer-Operated Services Brian Yates,  American University,  brian.yates@mac.com Unanticipated consequences of a multi-site evaluation of costs, benefits, cost-effectiveness, and cost-benefit of consumer-operated services (COS) versus traditional mental health services are described. Although the program announcement specified that costs of COS would be measured, asking COSs to report costs was translated into measurement not of the value of resources used, but only of monetary outcomes: benefits. Also, resistance to measuring costs prevented program expenditures from being measured with the specificity that could link use of resources to performance of program activities, with had been key to planned cost-effectiveness and cost-benefit analysis. Even evaluation of COS programs' use of non-monetary resources, such as volunteered time and donated space, was resisted. Research requirements and site characteristics interacted to cause one COS to operate under capacity, increasing cost per consumer so that findings for that program were likely unrepresentative.

Session Title: Advancing External Validity in Program Evaluation

Panel Session 389 to be held in Capitol Ballroom Section 4 on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Theories of Evaluation TIG

Chair(s):

Huey Chen,  Centers for Disease Control and Prevention,  hbc2@cdc.gov

Discussant(s):

Charles Reichardt,  University of Denver,  creichr@psy.du.edu

Abstract: Internal and external validity are vital to program evaluation. However, evaluation literature has focused attention mainly on internal validity. Few evaluators have formally addressed issues related to external validity in their evaluations. Program evaluation is an applied science. The neglect of external validity may curtail the usefulness of and limit the future development of program evaluation. The purpose of this panel is to advance external validity in program evaluation. Presenters of the panel will address conceptual and methodological issues of external validity. The presenters will: 1). Examine the limitations of the existing concepts and model of validity, 2.) Propose alternative conceptualization of external validity, 3). Propose alternative methods and models for achieving external validity, 4). Discuss future directions for dealing with internal and external validity.

External Validity, Policy and Practice Leonard Bickman,  Vanderbilt University,  leonard.bickman@vanderbilt.edu External validity lacks the more specific 'rules' found with other forms of validity. For example, internal validity has a list of threats and statistical validity has well defined power calculations. In contrast the determination of external validity is much more in the eye of the beholder. The policy and practice issues raised by this difference will be discussed by contrasting efficacy and effectiveness research.

What Works Where for Whom and Why? Stewart I Donaldson,  Claremont Graduate University,  stewart.donaldson@cgu.edu Recent debates about RCTs, gold standards, and credible evidence focus largely on internal validity concepts and how best to determine the average effects of a program, policy, or 'What Works' in general terms. However, policy and other decision makers desire credible and actionable evidence that generalizes or applies to present or future problems, programs, and policies of interest (i.e., external validity concepts). This paper will describe how the principles and procedures of program theory-driven evaluation science can be used to make evaluations more useful for decision making. A framework for increasing the external validity of primary evaluations and the syntheses of multiple evaluations will be presented and discussed.

External Validity: Alternative Models and Practice Melvin Mark,  Pennsylvania State University,  m5m@psu.edu External validity is a central concept in the Campbellian tradition. The term has been widely used, sometimes quite differently than Campbell intended. Even when external validity refers to the generalizability of findings, different models exist regarding how external validity is conceptualized and how it is thought to be achieved. Alternative models of external validity are selectively reviewed. I also identify different design and analysis procedures that have been suggested as ways of enhancing (or at least identifying the limits of) external validity. Fruitful directions for future work, both conceptually and in terms of evaluation practice, are described.

External Validity, Integrative Validity Model, and Evaluative Evidence Huey Chen,  Centers for Disease Control and Prevention,  hbc2@cdc.gov Validity has been one of the crucial issues in program evaluation. Recent controversies on the best evaluation methods and what should be considered as evidence in the evidence-based movements are related to validity. The controversies may reflect a need for re-examining the existing concepts and models of validity. Based upon the contribution by Campbell and Cronbach, this paper advances the concept of external validity and provides a comprehensive model of validity which allows evaluators to respond to the challenges more coherently. The paper also discusses sequential and concurrent evaluation approaches for evaluators to effectively deal with validity and how these strategies relate to program evaluation. Based upon the discussions, the article discusses possible directions for advancing issues related to validity in the future.

Session Title: The American Evaluation Association, Diversity Committee Asa G Hilliard Think Tank

Think Tank Session 390 to be held in Capitol Ballroom Section 5 on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the AEA Conference Committee

Chair(s):

Jenny L Jones,  Virginia Commonwealth University,  jljones2@vcu.edu

Presenter(s):

Itihari Toure,  First Afrikan Church,  fapc@firstafrikanchurch.org

Discussant(s):

Jenny L Jones,  Virginia Commonwealth University,  jljones2@vcu.edu

Cindy Crusto,  Yale University,  cindy.crusto@yale.edu

Leona M Johnson,  Hampton University,  leona.johnson@hamptonu.edu

Katherine Tibbetts,  Kamehameha Schools,  katibbet@ksbe.edu

Abstract: This session celebrates the life and legacy of Asa G. Hilliard, Ed.D., a.k.a. Nana Baffour Amankwatia II, a teacher, psychologist, and historian. Think tank goals are to: (1) introduce participants to Dr. Hilliard's work and teachings, (2) reflect on the values of Dr. Hilliard's work and how they can influence evaluation theory, methods, and practice and the professional development of evaluators, and (3) determine the best mechanism(s) to establish ongoing dialogue within the AEA regarding his work. Itihari Toure, a colleague and mentee of Dr. Hilliard's, will reflect upon his work and frame the session. Videos of Dr. Hilliard addressing topics relevant to evaluation will be presented. In small groups participants will reflect upon the teaching and determine the ways in which they would apply and teach the concept to others in the evaluation field. Co-sponsors: Indigenous Peoples in Evaluation, Multiethnic Issues in Evaluation, and Social Work Topical Interest Groups.

Session Title: How Can We Enhance University-Based Evaluation Programs to Expand Evaluation Literacy (Knowledge of Evaluation)?

Think Tank Session 391 to be held in Capitol Ballroom Section 6 on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Teaching of Evaluation TIG

Presenter(s):

Michelle Chandrasekhar,  Florida State University,  mchand@fsu.edu

Linda Schrader,  Florida State University,  schrader@coe.fsu.edu

Abstract: This session addresses questions raised in the 2006 AEA Think-Tank on university-based programs regarding concerns in the decline in the number of graduate evaluation programs. Preliminary results of a survey of current graduate evaluation programs will provide a framework for understanding how new evaluators are recruited and acquire knowledge in evaluation theory and practice. Participants will review these findings and discuss ideas for moving the profession forward. Topics for discussion will focus on recommendations for strengthening evaluation degree programs in terms of: 1. Recruiting diverse students 2. Engaging a diverse network of faculty and professionals representing various disciplines 3. Providing hands-on evaluation experiences for novice evaluators 4. Expanding the use of evaluation policy within organizations 5. Creating institutional capacity for evaluation literacy Think-tank participants will generate a list of recommendations. This list to be shared as a manuscript submitted to the American Journal of Evaluation under the 'Teaching Evaluation' section.

Session Title: Effectively Evaluating Community Impacts of a Statewide Tobacco Prevention and Control Program, Implemented amid Frequent Changes in Funding, Programming, and Staff

Panel Session 392 to be held in Capitol Ballroom Section 7 on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG

Chair(s):

Laura Feldman,  University of Wyoming,  lfeldman@uwyo.edu

Abstract: The Tobacco-Free Wyoming Communities (TFWC) program seeks to prevent and control the use of tobacco in Wyoming communities. While numerous studies show the correlation between state-level tobacco control programs and reduced tobacco use, no published studies use community-level data to show this correlation. In 2006, the Wyoming Department of Health asked the Wyoming Survey and Analysis Center (WYSAC) to evaluate the community-level impacts of the TFWC program. What was initially interpreted as a straightforward evaluation of program outcomes turned into a complex evaluation that ultimately guided both state- and community-level decision making. This panel summarizes the challenges inherent in evaluating long-term projects with long-term outcomes, but implemented amid frequent changes in staff, funding, programs, and targeted outcomes. The panel discusses how to overcome these challenges to measure the relationship between program expenditures and program outcomes. The panelists focus on lessons learned, especially in terms of techniques for evaluating long-term programs.

When Evaluation Design and Reality Collide Laura Feldman,  University of Wyoming,  lfeldman@uwyo.edu Laura Feldman will present on the development of the evaluation design and of the value of the logic model in guiding the original and final evaluation. She will focus on how WYSAC developed the logic model based on previous research and hands-on knowledge of the TFWC program (e.g., its expenditures, activities, and organization). She will also discuss how the evaluators adapted the evaluation design and the logic model as they learned that limitations in data collection would prohibit their ability to assess the model fully. Ms. Feldman holds an Ed. S. in evaluation and program development; she has more than 15 years of experience in program evaluation, program development, and program administration. She has worked on the evaluation of numerous educational and public health evaluations and is currently lead evaluator for Wyoming's Tobacco Prevention and Control project.

Using a Process Evaluation to Inform Program and Funding Decisions Tiffany Comer Cook,  University of Wyoming,  tcomer@uwyo.edu Shannon Williams will present on the process evaluation and how it was used to generate program recommendations. She will discuss the development and use of two data collection tools: a survey of local TFWC program staff and a focus group of experienced TFWC program staff. Ms. Williams will also discuss how the process data has been used to improve programming at both the state and local levels. Ms. Williams has a M.S. in Applied Statistics and Research Methods and is currently writing her dissertation, also in Applied Statistics and Research Methods. She lead the evaluation of the TFWC program.

Using Regression Analysis to Evaluate the Impact of TFWC Expenditures on Cigarette Sales Nanette Nelson,  University of Wyoming,  nnelso13@uwyo.edu Nanette Nelson will present on the regression analysis and its importance in evaluating the TFWC program. She will discuss the design of the regression model and explain how findings from the analysis were used to make recommendations for the allocation of future TFWC funds. She will also address how Wyoming's funding allocations relate to the Centers for Disease Control (CDC) recommendations and expected outcomes. Ms. Nelson has two M.S. degrees: one in Conservation Ecology and Sustainable Development and one in Agricultural and Applied Economics. In addition to conducting economic analyses as part of the evaluation of the Tobacco Prevention and Control project, Ms. Nelson works primarily evaluating environmental impacts.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Examining Parent Perspectives on Parent-Teacher Communication in Secondary Urban Schools

Roundtable Presentation 393 to be held in the Limestone Boardroom on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Social Work TIG and the Multiethnic Issues in Evaluation TIG

Presenter(s):

Deirdre J Sharkey Walker,  Texas Southern University,  dsharkey@houstonisd.org

Abstract: The purpose of the present study is to evaluate parent-teacher communication in an effort to improve parental involvement that target parents with children in urban schools. The results of the study indicated the most common communication between parent and teacher was when their child was performing poorly in the classroom. About forty two percent of the parents indicated they were receiving notes only when their child was performing poorly. On the other hand about forty eight percent of the parents reported they never received reports from the teacher on how well their child is doing in class.

Roundtable Rotation II: Intergenerational Parenting: Practice and Policy Implications

Roundtable Presentation 393 to be held in the Limestone Boardroom on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Social Work TIG and the Multiethnic Issues in Evaluation TIG

Presenter(s):

Maureen Rubin,  University of Texas San Antonio,  maureen.rubin@utsa.edu

Abstract: The past decade has seen a tremendous increase in the number of grandparent headed households in the US. According to the 1997 US Census, almost 4 million children were raised in grandparent headed households. There are various reasons that grandparents take on the responsibility of care giving. Although the challenges for grandparents with youth with disabilities are addressed, very limited literature is available on the strain of caregiving when severe emotional disturbances are present. This paper, based on a SAMHSA funded system of care (SOC) site in a southeastern state in the United States, takes a look at the importance of evaluation in documenting the needs of grandparents as caregivers and provides a discussion on the role of the mental health team in making sure that grandparents are seen as an integral part of the team.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Strategies for Conducting Evaluation with Hard-to-Find, Traumatized, and Vulnerable Populations

Roundtable Presentation 394 to be held in the Sandstone Boardroom on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

Adrienne Adams,  Michigan State University,  adamsadr@msu.edu

Megan Greeson,  Michigan State University,  greesonm@msu.edu

Rebecca Campbell,  Michigan State University,  rmc@msu.edu

Cris Sullivan,  Michigan State University,  sulliv22@msu.edu

Debra Patterson,  Michigan State University,  patte251@msu.edu

Abstract: Conducting evaluation/research that requires the participation of hard-to-find, traumatized, and vulnerable populations poses unique challenges and requires special considerations throughout the evaluation process. Issues of physical and emotional safety, confidentiality, and privacy are of particular concern and must be addressed in order to facilitate their participation. In this presentation we describe our work partnering with organizations that serve survivors of domestic and sexual violence. We take a participatory approach to tap the expertise of program staff who are best positioned to elucidate these issues. Through this collaboration we have developed a variety of strategies we employ during the design, recruitment, and data collection phases of the evaluation. Strategies include weaving evaluation into natural service delivery, innovative marketing and recruitment procedures, and adaptive data collection methods. The purpose of the round table is to generate broader discussion about conducting evaluation/research with a variety of hard-to-find, traumatized, and vulnerable populations.

Roundtable Rotation II: Jumping the Gate: Accessing Community Sources of Data in an Evaluation of Neighborhood Crime Prevention Councils when the Gatekeepers do Not Grant Entry

Roundtable Presentation 394 to be held in the Sandstone Boardroom on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

Kim Gilhuly,  Alameda County Public Health Department,  inertiate@yahoo.com

Claudia Albano,  City of Oakland,  calbano@oaklandnet.com

Liz Maker,  Alameda County Public Health Department,  liz.maker@acgov.org

Abstract: Neighborhood Crime Prevention Councils (NCPCs) are a community mobilization strategy used in Oakland, CA to engage residents in efforts to reduce crime and improve quality of life in their neighborhoods. An evaluation of the NCPCs relied on accessing the opinions of residents to discover if the Councils were reaching their goals of linking residents to each other and to city services, empowering residents to speak out, and making concrete changes. City staff who organize the NCPCs were gatekeepers to involved residents. While staff spoke for the residents, they did not necessarily represent the opinions of everyone (especially those critical of the NCPCs) nor were they cooperative with allowing the evaluator access to the community. Gatekeepers who do not “open the gates” can negatively impact an evaluation. This roundtable is a case study in how one evaluator “jumped the gate”, or was able to gain access and work directly with community despite unwilling gatekeepers, and ultimately achieve more accurate community representation.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Methodological Issues In Evaluation of Educational Programs Utilizing Student Level Data

Roundtable Presentation 395 to be held in the Marble Boardroom on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

John Dantzler,  University of Alabama Birmingham,  dantzler@uab.edu

Jason Fulmore,  University of Alabama Birmingham,  jfulmore@uab.edu

Abstract: The presentation will focus on unique methodological issues that surround the evaluation of educational programs which require the collection of student level data. Specifically, the presenters will discuss access to sites, confidentiality, data availability, and analytical strategies. Two evaluations will be presented along with the strategies employed and barriers encountered.

Roundtable Rotation II: Gleaning Assessment and Evaluation Data from In-Class Learning Activities

Roundtable Presentation 395 to be held in the Marble Boardroom on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Bryan Bradley,  Brigham Young University,  bryan_bradley@byu.edu

Abstract: The best and most informative assessments of learning are performance activities (in or out of the classroom) where students actively engage in tasks in which they apply their learning at the higher cognitive-behavior levels, such as “analysis”, “synthesis”, or “create.” The challenge for educators is in capturing valid and reliable data from learning activities to truly assess student learning, provide meaningful feedback, and inform institutional decisions. This round table will help participants find assessment data in the actual learning activities students engage in as they progress through a given course. Included in this session will be a concise review of current literature on this topic and an interactive discussion to help participants mine their own learning activities for meaningful assessment data.

Session Title: A Community of Practice Approach to Evaluation Capacity Building

Skill-Building Workshop 396 to be held in Centennial Section A on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

Presenter(s):

Beverly Parsons,  InSites,  beverlyaparsons@aol.com

Deborah Watson Trujillo,  Research Evaluation Associates for Latinos,  dr.trujillo@real-consulting.org

Patricia Jessup,  InSites,  pat@pjessup.com

Theresa Rosner Salazar,  Research Evaluation Associates for Latinos,  dr.salazar@real-consulting.org

Abstract: If you're struggling to build evaluation capacity among people involved in social change initiatives, Communities of Learning, Inquiry and Practice (CLIPs) may be useful to you. The testing of the CLIP process at a California community college under a National Science Foundation grant showed that the process 1) builds collaborative relationships, 2) enhances evaluative inquiry skills, and 3) leads to changes in professional practice. CLIPs are a structure for self-selected groups of organizational members to collaboratively investigate questions about their work. They undertake a three step evaluative inquiry process. The adaptable modules integrate research on complex adaptive systems, inquiry processes, learning theory, communities of practice, appreciative inquiry, and change processes. This session helps you develop the basic skills and knowledge to establish CLIPs and provides you with a CD of how-to modules. The CD contains many resources including videos to help you develop CLIPs in your own setting.

Session Title: Evaluation Policy in Operation: A Case Study of the New York City Center for Economic Opportunity

Panel Session 397 to be held in Centennial Section B on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Chair(s):

Jennifer Hamilton,  Westat,  jenniferhamilton@westat.com

Abstract: In December 2006, New York City's Mayor Michael Bloomberg established the Center for Economic Opportunity (CEO) to implement, monitor, and evaluate the City's new anti-poverty initiative. The overall goal of the initiative is to reduce poverty in the City through a range of programs targeted to the working poor, disconnected youth, and young children. A key hallmark of the initiative is an explicit policy on evaluation as a tool for accountability and decision-making. CEO operates results-driven programs and in turn, is working to fund evaluations that can be of the highest rigor possible to provide information on the implementation and outcomes of the programs. This session will include presentations from the director and other staff of the internal evaluation unit and the principal investigators of the coordinated set of external evaluations to provide insights into the genesis, development, and operation of CEO's evaluation policy.

Evaluation Policy in the New York City Center for Economic Opportunity Hector Salazar-Salame,  New York City Center for Economic Opportunity,  hsalame@cityhall.nyc.gov Carson Hicks,  New York City Center for Economic Opportunity,  chicks@cityhall.nyc.gov Kristen Morse,  New York City Center for Economic Opportunity,  David Berman,  New York City Center for Economic Opportunity,  bermand@hra.nyc.gov The first presenter, a CEO internal evaluator, will provide an overview of the New York City Center for Economic Opportunity (CEO) and the evaluation policy developed to help support CEO's decision-making. This presentation will begin by describing the impetus for CEO, how it is organized, and the rationale for the programs that are funded, as well as a descriptive overview of the programs (as they are designed to serve the key target groups of the working poor, disconnected youth, and young children.) The presenter will then outline CEO's policy on evaluation and describe the thinking behind the structure that involves internal evaluation, collection and use of performance data, use of performance-based contracts, and a coordinated set of external evaluations involving a range of approaches.

Setting the Stage for Evaluation: The Program Review Approach in the NYC Center for Economic Opportunity Evaluation Stan Schneider,  Metis Associates,  sschneider@metisassoc.com This presentation will describe the 'program review' approach used as a first step in the evaluation process with each of the programs funded within the CEO initiative. The approach, patterned in part after evaluabilty assessment (Wholey 1994), begins with the collection and review of documents, discussions with CEO staff, and joint meetings with CEO staff and agency staff to obtain an understanding of the design and operation of each program, as well as the research questions that CEO and the agency have about the program and its performance. The development of a logic model guides this initial stage, and the subsequent data collection and analysis activities. This presentation will discuss the evaluation team's experience in conducting the 40+ reviews, highlight lessons learned both methodologically and substantively, and how the lessons have been provided to CEO, the agencies, and the programs.

Strengthening the Sensitivity of Evaluation Designs to Detect Outcomes: The Use of a Technical Review Group in the NYC Center for Economic Opportunity Evaluation Debra Rog,  Westat,  debrarog@westat.com Each program in the NYC CEO initiative will have a program review conducted by the external evaluation team. An action plan, developed as part of the program review report, outlines possible evaluation designs and methods for addressing the evaluation questions raised by CEO and the agency, and through the review itself. To support the development of the most rigorous designs feasible that are also coordinated with one another, the evaluation incorporates a technical review process. This process, based on the concept of design sensitivity posited by Lipsey (1990), reviews features of each outcome study (e.g., the strength and fidelity of the target program; the quality of the measurement) that can affect the study's effect size. The presenter will describe how the technical review group operates as part of the cross-program evaluation team to ensure that each action plan addresses these features.

Using Results to Inform Action: The Experiences of the New York City Center for Economic Opportunity Kristen Morse,  New York City Center for Economic Opportunity,  kmorse@cityhall.nyc.gov Carson Hicks,  New York City Center for Economic Opportunity,  chicks@cityhall.nyc.gov David Berman,  New York City Center for Economic Opportunity,  bermand@hra.nyc.gov The final presenter from CEO will describe how the results from the evaluation activities are being used to guide program decision making. Any changes to the evaluation policy based on the activities to date will be described, and future plans for CEO based on evaluation findings will be discussed.

Session Title: Evaluation Policy in the Environmental Field: Are We in an Age of Slow Growth, Rapid Transformation or Running in Place?

Panel Session 398 to be held in Centennial Section C on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Presidential Strand

Chair(s):

Andy Rowe,  ARCeconomics,  andy.rowe@earthlink.net

Discussant(s):

Kathryn Newcomer,  George Washington University,  newcomer@gwu.edu

Abstract: Until recently, 'environmental program evaluation was best described as "emergent" and "growing." Environmental institutions' investments in environmental program evaluation have typically been 'ad hoc' or 'shoe-string', done in response to external demands for performance information and not because a visible evaluation policy supported its execution. However, we have real signs that environmental program evaluation can be described as 'maturing, and that environmental organizations are considering the role of evaluation policy within their performance management responsibilities. Through the examples of Washington State's King County, and the United States Environmental Protection Agency the presenters and discussant will engage the audience in a dialogue about Environmental Program Evaluation policy: At this time, how important are formal policies for promoting environmental program evaluation practice? If it is important, what is our vision of environmental evaluation policy? How can environmental institutions close the gap between having a vision of evaluation policy and having a practicing evaluation culture?

Evaluation Policy, Planning and Implementation: A Local Government Perspective on the Environmental Field Michael Jacobson,  King County Executive's Office,  michael.jacobson@kingcounty.gov King County, Washington has been finding ways to use program evaluation, evaluation tools, and evaluation approaches in its efforts to deliver better public services and improve environmental outcomes. Although most program evaluations are conducted by the County's legislative Auditor and serve an oversight function, there are instances where programs have conducted program evaluations for improvement purposes (Local Hazardous Waste, Public Health programs). In addition, the county has used logic model methodology to establish priority outcomes and performance measures as the basis of the county's KingStat performance management program. Furthermore, evaluation approaches are being built into program design and performance systems, such as the county innovative use of the Environmental Behavior Index, a social marketing approach to environmental behavior in county residents. As the County works on these performance management building blocks a question stands out: does this all add up to an "evaluation policy" for the County environmental programs?

Evaluation Policy on the Move at the United States Environmental Protection Agency Betsy Shaw,  United States Environmental Protection Agency,  shaw.betsy@epa.gov In March 2008, EPA's Deputy Administrator announced a vision and action statement specifically in support of program evaluation. The goal of the announcement was to institutionalize a corporate and strategic approach to program evaluation that has the Agency operating processes for identifying, prioritizing, investing in, and then using program evaluation results. This is the first time that "program evaluation" as a distinct activity has received such high-level endorsement from EPA's senior leadership. Now the hard work of forging a durable program evaluation policy at EPA has just begun. Key elements supporting an evaluation exist: senior level support, growing use of logic modeling and performance measurement, and a core group of staff with evaluation expertise. However, other critical elements are nascent or nonexistent: regular budget, regular access to external experts, applied experience within programs, and evaluation partnerships with other government players at the federal, state, tribal, or local level.

Session Title: What difference does it make? Systems Approaches to Evaluation

Think Tank Session 399 to be held in Centennial Section D on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Systems in Evaluation TIG and the Theories of Evaluation TIG

Presenter(s):

Glenda Eoyang,  Human Systems Dynamics Institute,  geoyang@hsdinstitute.org

Abstract: Evaluators face many barriers. Do systems approaches to evaluation offer any bridges to overcome those barriers? Systems thinking, tools, techniques, and methods have been apparent at AEA for over five years. This Think Tank will explore some fundamental questions about systems and evaluation, including: - What are 'non-systemic' evaluations? When, if ever, are they a good choice? - How are the multiple systems approaches similar to each other? Different from each other? - What are good selection criteria when choosing among systems-based evaluation approaches? - What are the risks and benefits of working with systems-based approaches as fundamental theories of change? As models of behavior and interaction? As methods of design, data collection, analysis, reporting? As tools to collect and analyze data? In small discussion groups, participants will address these questions, document their differences and agreements, and share their findings through an article that will be submitted for publication.

Session Title: New Developments in Collaborative, Participatory, and Empowerment Evaluation

Panel Session 400 to be held in Centennial Section E on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Chair(s):

David Fetterman,  Stanford University,  davidf@stanford.edu

Discussant(s):

Christina Christie,  Claremont Graduate University,  tina.christie@cgu.edu

Abstract: This session is designed to share some of the most recent developments in collaborative, participatory, and empowerment approaches to evaluation. Recognized leaders of each area will present the latest developments in their area. Liliana Rodriguez-Campo and Rita O'Sullivan will represent the collaborative camp. Lyn Shulha will present developments in participatory evaluation. David Fetterman and Abraham Wandersman will represent the empowerment evaluation approach. Christina Christie will compare theories associated with these approaches. The process will include a participatory component. In addition to a question and answer component, the session will invite members of the audience to participate and share their insights and experience. Topics will include process use to online surveys.

Collaborative Evaluation: A Step-By-Step Model for Improving Collaborative Evaluations Liliana Rodriguez Campos,  University of South Florida,  lrodriguez@coedu.usf.edu Rita O'Sullivan,  University of North Carolina Chapel Hill,  ritao@email.unc.edu This session will be presented by Liliana Rodriguez-Campos and Rita O'Sullivan. The focus will be on two areas: 1) specific steps that have been refined to enhance the process of conducting collaborative evaluations; and 2) recent innovations and developments associated with practice in the field.

Participatory Evaluation: Moving Beyond Technical Skills to Help Guiding Participatory Evaluators Lyn Shulha,  Queen's University at Kingston,  shulhal@educ.queensu.ca This session about latest developments in participatory evaluation step beyond the mechanical steps and skills associated with conducting participatory evaluations. Guiding principles, value orientations, and related topics will be highlighted to enhance actual practice.

Empowerment Evaluation: New Developments Ranging from Translating Findings to Policy Makers to Interactive Web-based Tools David Fetterman,  Stanford University,  davidf@stanford.edu Abraham Wandersman,  University of South Carolina,  wandersman@sc.edu This session will highlight recent developments in the area of empowerment evaluation. David Fetterman will discuss recent developments associated with translating empowerment evaluation findings into the language of policy makers. Specifically, he will focus on his work in the area of tobacco prevention, translating the number of people who quit smoking into dollars saved in terms of excess medical expenses. He will also highlight the new Arkansas Evaluation Center, focusing on empowerment evaluation. Abraham Wandersman will present recent developments associated with iGTO or interactive Getting to Outcome. This is a 10 step approach to empowerment evaluation on the web. He will present recent efforts to use this tool in 48 coalitions and two states.

Collaborative, Participatory, and Empowerment Evaluation: Comparing and Contrasting Theories Christina Christie,  Claremont Graduate University,  tina.christie@cgu.edu This presentation will highlight underlying theories associated with collaborative, participatory, and empowerment evaluation. Issues associated with social justice, capacity building, and self-determination, will explored. Similarities and differences will be highlighted.

Session Title: Federal Evaluation Policy: Impact on Funding Priorities, Evaluation Research and Practice for Broadening Participation in Science, Technology, Engineering and Mathematics (STEM)

Panel Session 401 to be held in Centennial Section F on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Multiethnic Issues in Evaluation TIG

Chair(s):

Elmima Johnson,  National Science Foundation,  ejohnson@nsf.gov

Abstract: Current Federal evaluation policy has its roots in both the current political emphasis on accountability and scientific rigor as well as budget realities. The impact is felt, not only in agency funding priorities, but also in evaluation research and practice. This session will examine the impact of federal legislation such as the No Child Left Behind Act on educational program evaluation activities and initiatives. Attention will be given to how policy has influenced evaluation capacity building and evaluation practices for supporting underrepresented groups. Questions to be addressed include: (1) what challenges do scientific rigor pose for the evaluation of educational initiatives and opportunities for underrepresented groups and (2) what should be the role of evaluation in advancing the knowledge base on broadening participation, while addressing accountability requirements. Additionally the presentation will outline a comprehensive approach for assessing value, rigor and impact of education programs focused on diversity and equity.

Federal Evaluation Policy in STEM: A Historical Perspective Elmima Johnson,  National Science Foundation,  ejohnson@nsf.gov This presentation will cover how policy has influenced evaluation capacity building and evaluation practices for supporting underrepresented groups. NSF's Broadening Participation activities will be highlighted.

Metrics for Monitoring Broadening Participation Efforts Toni Clewell,  Urban Institute,  tclewell@ui.urban.org This presentation will focus on the identification of program monitoring metrics, indicators for program evaluation, and rationale for their use.. The issue is what represents an acceptable, valid, and sufficient set of indicators applicable across groups of programs by which to measure progress in broadening participation. .The presenter has a wealth of experience in the identification and analysis of BP efforts in STEM education as well as the identification of metrics to measure outcomes across projects.

Designs and Indicators for Program Evaluation Bernice Anderson,  National Science Foundation,  banderso@nsf.gov This presentation will focus on which study design options are appropriate under which sets of circumstances. More specifically, of the various assessment methods available, each has a certain utility and appropriateness under a defined set of circumstances. The presenter for this session will draw on their expertise to define the context of NSF's portfolio of programs.

Session Title: Participatory Evaluation With Youth Through Community Forums: A Demonstration of a Youth-Adult Training Curriculum

Demonstration Session 402 to be held in Centennial Section G on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Extension Education Evaluation TIG

Presenter(s):

Mary Arnold,  Oregon State University,  mary.arnold@oregonstate.edu

Elissa Wells,  Oregon State University,  elissa.wells@oregonstate.edu

Roger Rennekamp,  Oregon State University,  roger.rennekamp@oregonstate.edu

Abstract: Despite the growing interest in the area of youth participatory evaluation, very few models and materials exist to train youth and adults on how to conduct a participatory evaluation project. The purpose of this session is to provide a hands-on overview of a training curriculum used with 4-H youth development programs. The curriculum is designed to train youth and adult partners in participatory evaluation using community forums as a method of data collection. The session will cover the content of the curriculum as well as the process by which the curriculum is taught. Curriculum topics include: training on youth-adult partnerships, framing issues for community evaluation, hosting community forums, analyzing forum data, summarizing and reporting results, and developing a community action plan. Participants will learn about the process of teaching the curriculum, thus providing participants with strategies and ideas for engaging youth in participatory evaluation.

Session Title: Program-Level Evaluations in Higher Education: Expanding Practices and Policy

Multipaper Session 404 to be held in Mineral Hall Section A on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Assessment in Higher Education TIG

Chair(s):

Howard Mzumara,  Indiana University-Purdue University Indianapolis,  hmzumara@iupui.edu

Discussant(s):

Stanley Varnhagen,  University of Alberta,  stanley.varnhagen@ualberta.ca

Outcomes Assessment Utilization in the Context of Teacher Education Program Review Presenter(s): Georgetta Myhlhousen-Leak,  University of Iowa,  leak@q.com Abstract: This research investigated the factors and types of use present in teacher education outcomes assessment. A review of the literature suggests a significant lack of use. The absence of a definable structure for understanding use and its potential for desirable change has made the process of incorporating outcomes assessment into the processes of higher education a difficult, unclear and sometimes disjointed experience for many administrators and faculty. Use as identified and described in the evaluation utilization literature was applied to investigate how program administrators, faculty administrators, and faculty members who act as evaluators and/or intended users conceptualize use and how those conceptualizations are reflected in planning, implementation and reporting of outcomes assessment. This research provides evidence and insights into the nature of evaluative use and adds to the collective knowledge of use and how use occurs within the evaluation policy context of teacher preparation program review in higher education.

Quasi-experimental Outcome Evaluation of Undergraduate and a Graduate Entrepreneurship Education Programs Presenter(s): Elaine Rideout,  North Carolina State University,  ecrideout@econinvest.org Denis Gray,  North Carolina State University,  denis_gray@ncsu.edu Abstract: To the degree entrepreneurship education can increase the number and effectiveness of entrepreneurs, particularly for high technology ventures, economic growth may result. Unfortunately, very little empirical evidence exists that connects entrepreneurship education to subsequent economic growth. This paper will describe a quasi-experimental control group study of two long-running (10+ years) and respected university-based high-tech entrepreneurship education (E-ed) models – one for undergraduates and the other for graduate students. Data collection is currently underway and will identify concrete E-ed economic impacts and mediating factors (personal and environmental) associated with entrepreneurial propensities, activities and success. Impact on entrepreneurial intentions, enterprise activities, entrepreneurship activities (including start ups) will be reported. Results will also report evidence about critical E-ed mediators including personality characteristics, entrepreneurial orientation preferences and other variables. Implications for the design, operation, and evaluation of University entrepreneurship education programs will be discussed.

Thinking Differently About Student Assessment and Program Evaluation in a School of Education: A Self-Study and Blue Print for Change! Presenter(s): Patrick Lowenthal,  Regis University,  plowenth@regis.edu John White,  Regis University,  jwhite@regis.edu Karen Cooley,  Regis University,  kcooley@regis.edu M Sue Davies,  Regis University,  mdavies@regis.edu Abstract: The faculty in the School of Education at Regis University conducted a self-study as a part of the national accreditation process for the Teacher Education Accrediting Council (TEAC). Through this process, the faculty realized that they did not have enough evidence of student learning. This sparked a re-evaluation of how student assessment and program evaluation were conducted in the school. The faculty developed a new assessment and evaluation process that requires students to submit pre-identified artifacts from individual courses to an electronic portfolio. At certain stages (i.e., “gates”) throughout the program, students must submit their portfolio for faculty review in order to continue in the program. Data collected by “gate keepers” enables the faculty to assess student learning—both individually and collectively—but also to evaluate the program as a whole—whether that be at the instructor level, the course level, or the program level.

Evaluating Foreign Language Student Residences: Best Practices and Lessons Learned Presenter(s): Cary Johnson,  Brigham Young University,  cary_johnson@byu.edu Wendy Baker,  Brigham Young University,  wendy_baker@byu.edu Jennifer Bown,  Brigham Young University,  jennifer_bown@byu.edu Rob Martinsen,  Brigham Young University,  rob.martinsen@byu.edu Abstract: Foreign language student residences provide language learners with an opportunity to be immersed in their foreign language without the high cost associated with traveling and living in those countries. But how effective are these residences in helping learners acquire the language? The language residences selected for the evaluation were French, German, Russian, and Japanese. Qualitative methods of evaluation include video-taped dinner conversations and interviews with the participants. Quantitative methods include a language use survey, log of daily language use, ACTFL Oral Proficiency Interview scores, a computer-adaptive proficiency test, and a pronunciation test. This presentation will focus on the outcomes of the evaluation as well as the lessons learned in conducting language program evaluations.

Session Title: Diverse Approaches and Methods in Human Services Evaluation

Multipaper Session 405 to be held in Mineral Hall Section B on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Human Services Evaluation TIG

Chair(s):

Vajeera Dorabawila,  New York State Office of Children and Family Services,  vajeera.dorabawila@ocfs.state.ny.us

Discussant(s):

James Sass,  Los Angeles Unified School District,  jimsass@earthlink.net

The HousingPlus Project: A Collaborative Values-Based Approach to Evaluating the Implementation of Supportive Housing in Ottawa, Ontario Presenter(s): John Sylvestre,  University of Ottawa,  jsylvest@uottawa.ca Purnima Sundar,  Carleton University,  purnimasundar@hotmail.com Matthew Manion,  University of Ottawa,  mmani069@uottawa.ca Jaclynne Smith,  University of Ottawa,  jsmit3@uottawa.ca Katie Bendell,  University of Ottawa,  kbendell@hotmail.com J Bradley Cousins,  University of Ottawa,  bcousins@uottawa.ca Tim Aubry,  University of Ottawa,  taubry@uottawa.ca Abstract: HousingPlus is a collaboration among eight supportive housing providers (including tenants, senior management, staff) and a university-based evaluation team. The objectives are 1) to develop tools and methods for evaluating the implementation of supportive housing, 2) to evaluate program implementation, and 3) to use evaluation findings to assist participating agencies to collaborate in developing solutions to shared implementation challenges. In support of these objectives, participants developed a tool to evaluate the implementation of supportive housing across the participating agencies. This tool incorporated nine housing values that participants identified as underlying the provision of supportive housing programs. Building on a “key component profile” approach to evaluating program implementation (Cousins et al., 2004), the principles were used to specify levels of program implementation (from low to high) for key program components within four program domains (Housing, Support, Person, Systems). The presentation describes the evaluation tool, and the process used to develop it.

Developing a Data-Driven Approach to Program Management: Lessons Learned from a Community-based AIDS Service Organization in New York City Presenter(s): Margaretta Fox,  Harlem United Community AIDS Center Inc,  mfox@harlemunited.org Rashi Rohatgi,  Harlem United Community AIDS Center Inc,  rrohatgi@harlemunited.org Abstract: Using semi-structured interviews with key stakeholders (e.g., Board members, agency executives, managers, and front-line staff, etc.), as well as service and administrative data, this paper will provide a case study of the development of internal evaluation capacity at an AIDS service organization in New York City. It will explore: 1) the history of the agency and the impetus behind our decision to build internal capacity in evaluation; 2) successes and challenges of program/evaluation collaborations, to date, from a variety of stakeholder perspectives; 3) data on how evaluation efforts have reshaped programs and contributed to organizational success; and 4) future directions for our evaluation initiatives. Of particular interest will be how the agency has used evaluation activities to improve and redirect programs and create a more data-driven organizational culture.

Long-term Evaluation for Service Improvement: Practice and Potential Presenter(s): Sharon Baggett,  Portland State University,  sharon@pcez.com Paul Iarrobino,  Multnomah County Aging and Disability Services,  paul.iarrobino@co.multnomah.or.us Abstract: Whether an agency service is assessed over-time may be driven by available funding, the need for outcome measures by key stakeholders, and/or the commitment of management to consistent quality improvement. Long-term evaluation, however, allows service agencies to better adapt services to changing customer profiles and needs. In 1999, Multnomah County (Portland, Oregon) Aging and Disability Services (ADS) began a long-term evaluation of their telephone information and assistance services. Five surveys have been completed, including 1997, 1999, 2001-2002, 2004, and 2006. A sixth is scheduled in 2008. The evaluation has, over time, adapted sampling approaches, survey items, and other aspects of implementation, such as telephone versus on-line surveys for after-hours customers, in an effort to improve the information gathered and thus, enhance the information available to staff for program improvement. This presentation describes the evolution of the evaluation, its uses and importance to program staff, and lessons learned over long-term implementation.

Evaluation of a Statewide Prevention Program: An Illustration of an Effective Research-Practitioner Model Presenter(s): Marcia Hughes,  University of Hartford,  mhughes@hartford.edu Timothy Black,  University of Hartford,  tblack@hartford.edu Abstract: Since its inception in 1995, program evaluation and research have been integrated components of Connecticut’s Nurturing Families Network, a statewide program that focuses on high-risk, first-time mothers and is designed to promote positive parenting and prevent child maltreatment. Guided by questions at the national level, descriptive and outcome data have been routinely analyzed and used for monitoring changes occurring in areas that the program is trying to impact. Researchers have also worked closely with practitioners and front-line workers using their expertise and observations to gain insight into emerging issues. Interviews, focus groups, and ethnographic field work have been used to acquire an understanding of program intervention, program dynamics, and characteristics of the families receiving services. This paper chronicles key stages of evaluation and research as the program has evolved, and illustrates how carefully designed research can be used to inform and refine practices and create program change over time.

Session Title: Practical Guidance and Tools for Advocacy Evaluation

Demonstration Session 406 to be held in Mineral Hall Section C on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Advocacy and Policy Change TIG

Presenter(s):

Jane Reisman,  Organizational Research Services,  jreisman@organizationalresearch.com

Tom Kelly,  Annie E Casey Foundation,  tkelly@aecf.org

Kendall Guthrie,  Bill and Melinda Gates Foundation,  kendall.guthrie@gatesfoundation.org

Anne Gienapp,  Organizational Research Services,  agienapp@organizationalresearch.com

Sarah Stachowiak,  Organizational Research Services,  sstachowiak@orginaizationalresearch.com

Justin Louie,  Blueprint Research and Design Inc,  justin@blueprint.com

Catherine Crystal Foster,  Blueprint Research and Design Inc,  catherine@blueprintrd.com

Abstract: For the past few years, funders and evaluators have been working on the theoretical frameworks for evaluating advocacy efforts. Now the rubber is hitting the road to see how this will play out in practical applications. To support the implementation and use of evaluation for advocacy work, funders are supporting the development of guidance resources and tools for grantees and program officers as well as providing capacity-building to grantees. This session will share examples of these strategies that have been developed for Annie E. Casey's Kids Count state grantees, Gates Foundation-funded community engagement grantees, and PICO affiliates funded by The California Endowment. Attendees will learn about the frameworks undergirding these projects and see concrete examples of guidance resources and tools. Additionally, presenters representing funders and evaluators will reflect on successes, lessons learned, and further application of these kinds of materials and processes for the burgeoning field of advocacy evaluation.

Session Title: More Bang for the Buck: Getting the Most From a Statewide Evaluation of Programs for At-Risk Students

Panel Session 407 to be held in Mineral Hall Section D on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG

Chair(s):

Ken Seeley,  National Center for School Engagement,  kseeley@coloradofoundation.org

Abstract: Increasingly educational researchers are demanding comprehensive and often costly evaluation designs. How can evaluators do justice to multi-program initiatives with limited resources? Panelists will talk about the evaluation of a Colorado categorical program designed to educate expelled and at risk students. Each year more than 50 school districts, charter schools, private and parochial schools are funded to provide academic and support services for expelled and at risk students. Recently state lawmakers legislated a 1% allocation of the total monies to evaluation. Through online data collection, selective in-depth inquiry, and analysis of data collected at the state level, the evaluation of this program has provided enough evidence to not only continue but also increase the dollars allocated by policy makers. Come to this panel to learn how to satisfy policy makers with enough data despite limited resources.

More Mileage from One Tank: Limiting Program Staff Burden but Reporting 'Enough' to Key Decision Makers Cindy Wakefield,  Colorado Department of Education,  wakefield_c@cde.state.co.us Cindy Wakefield, from the Colorado Department of Education will provide a context for this panel including the history of the Expelled and At Risk Student Services program (EARSS), which sprang from an amendment to the Colorado Constitution. Cindy will also explain how the evaluation has been funded since early 2000 and the recent legislation, which allocates 1% of the program category to evaluation. Most importantly, the evolution of the evaluation will be discussed. Key aspects of this presentation will include the process of moving from a paper based to online system, how we determined key data elements for collection, challenges of getting accurate data and attempts to limit the burden on school staff. Finally, Cindy will give details about how the evaluation results have been reported to the state legislature and joint budget committee as well as how these legislative bodies have used these data.

Taking a Bird's Eye View: Using State Student Identifiers in Combination with Standardized Test Scores to Evaluate Legislated Programs Heather MacGillivary,  University of Colorado Boulder,  heather.macgillivary@colorado.edu Legislators and other policy makers want to know how programs impact student success, however limited evaluation resources often prohibit in-depth academic assessments or long-term follow-up. To address this predicament, evaluators of the Expelled and At Risk Student Services (EARSS) program, took advantage of statewide data systems to assess student outcomes. The EARSS Leadership Team selected five programs to participate in this pilot longitudinal study. Student state ID's and corresponding educational information were obtained for each student who participated in five programs. Program staff also provided student participation and program exit data. These data were combined with data from the student state identification system and the student assessment data for multiple years proceeding and following program enrollment. Heather MacGillivary the principal evaluator will discuss the challenges of conducting this research, including issues of confidentiality, data access, analysis and interpretation of the results.

Adding Depth to a Statewide Evaluation: A Qualitative Study of Family School Partnerships for At Risk Students Judy Martinez,  Colorado Department of Education,  martinez_j@cde.state.co.us Large statewide evaluations can typically only provide an overview of aggregate results, however this is often not enough to make local programmatic improvements. This was the challenge facing the evaluators of the Colorado's Expelled and At-Risk Student Services initiative. In particular, we knew from program feedback and previous evaluation results that involving families of at risk students was challenging. A qualitative study of family involvement was undertaken to uncover promising practices. Over thirty telephone interviews were conducted with program staff. Six parent focus groups were completed and multiple parent involvement activities were observed in communities across Colorado. Judy Martinez from the Colorado Department of Education will discuss the findings as well as how these findings are being used to improve practice at the local level. Judy will also discuss other strategies for adding depth such as collecting success stories, funding a local evaluator, and investigating clusters of similar programs.

Session Title: Evaluation Meets Management: What Every Evaluator Should Know About His or Her Role in Performance Measurement - Examples From the Centers for Disease Control and Prevention (CDC)

Panel Session 408 to be held in Mineral Hall Section E on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Evaluation Managers and Supervisors TIG

Chair(s):

Thomas Chapel,  Centers for Disease Control and Prevention,  tchapel@cdc.gov

Abstract: The increasing emphasis on accountability forces high-level decision makers to contemplate program performance in a disciplined way. However, in certain organizations, planners, budgeters, evaluators, and performance monitors work in isolation from one another, using approaches and terms so differently that opportunities to meld insights into a common approach for improving the organization are missed. This panel presents three CDC programs with strong evaluation efforts under way and where the prior thinking on program evaluation and logic modeling served the program well in addressing performance measurement mandates as they occurred. Program representatives will describe their situations, their prior evaluation approaches, the nature of the mandates to which their programs are subject, and how evaluation efforts and insights have helped meet those mandates. The payoffs of this closer association for both performance measurement and evaluation will be presented.

Indicator Development, Measurement, and Data Collection: Successes, Hazards, and Detours Experienced by the Prevention Research Centers Jo Anne Grunbaum,  Centers for Disease Control and Prevention,  jgrunbaum@cdc.gov Demia Wright,  Centers for Disease Control and Prevention,  amy7@cdc.gov Nicola Dawkins,  Macro International Inc,  nicola.u.dawkins@macrointernational.com Amee Bhalakia,  Macro International Inc,  amme.m.bhalakia@macrointernational.com Natalie Birnbaum,  Northrop Grumman Corporation,  nib6@cdc.gov Eduardo Simones,  Centers for Disease Control and Prevention,  ejs0@cdc.gov The Prevention Research Centers (PRC) Program is a network of 33 academic research centers funded by the Centers for Disease Control and Prevention. Researchers collaborate with public health agencies and community members to conduct applied prevention research. A national evaluation strategy began in 2001 to address program accountability and produced a logic model, program indicators, and qualitative studies. The indicator data describe aspects of prevention research that can be tracked over time, such as the number of research and training projects occurring at the centers, PRCs' publications and presentations, community involvement in the research, and adoption of PRC-tested interventions by communities. This presentation describes the development of both the indicators and an Internet-based information system for data collection and the participatory processes used throughout development. Strategies that worked well in addition to the hazards and detours encountered along the way will be shared.

Performance Measures: An Important Tool when Conducting Multi-Site Evaluations Ann Ussery-Hall,  Centers for Disease Control and Prevention,  aau6@cdc.gov The Steps Program is a multi-site, multi-level, multi-focus program. Through Steps, CDC funds 40 communities nation-wide to implement chronic disease prevention and health promotion activities. The Steps team developed a set of core performance measures to systematically measure program implementation and progress toward intended health outcomes throughout the five-year program. These standardized measures provide a cumulative, national view of the program. Because communities have their own evaluation plans and data collection methods, the performance measures are an important tool for providing unifying, cross-cutting information. Annually, the core performance measures provide Steps stakeholders with a picture of what is occurring nationally. This systematic approach to collecting data from communities has allowed CDC to create a national picture of the local activities. The utility of the core performance measures will continue as CDC considers next steps, including dissemination of materials and lessons learned, and replication of Steps components.

Indicator Development to Tell Your Program's Story CDC's National Asthma Program Leslie Fierro,  Centers for Disease Control and Prevention,  let6@cdc.gov Elizabeth Herman,  Centers for Disease Control and Prevention,  ehh9@cdc.gov Measurement of program 'indicators' that align with the logic underlying a program can help identify and communicate areas for improvement, document progress made toward intended outcomes, and demonstrate successes. Indicators developed in this way have the potential to 'fill in' the gaps left by performance measures typically produced for the purposes of GPRA and PART and may enable programs to tell their story more fully. This presentation will track the efforts of CDC's National Asthma Program in developing process and outcome indicators that reflect the story and logic of 35 state asthma programs. The collaborative process used to facilitate a shared vision of the program's purpose, to articulate evaluation questions, and to develop a process for measuring indicators will be described.

Session Title: Evaluating Afterschool Programs: Issues of Policy and Practice

Multipaper Session 409 to be held in Mineral Hall Section F on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

James Van Haneghan,  University of South Alabama,  jvanhane@usouthal.edu

Promoting Developmentally Sensitive Programming and Analysis of Afterschool Programs for Middle School Youth Presenter(s): Tiffany Berry,  Claremont Graduate University,  tiffany berry Katherine Byrd,  Claremont Graduate University,  katherine.byrd@cgu.edu Krista Collins,  Claremont Graduate University,  krista.collins@cgu.edu Susan Menkes,  Claremont Graduate University,  susan.menkes@cgu.edu Abstract: Over the past decade, multiple process and outcome evaluations have been conducted across a range of different afterschool programs, as shown in a recent Harvard Family Research Project publication (Little, Wimer, & Weiss, 2008). These evaluations have explored how program dosage, program quality, type of enrichment activity, etc. affects the academic and social development of youth. This literature has provided the foundation for the articulation of “universal” principles of effectiveness, which if well-implemented, will result in pronounced academic and social benefits. Although this research is compelling, it is limited by not addressing the unique contexts, circumstances, and developmental levels that accompany the diverse landscape of youth afterschool. Using results from an on-going evaluation of Afterschool Allstars, Los Angeles, this paper will examine how afterschool programs and the evaluation field could benefit from measuring these universal principles within a contextually-sensitive framework by incorporating child, staff, and environmental characteristics into the design.

The Growth of Out-of-School Time Evaluations: Impact and Consequences Presenter(s): Tara Donahue,  Learning Point Associates,  tara.donahue@learningpt.org Abstract: Over the past decade, out-of-school time research and evaluation has grown exponentially. These studies have had a major impact on afterschool and summer program policy and funding at the national, state, and local level. Additionally, this body of work has helped develop an understanding of the components of a quality afterschool program as well as provide resources to practitioners to help them use data for continuous program improvement. This paper summarizes the evaluation studies that have had the most impact on the field, the consequences of those studies, and ways in which the future direction of out-of-school time evaluation and research can continue to impact the field.

Building Evaluation Capacity While Navigating Afterschool Program Policies Presenter(s): Sae Lee,  Harder and Company Community Research,  slee@harderco.com Monica Kaiser,  Kaiser Group Inc,  kaisermonica@mac.com Abstract: The current policy environment for afterschool programs presents a challenge for building evaluation capacity for many afterschool programs. There are considerable pressures for afterschool programs to focus on academic outcomes and utilize experimental design to demonstrate program impact. In this policy climate for afterschool programs, it becomes increasingly important to take into account contextual factors, such as participants’ school and community, to ensure that expected outcomes and planned measures are appropriate. This presentation will discuss a collaborative partnership between the external evaluator and Woodcraft Rangers afterschool program in building capacity to develop evaluations that address both policy needs as well as understanding program outcomes in context. The process of evaluation capacity building, challenges encountered and benefits derived by both the evaluator and the afterschool program will be discussed.

Session Title: Ethics and Qualitative Evaluation Methods: A Complex Intersection

Panel Session 410 to be held in Mineral Hall Section G on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Qualitative Methods TIG

Chair(s):

Jennifer Greene,  University of Illinois Urbana-Champaign,  jcgreene@uiuc.edu

Discussant(s):

Rodney Hopson,  Duquesne University,  hopson@duq.edu

Abstract: While in broad outline qualitative and quantitative evaluation research methods share similar concerns with each other and with other types of social research, qualitative evaluation entails particular attention to interpersonal relationships within the evaluation team and between evaluators and participants. This is because the evaluator's primary tool in the process of observation and interview is the ability to form trusting and trustworthy relationships. As self-understanding is at the basis of ethical, effective relationship development, the first paper in the session addresses the importance of ongoing self-calibration as the foundation for relating authentically to others across barriers posed by culture, gender, age, abilities, social class and so on. The second presenter explores self and other within the setting of ethnography with extended and evolving relationships between researcher and informant while the final paper brings notions of ethics as they pertain to self-in-relationship-with-other in the specific sensitive research topic of sexual orientation.

Cultivating Self as Responsive Instrument for Excellence and Ethical Practice in Evaluation Hazel Symonette,  University of Wisconsin Madison,  symonette@bascom.wisc.edu Evaluators are privileged authorities with social powers to define reality and make impactful judgments about others. Yet, from our privileged standpoints, we often look but still do not see, listen but do not hear, touch but do not feel. Such limitations handicap our truth-discerning and judging capacities. When we fail to systematically address the ways our sociocultural lenses, filters and frames may obscure or distort more than they illuminate, we do violence to others' truths and erode validity. To address these issues, we must constantly expand our understandings of self in dynamically diverse contexts within power and privilege hierarchies and our understandings of the contexts embodied in the self. This paper explores the need for calibrating and cultivating our most valuable instrument--the SELF--as an open and expansively learning-centered, responsive instrument. This is an ethical responsibility as well as an essential pathway for professional excellence.

Ethnography: The Dialectic between Poles: Informing as an Inter-subjective Process Anne Ryen,  Agder University,  anne.ryen@uia.no My focus is on the researcher-informant relationship in ethnography when using an ethno-informed approach. I claim that field-relations interact with our data, and that we need to explore the finely grained aspects of how such complex field relations evolve across time and contexts while still avoiding the potential pitfalls of subjective reflexivity. Rather, an inter-subjective perspective highlights informing as an interpersonal process with emotions as well as eruptions, as core aspects of this kind of inquiry. The importance of this focus is at least twofold. First, the interconnectedness leads to reflections on self and other enabling us to resist 'Othering' as the classic colonial twist of much cross-cultural work. Second, this then leads to credible and legitimate qualitative evaluation by our analyses being firmly based on detailed, natural occurring data. My discussion draws on my research on Asian business in East-Africa.

LGBTQ Issues: Protecting Vulnerable Subjects in ALL Evaluations SJ Dodd,  City University of New York,  sdodd@hunter.cuny.edu The purpose of this presentation is to explore the issue of ethical evaluation practice from the perspective of lesbian, gay, bisexual, transgender and queer (LGBTQ) individuals. It also highlights the particular ways in which standard aspects of ethical evaluation practice, such as informed consent, confidentiality and protection from harm may vary for LGBTQ individuals. This presentation draws attention to areas where LGBTQ persons may be especially vulnerable in evaluation situations and where careful consideration by the evaluator is essential to protecting the human subjects involved. Statistically, it is likely that any evaluation may include some subjects who overtly or covertly identify as lesbian, gay, bisexual, transgender, or queer. Therefore, the presentation culminates with suggestions for best practices to consider when conducting evaluations involving human subjects regardless of the particular focus of the evaluation.

Roundtable: Nontrivial Pursuits: Can We Use Games for Serious Work?

Roundtable Presentation 411 to be held in the Slate Room on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the AEA Conference Committee

Presenter(s):

Brenda Peters,  Benjamin Rose Institute,  bkpeters@oh.rr.com

Abstract: Since games exist in all human societies, it is reasonable to conclude that they are important to us far beyond modern connotation of leisurely pastimes. They serve several functions, from teaching strategy to moral instruction to social interaction. Educators develop games to stimulate learning from preschool to corporate environments. Psychological experiments explore human behavior through simulations. When designing a learning game, I noted skills needed to create a game are strikingly similar to skills and activities in evaluation. So I ask: could games be used as a technique in evaluation? Data collection? Needs assessment? For reaching stakeholders and understanding program models? Engaging clients in the evaluation process? Bridging cultural differences? What kinds of games would work, in what context? I hope this roundtable will provide an opportunity to discuss some of the evaluative potential and inherent challenges in using games as a technique in evaluation.

Session Title: Building Policy and Practice in the Arts

Multipaper Session 412 to be held in the Agate Room Section B on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Evaluating the Arts and Culture TIG

Chair(s):

Paul Lorton Jr,  University of San Francisco,  lorton@usfca,edu

Discussant(s):

Brian Lawton,  University of Hawaii,  blawton@hawaii.edu

Evaluating the Effectiveness of Academic Capital in Arts for Learning Programs Presenter(s): Gina Weisblat,  Cleveland State University,  boo500@aol.com Abstract: Founded in 2004 with a grant from the Ford Foundation, the Art is Education initiative of the Cleveland Integrated Arts Collaborative initiative seeks to close the achievement gap by providing all students in Cleveland Metropolitan School District with experiences in the arts, everyday. Art is Education initiative plans evaluate the impact of this four year project, (beginning Fall 2008) using Academic Capital Theory. The central thesis of academic capital theory is that 'relationships and environment matter'. The fundamental idea is that social networks are a valuable asset, and that those networks coupled with an individual’s background and outlook help define one’s ability to succeed individually and contribute to the greater good and a community of academic learners. The components of academic capital are the critical factors that enable people to learn. Classroom observation, focus groups, formal assessment and interviews will be used to measure the effectiveness of this approach.

Fund-raising Effectiveness in the Nonprofit Lively Arts: Moral Hazard as an Unintended Outcome Presenter(s): Charles Gray,  University of St Thomas,  cmgray@stthomas.edu Abstract: This research is designed to ascertain the relationships among government grants, private donations, and earned income for the lively or performing arts. The target organizational population is approximately 35 of the most significant (budgets ranging from under $100,000 to more than $41 million) performing arts (dance, theater, and music) organizations in the Minneapolis-St. Paul metropolitan area. In a predecessor project, a cross-section analysis of a single budget year indicates that, across all included organizations, the combination of $1 of private contributions and $1 of government grants crowds out $2.15 of earned revenues. This paper expands the original project to include several years of budget data, permitting a panel study and exploration of lagged effects in support of evaluating development and marketing in the arts.

The Deconstruction of the Evaluation Report: Whose Policy is it Anyway? Presenter(s): A Rae Clementz,  University of Illinois,  clementz@uiuc.edu April Munson,  University of Illinois Urbana-Champaign,  amunson2@uiuc.edu Abstract: The final evaluation report is a fixture of evaluation practice. Whether the format is mandated or simply expected, the structure of an evaluation report is a kind of evaluation policy. This paper deconstructs the implications of a reporting policy in a city-wide school-based arts initiative. David Easton described policy as "an authoritative allocation of values" (1954), but whose values are being represented? In this initiative, reporting policy had consequences on the question that could be asked, data collected, participant and community relations, even skewing the validity and usefulness of the results. We give particular emphasis to the imposition of a report structure on the multiple perspectives that exist in an evaluation, setting up a narrative that weights some values and silences others. Practical examples are presented of alternative methods for reporting evaluation results, with an emphasis on educative strategies and the potential offered by digital media, performance, and participation.

Formalizing Evaluation in an Arts Integration School/University Partnership (Project AIM Columbia College, Chicago): Building Policy and Practice Presenter(s): Vanessa McKendall Stephens,  Face Value Evaluation Consulting Inc,  vanessa@facevalu.org Cynthia Weiss,  Columbia College Chicago,  cweiss@colum.edu Mary Ellen Murphy,  Independent Consultant,  mesm2@msn.com Lara Pruitt,  Columbia College Chicago,  lkpruitt@yahoo.com Beverly Dretzke,  CARE,  dretz001@umn.edu Shawn Lent,  Columbia College Chicago,  slent@colum.edu Sadira Muhammad,  Columbia College Chicago,  smuhammad@colum.edu Abstract: The paper presentation will discuss the results of the Arts Integration Mentorship Project’s (Project AIM) work with evaluators to develop a systematic approach to evaluating its efforts with selected Chicago and Evanston elementary and middle schools. As a school partnership of Columbia College Chicago’s Center for Community Arts Partnerships, AIM developed a participatory, reflective process to further clarify its model and engage participants in learning what works. The evaluation supported Project AIM’s ability to embed organizational practices and principles that guide inquiry related to the program theory and evaluation questions. Contextually specific evaluation policy was developed in partnership with teaching artists, classroom students and teachers, Project AIM and college personnel. Both evaluator and program staff perspectives will be presented along with useful tools. The program’s unique approach has been documented in the recently published book, AIMprint: New Relationships in the Arts and Learning, co-edited by Cynthia Weiss and Amanda Lichtenstein.

Session Title: Multi-Level Models for Evaluating Training and Knowledge Management

Multipaper Session 413 to be held in the Agate Room Section C on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Business and Industry TIG

Chair(s):

Kelly Hannum,  Center for Creative Leadership,  hannumk@ccl.org

From Evaluation to Development: The Role of Mentoring in 360-Degree Assessment Presenter(s): Ray Haynes,  Indiana University,  rkhaynes@indiana.edu Rajashi Ghosh,  University of Louisville,  r0ghos01@gwise.louisville.edu Abstract: This presentation offers a contextual discussion of the organizational climate and culture necessary for the implementation of 360-degree assessment and feedback. It proposes that mentoring and coaching serve as the primary feedback processes for fostering developmental actions after the 360-degree assessment process. Additionally, it offers an evaluative framework for assessing the implementation of 360-degree assessment programs in organizational settings. The evaluative framework provides guidance in assessing the efficacy of the developmental feedback provided through the mentoring and coaching process. Finally, this presentation applies Kirkpatrick’s (1959) four stage evaluation model to evaluate the role of mentoring in transforming 360-degree feedback and assessment from an evaluative appraisal to a developmental intervention.

Evaluating Training on Strategic Interviewing with Professional Skepticism: Levels I - V Presenter(s): John Mattox,  KPMG,  jmattox@kpmg.com Darryl Jinkerson,  Abilene Christian University,  darryl.jinkerson@coba.acu.edu Pete Sanacore,  KPMG,  psanacore@kpmg.com Michele Graham,  JVA Consulting LLC,  magraham@kpmg.com Abstract: Statement of Auditing Standards No. 99 (SAS 99) requires auditors to consider fraud at various stages of an audit of financial statements. However, detecting fraud is a difficult task for auditors because most have not experienced fraud in their professional careers (Montgomery, Beasley, Menelaides, and Palmrose, 2002). SAS No. 99 stresses the magnitude of auditors maintaining professional skepticism throughout the course of an audit “regardless of past experience with the entity or prior beliefs about management’s honesty and integrity” (AICPA 2002, AU 316.13). The current study presents a multi-level evaluation (levels 1 – 5) of an 8 hour local office program recently offered by a large international auditing firm focused specifically on strategic interviewing with professional skepticism. Objectives of the course included improving the quality of the audit by developing an attitude of professional skepticism in the context of an audit, applying the vendor’s tactical behavior assessment model in detecting deception, identifying and recognizing verbal and non-verbal deceptive behavior, and applying effective interviewing and questioning techniques.

Chain of Impact: Building an Evaluation Framework for Business Results Presenter(s): Bruce Aaron,  Accenture,  bruce.aaron@accenture.com Abstract: A comprehensive measurement framework with logically defined evaluation levels can be designed for a wide range of programs using the “V-model”, a methodology adopted from the field of application and systems development. This session demonstrates how the V-model has been applied in evaluation of learning and knowledge management (KM) programs within Accenture, a large global company providing management consulting, outsourcing, and technology services. The model provides a roadmap for data collection across user-defined evaluation levels, including ROI (return on investment), and its application within Accenture has been recognized by best case study awards from ASTD and the ROI Institute. This session will introduce the V-model and illustrate how the framework has been used in evaluation of training and KM programs. The demonstrated advantages of developing this causal model will include isolation of program effects, identification of areas for improvement, and determination of ROI.

Session Title: Evaluation Research at the National Institute of Justice

Panel Session 414 to be held in the Granite Room Section A on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Crime and Justice TIG

Chair(s):

Angela Moore,  United States Department of Justice,  angela.moore.parmley@usdoj.gov

Abstract: Evaluation research at the National Institute of Justice will be presented and discussed. A number ongoing evaluation projects will be described including technology evaluation research and results from recently completed projects. Future plans for evaluation research will be outlined.

Technology Evaluation Research at the National Institute of Justice Phelan Wyrick,  National Institute of Justice,  phelan.wyrick@usdoj.gov Dr. Wyrick is an Evaluation Specialist with NIJ. His contribution will be to provide information regarding technology evaluation research across the Institute.

Evaluation Research in the Crime Control and Prevention Research Division Winnie Reed,  Crime Control and Prevention Research Division,  winnie.reed@usdoj.gov Winnie Reed is Chief of the Crime Control and Prevention Research Division. She will discuss past, present and future research within the Division.

Evaluation Research within the Office of Research and Evaluation Angela Moore,  United States Department of Justice,  angela.moore.parmley@usdoj.gov Angela Moore is Assistant Deputy Director of the Office of Research and Evaluation. She will discuss past, present and future evaluation research within ORE.

Session Title: Perspectives on the Evaluation of a College and School District Partnership for Urban Teacher Preparation

Panel Session 415 to be held in the Granite Room Section B on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Sandra Foster,  Mid-continent Research for Education and Learning,  sfoster@mcrel.org

Discussant(s):

Jean Williams,  Mid-Continent Research for Education and Learning,  jmwilliams@mcrel.org

Abstract: The Metropolitan State College of Denver (MSCD) and Denver Public Schools (DPS) were awarded a Teacher Quality Enhancement (TQE) partnership grant to prepare teachers to teach in urban middle and high schools. These partners share the goal of developing and validating a model for training urban teachers that will be nationally recognized, of high quality, forward-thinking in its approach, and sustainable over time. Mid-continent Research for Education and Learning (McREL) is serving as the external evaluator of the program. First, the evaluation design will be presented followed by discussions from the key stakeholders (MSCD and DPS) of their perspectives related to the TQE partnership and their involvement in the evaluation of the program.

An Evaluation Design for an Urban Teacher Preparation Program Trudy L Clemons,  Mid-Continent Research for Education and Learning,  tclemons@mcrel.org Ruby C Harris,  Mid-Continent Research for Education and Learning,  charris@mcrel.org Sandra Foster,  Mid-Continent Research for Education and Learning,  sfoster@mcrel.org Jean Williams,  Mid-Continent Research for Education and Learning,  jmwilliams@mcrel.org The evaluation of the Teacher Quality Enhancement (TQE) partnership grant is designed to provide program decision makers with objective, informative, and easily accessible data that will help them to make mid-course corrections, determine what's working and why, and describe both anticipated and unanticipated program outcomes. In order to accomplish this, the evaluation is focused on five tasks; (1) benchmarking best practices in university-school partnerships which have improved teacher recruiting, preparation, and retention in urban schools, (2) conducting systems analysis to understand the structure of organizations and provide rich and detailed information that will help institutionalize the more effective and efficient components of the system, (3) evaluating the alignment of redesigned curriculum, (4) examining the extent to which the urban teacher coursework at MSCD supports the field experience of students, and (5) examining the extent to which the field experience of students is enhanced through the structure of 'urban apprentice schools.'

The Evaluation of an Urban Teacher Preparation Program from the College Perspective Esther M Rodriguez,  Metropolitan State College of Denver,  erodri25@mscd.edu As the Principal Investigator and Director of the Teacher Quality Enhancement (TQE) grant at Metropolitan State College of Denver, this panelist will share a unique perspective on the evaluation of an urban teacher preparation program. The panelist will discuss developing a partnership with a school district as well as the process of fitting a new program within the existing organizational culture and priorities of an existing teacher education program. In addition, discussion will focus on how the evaluation is being used to determine best approaches to the goals of (1) establishing a foundation for a program to prepare teachers for urban and hard to staff schools, (2) developing authentic collaboration between the college and school districts, and between the urban teacher preparation program and the current teacher education program, and (3) ultimately creating an urban education center.

The Evaluation of an Urban Teacher Preparation Program from the School District Perspective Theress Pidick,  Denver Public Schools,  theress_pidick@dpsk12.org As the Director of the Teacher Quality Enhancement (TQE) grant and University Partnerships in Denver Public Schools, this panelist will share a unique perspective on the evaluation of an urban teacher preparation program. This panelist will also discuss the developing university partnerships as well as the process of fitting a new program within the existing organizational culture and priorities of a school district. In addition, discussion will focus on how the evaluation is being used to determine best approaches to the goals of (1) retaining teachers in hard to staff schools, (2) placing teachers in the district who already have a familiarity with the curriculum, practice, and culture, and (3) creating a cadre of qualified teachers for placement in urban schools.

Session Title: Approaches for Assessing Emergency Preparedness: Performance Indicators, Evaluator Technique Alignment, and Modeling Personal Behavior Change

Multipaper Session 416 to be held in the Granite Room Section C on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Disaster and Emergency Management Evaluation TIG

Chair(s):

Karen Pendleton,  University of South Carolina,  ktpendl@gwm.sc.edu

Using Frame of Reference and Rater Technique Alignment to Increase Inter-Rater Accuracy and Reliability: Public Health Disaster Response Tabletop Exercises Presenter(s): Lisle Hites,  Tulane University,  lhites@uab.edu Abstract: Public Health disaster discussion based or Tabletop Exercises (TTX) have become a common tool for helping to prepare Public Health emergency responders for all hazards response. However, while the Homeland Security Exercise and Evaluation Plan (HSEEP) gives specific requirements for the evaluation of such TTX, there has been little attention paid to the actual rating process employed by exercise evaluators. This presentation will discuss this issue and present findings from the field that compare traditional/typical TTX where evaluators do not undergo any intervention, with TTX where evaluators receive a short intervention involving aspects of Frame of Reference (FOR) and Rater Technique Alignment (RTA) interventions immediately prior to evaluation of a TTX.

Emergency Preparedness Evaluation at United States Nuclear Power Plants Presenter(s): Otto Gustafson,  Western Michigan University,  ottonuke@yahoo.com Abstract: This paper will outline the operating experience and regulatory environment driving emergency preparedness efforts in the civilian nuclear power industry. A case study will be presented discussing the key elements and activities of an emergency response organization, including organizational structure, continuing training, as well as drill and event performance. Additionally, this paper will examine the evaluation methodology utilized by U.S. civilian nuclear power plants in assessing and reporting the state of their emergency preparedness. Specific attention will be paid to emergency preparedness criteria and indicators developed by the Nuclear Energy Institute for use by both the licensees and the Nuclear Regulatory Commission. The author will identify key areas for improvement in the U.S. civilian nuclear power industry’s emergency preparedness evaluation and outline associated recommendations.

Session Title: Determining What is Critical in the Program Evaluation of A Multi-Site College Access Program

Panel Session 417 to be held in the Quartz Room Section A on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the College Access Programs TIG

Chair(s):

Imelda Castaneda-Emenaker,  University of Cincinnati,  castania@ucmail.uc.edu

Discussant(s):

Jennifer E Williams,  JE Williams and Associates LLC,  jew722@zoomtown.com

Abstract: This panel illustrates how a rigorous program evaluation can be built upon the conceptualization of key constructs for particular components of a college access program called Gaining Early Awareness and Readiness for Undergraduate Program (GEAR UP). The presenters are all part of the evaluation team for Ohio's multi-site GEAR UP program. The panel shares how the identification of relevant evaluative criteria and determination of their importance can lead to meaningful program expansions, strong support of stakeholders, and program sustainability. We discuss the development and application of evaluative criteria then narrow the focus onto one important construct: school climate as condition for students' success. Presentations feature lessons learned along with the theoretical foundations, process of merit determination, and results that contributed to the utility of the findings. Tentative suggestions for changes in local, state, and federal policies are discussed.

Upping the Ante: Establishing Focused Evaluative Criteria for a Program Evaluation of a College-Access Program Janet Matulis,  University of Cincinnati,  janet.matulis@uc.edu This presentation focuses on the use of evaluative criteria driving the program evaluation of a college access program. Program goals are typically used as benchmarks for evaluation purposes; however, our experience as the evaluation team of Ohio's multi-site Gaining Early Awareness and Readiness for Undergraduate Program (GEAR UP) is consistent with Davidson's (2005) claims regarding the importance of establishing focused evaluative criteria. This develops a systematic process for honing in on valid conclusions about program performance. This presentation discusses how the Ohio GEAR UP evaluation has become more purposive by focusing on the underlying constructs and evaluative criteria used in our surveys of teachers, students, and parents. Evaluative criteria drove the revisions to GEAR UP's surveys that further clarified constructs for different components considered for the success of the program. Policy implications of this process are explored.

School Climate and Student Achievement: Theoretical Considerations in Evaluation Hsin-Ling Hung,  University of Cincinnati,  hsonya@gmail.com This presentation centers on theoretical concerns related to one of the key constructs in the program evaluation of Ohio's multi-site Gaining Early Awareness and Readiness for Undergraduate Program (GEAR UP): the effects of school climate on student achievement. Our evaluation team's concept is grounded in the view that school climate is the total environment quality within the school building (Anderson, 1982) and that school climate defines the school level learning environment (Johnson & Stevens, 2006). Tagiuri's (1968) taxonomy of climate is presented followed by his three school climate theories. Variables in Tagiuri's model of climate and research on the relationship between school climate and student achievement are examined. Lessons learned from the theories and researches about the assessment of school climate as an important influence on student academic success are addressed. Policy implications related to school climate, student achievement, and college access are discussed.

Merit Determination in Evaluating Conditions for Students' Success in a College Access Program Imelda Castaneda-Emenaker,  University of Cincinnati,  castania@ucmail.uc.edu In this panel, the presenter shares the perspectives of an experienced evaluator of Ohio's Gaining Early Awareness and Readiness for Undergraduate Program (GEAR UP). The presentation focuses the merit determination processes used in the design of a major data collection instrument for this multi-site college access state program: the surveys for teachers, students, and parents. Although the surveys are meant to gather data for all program components, the school climate as condition for students' success is highlighted as an example of one of the major prioritized constructs. The presentation illustrates the use of some strategies posited by Davidson (2005) in determining the importance of this particular construct drawing on the knowledge of key stakeholders, using evidence from literature, and using program theory. This process ensures that data of significance to the project stakeholders are gathered, analyzed, and used for appropriate program improvements and reports for accountability. Policy implications are discussed.

An Illustration of Utility of Evaluation Findings: Addressing Teachers' Perceptions Jerry Jordan,  University of Cincinnati,  jerry.jordan@uc.edu As part of the multi-site Ohio Gaining Early Awareness and Readiness for Undergraduate Program (GEAR UP) evaluation team, this presenter focuses on the characteristics of school climate brought to light as a result of the team's evaluative process. The merit determination phase of our evaluation highlighted student expectations as central to school climate. Our measures discerned a gap between teachers' perceptions of student college-going potential and the students' own perceptions of their college-going potential. This finding helped the program focus ongoing program objectives and allowed individual sites to target changes in teachers' perceptions as one short term goal. This presentation highlights how the program has adjusted to these findings. Finally, this presentation addresses the difficult issue of how policy makers might influence teacher perceptions and expectations.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Organizational Characteristics That Support Meaningful Monitoring and Evaluation Functions

Roundtable Presentation 418 to be held in the Quartz Room Section B on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

Presenter(s):

Jason Ackerman,  Catholic Relief Services,  jasonackerman1@gmail.com

Stuart Belle,  World Vision,  sbelle@worldvision.org

Carlisle Levine,  CARE,  clevine@care.org

Abstract: NGO accountability and measures of effectiveness are frequent conference topics. Does your agency's monitoring & evaluation function enable or constrain solutions to these challenges? The answer depends on a number of agency characteristics, and the influence of the monitoring & evaluation function within it. Using a summary of their field experience and industry literature, roundtable presenters will describe organizational characteristics with a tendency to either enable or constrain an agency's ability to create solutions. The life-cycle of an agency is often decades long. Observing the impact of organizational characteristics can be an equally long-term task. However, the intense operating tempo of an international emergency has a specific duration and the response activities reflect a microcosm of an agency's culture, organization, etc. Presenters will focus on the success of monitoring and evaluation functions within international emergency response settings, and will also offer a comparison to their success within international development settings.

Roundtable Rotation II: Collaboration as a Means to Building Capacity: Evaluating the Impact of Collaboration on Science, Technology, Engineering and Mathematics Programs in the National Girls Collaborative Project

Roundtable Presentation 418 to be held in the Quartz Room Section B on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

Presenter(s):

Karen Peterson,  Puget Sound Center for Teaching, Learning, and Technology,  kpeterson@psctlt.org

Carrie Liston,  Puget Sound Center for Teaching, Learning, and Technology,  cliston@psctlt.org

Abstract: Evaluators from the National Girls Collaborative Project, which aims to increase the levels of collaboration between individuals and groups that work to support girls’ interest and achievement in STEM, will present data on these efforts, including levels of collaboration of participating programs and other types of organizations, incentives an opportunities to collaborate in-person and online, and challenges to collaboration. Data sources include a survey sent to almost 600 programs on collaboration experiences, feedback from in-person and online events encouraging collaboration, and case-studies of collaborative projects. We will look at a “collaboration rubric”, adapted from the work of Hogue (1993), Borden and Perkins (1988, 1999) and Frey, Lohmeier, Lee, Tollefson & Johanning (2004), developed to capture increasing levels of collaboration between different groups. Challenges of measuring collaboration will be discussed, including forming a standard definition, identifying indicators, creating data collection tools, and determining a standard of success.

Session Title: Health Evaluation TIG Business Meeting and Multipaper Session: Accounting for Differential Implementation in Evaluations of Multi-Site Community Initiatives

Business Meeting with Panel Session 419 to be held in Room 102 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Health Evaluation TIG

TIG Leader(s):

Ann Zukoski,  Oregon State University,  ann.zukoski@oregonstate.edu

Jeannette Oshitoye,  Nemours Health and Prevention Services,  joshitoy@nemours.org

Robert LaChausse,  California State University at San Bernardino,  rlachaus@csusb.edu

Shyanika Rose,  Battelle,  rosesw@battelle.org

Chair(s):

Doug Easterling,  Wake Forest University School of Medicine,  dveaster@wfubmc.edu

Abstract: This panel brings together evaluators from three community-change initiatives who have developed innovative strategies for measuring and modeling 'site-level dose.' The session begins with a review of why it is important to account for differential implementation in evaluating multi-site initiatives, along with an overview of the challenges associated measuring implementation in the context of comprehensive community initiatives. The remaining papers describe the specific approaches to measurement and statistical modeling employed in the three evaluations. Each evaluation faced the challenge that the intervention was defined according to general principles rather than specific services or activities. However, each evaluation was able to develop reliable methods of assessing how fully the intervention was implemented, and was able to incorporate these assessments as moderator variables in analyses of impact. Site-level dose analyses provided a more complete and nuanced appraisal of the intervention's effectiveness than was available through simple Treatment by Time analyses.

Assessing Site-Level Dose in Multi-Site Community Initiatives: Rationale, Challenges, and General Strategies Doug Easterling,  Wake Forest University School of Medicine,  dveaster@wfubmc.edu Todd Rogers,  Public Health Institute,  txrogers@pacbell.net Mark Wolfson,  Wake Forest University,  mwolfson@wfubmc.edu This paper provides background on the use of 'site-level dose' (SLD) as an adjunct to evaluations of a multi-site initiatives. The basic logic is that when a program model is fully implemented in only a few sites, including SLD as a moderator variable may allow for the detection of program effects that do not attain significance in an overall analysis of the program. However, this approach proves challenging in the case of comprehensive community initiatives. This paper describes the measurement challenges that arise when the initiative has a complex and/or 'fuzzy' program model, or when sites are encouraged to tailor the model to local conditions. We also consider the threats to validity that occur when SLD is used to predict change in participant behavior. These challenges set the stage for the remaining three papers which focus on successful strategies for measuring SLD and for incorporating SLD into impact analysis.

Measuring and Modeling Site-Level Dose in the Evaluation of Free to Grow Todd Rogers,  Public Health Institute,  txrogers@pacbell.net Doug Easterling,  Wake Forest University School of Medicine,  dveaster@wfubmc.edu Rebecca Neiberg,  Wake Forest University,  rneiberg@wfubmc.edu Mark Wolfson,  Wake Forest University,  mwolfson@wfubmc.edu Free to Grow (FTG) was a national initiative in which Head Start (HS) programs partnered with community organizations to implement a comprehensive approach to the prevention of substance abuse and child abuse. The impact evaluation of FTG used a quasi-experimental design in which 14 FTG communities were compared with 14 comparison sites. Changes in parenting and neighborhood conditions were assessed through surveys of caregivers of young children, some of whom were enrolled in HS (cohort study) and some from the larger community (repeated cross-sectional). The primary analysis found no evidence that the intervention had an effect. To account for possible implementation failure, the FTG sites were classified into high versus low implementers. The paper describes how implementation was measured and modeled within the impact analyses. No consistent effect was detected within the two Cohort Samples. However, within the Community Sample, high-implementers showed positive improvements on two measures of neighborhood conditions.

Measuring and Modeling Site-Level Dose in the Evaluation of the Study to Prevent Alcohol Related Consequences (SPARC) Kimberly Wagoner,  Wake Forest University,  kwagoner@wfubmc.edu Scott D Rhodes,  Wake Forest University,  srhodes@wfubmc.edu Thomas P McCoy,  Wake Forest University,  tmccoy@wfubmc.edu Heather Champion,  Wake Forest University,  hchampio@wfubmc.edu Ashley Wagoner,  Wake Forest University,  aswagone@wfubmc.edu Mark Wolfson,  Wake Forest University,  mwolfson@wfubmc.edu SPARC was a five-year randomized community-trial using a community-organizing approach to implement environmental strategies in and around North Carolina college campuses. The Environmental Strategy and Implementation Survey (ESIS), completed by 'point-people' at the 10 schools (5 Intervention, 5 Comparison), measured each site's Inputs and Processes in implementing the intervention. These measures were used as covariates and effect modifiers to assess if schools implementing a SPARC-like intervention showed greater improvements in outcomes. Overall, Intervention schools had higher ESIS scores than Comparison schools, indicating that Intervention schools implemented higher levels of the intervention. ESIS by Time interactions were comparable to Treatment by Time interactions across measures that evaluated student drinking behavior and associated consequences. ESIS provided confirmation of the study's main effects findings and indicated that implementation occurred as anticipated.

Measuring and Modeling Site-Level Dose in the Evaluation of Enforcing Underage Drinking Laws (EUDL) Rebecca Neiberg,  Wake Forest University,  rneiberg@wfubmc.edu Kimberly Wagoner,  Wake Forest University,  kwagoner@wfubmc.edu Jill Newman,  Wake Forest University,  jinewman@wfubmc.edu, Eun-Young Song,  Wake Forest University,  esong@wfubmc.edu Barbara A Martin,  Wake Forest University,  bmartin@wfubmc.edu Debbie Pleasants,  Wake Forest University,  dpleasan@wfubmc.edu Mark Wolfson,  Wake Forest University,  mwolfson@wfubmc.edu The Enforcing Underage Drinking Laws Randomized Community Trial (EUDL-CT) was funded by the Office of Juvenile Justice and Delinquency Prevention to determine the effects of a local coalition-based approach to implement most promising strategies for increasing enforcement of laws related to underage drinking and reducing underage drinking. Assessments of quality, quantity, support and sustainability for the core strategies of compliance checks, enforcement focused on social availability of alcohol, DWI law enforcement, and policy initiatives were used to characterize the strength of dose in each community. Data from multiple sources, including Law Enforcement and Local Coalition surveys, Activity Tracking Data and site visit records, were employed. Combination of data reduction algorithms and consensus meetings of evaluation team members produced dose measures for the core strategies. These measures are being used for secondary analyses that will examine the relationship between degree of implementation and outcomes.

Session Title: Data Collection Challenges With Populations Having Low-Literacy and Disabilities

Multipaper Session 420 to be held in Room 104 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Special Needs Populations TIG

Chair(s):

Gordon Bonham,  Bonham Research,  gbonham@bonhamresearch.com

Who Should Speak for Individuals with Intellectual Disabilities? Evaluating Quality of Life at Community Providers Presenter(s): Gordon Bonham,  Bonham Research,  gbonham@bonhamresearch.com Abstract: Quality of life of people with intellectual disabilities is difficult to measure. Questions arise about both self response and proxy response. The Ask Me! Survey collects data annually for 1,200 individuals with developmental disabilities. Peer interviewers, allow three-fourths of the selected people to respond for themselves. Two proxies provide information for each person who cannot respond. Self-respondents answer more questions, produce more reliable scales, and have nearly the same internal consistency as proxies. Self-respondents report lower physical well-being and higher self-determination than do proxies. Two proxies agree most on emotional well-being and least on self-determination. Two day staff agree the most; family and staff proxies agreed the least. Self and proxy responses can be combined for many analysis with the statistical controls. Participatory evaluation policy can be put into practice, but doesn’t resolve all the problems in evaluating agencies serving people with differing intellectual abilities.

Adapting Appreciative Inquiry for Use at the Community Level in South Africa: Experiences with AI among Orphan and Vulnerable Children (OVC) Programs Presenter(s): Beverly Sebastian,  Khulisa Management Services (Pty) Ltd,  bsebastian@khulisa.com Peter Njaramba,  Khulisa Management Services (Pty) Ltd,  pnjaramba@khulisa.com Mary Pat Selvaggio,  Khulisa Management Services (Pty) Ltd,  mpselvaggio@khulisa.com Abstract: To document the strengths of 32 PEPFAR-funded OVC programs in South Africa, Khulisa designed an evaluation using an Appreciative Inquiry (AI) approach. Per AI methodology, we randomly paired respondents from each program (staff, volunteers, beneficiaries, and community members) to interview one another using AI-framed tools. Unfortunately, this proved challenging due to participants’ low literacy levels and lack of experience in interviewing. Participants - especially beneficiaries and community members - found the questions complex and consequently their stories lacked detail. Subsequently we modified our approach to have trained fieldworkers interview respondents in focus groups using AI-framed tools. This led to better understanding of (and responses to) the AI questions, as it allowed probing and gathering of richer stories – participants found the emotional telling of, and listening to, stories as informative and therapeutic; service providers felt more encouraged in their work; and beneficiaries and stakeholders had better understanding of their OVC program.

The Peer Employment Benefits Network: Evaluating the Effectiveness of Peer-to-Peer Communication among People with Disabilities Presenter(s): Jennifer Sullivan Sulewski,  University of Massachusetts Boston,  jennifer.sulewski@umb.edu Abstract: The Peer Employment Benefits Network is a pilot project using peer-to-peer networking to change the “word on the street” about employment options for people with disabilities. Training was provided to a select group of “peer leaders” who then conducted outreach to others with disabilities. The peer-to-peer networking design presented an evaluation challenge in that peer interactions are delicate and intrusions such as observation or formalized data collection would alter the nature of the interaction itself. We used multiple methods and data sources to evaluate the project’s effectiveness while protecting the confidentiality of the peer-to-peer interaction. Data collected included tests of peer leaders’ knowledge (to assess training effectiveness), peer leaders’ self-reports on their outreach activities (to assess the amount of peer-to-peer networking), a survey of individuals peer leaders talked to (to assess the quality and usefulness of the peer-to-peer interaction), and interviews with peer leaders and other stakeholders.

Using Readability Tests to Improve the Accuracy of Evaluation Documents Intended for Low-Literate Participants Presenter(s): Julien Kouame,  Western Michigan University,  julienkb@hotmail.com Abstract: This project was to develop and evaluate a simple and understandable survey for formative evaluation and to assess the effect of the readability test on low-literate participants. A child abuse evaluation survey designed and pretested was borrowed for this assessement. The evaluation was conducted with 65 low-literate participants (10 years of formal schooling) for whom English was their second language. Participants were randomly assigned into two groups. One group used a form of the survey which content was tested to suit the readability level using Flesch?Kincaid formula. Participants were asked to evaluate instructions and the understandability of each item. For each group, the understanding level was calculated. Differences in understanding were determined by 2 tests. The two documents were generally well understood. However, the document with the readability test presents better understandability score. This shows that the readability test has a positive effect on the comprehension of the survey.

Session Title: Advances in Propensity Score Research: Improving Methods to Reduce Bias in Quasi-Experiments

Panel Session 421 to be held in Room 106 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Chair(s):

MH Clark,  Southern Illinois University at Carbondale,  mhclark@siu.edu

Abstract: Quasi-experiments are useful for studies that need to be conducted in applied settings where random assignment to treatment groups is not practical. However, a major disadvantage in using these designs is that the treatment effects may not yield unbiased estimates. Propensity scores, the predicted probability that cases will be in a particular treatment group, are often used to help model and correct for this selection bias. The studies presented as part of this panel represent the latest findings in propensity score research. This panel will present the effectiveness of reducing bias using propensity score analysis compared to (a) conventional regression adjustment for covariates, and (b) single covariate matching using a variety of matching techniques.

Can Propensity Scores Reduce Regression to the Mean? MH Clark,  Southern Illinois University at Carbondale,  mhclark@siu.edu Computer simulations were used to examine how various matching methods reduced regression to the mean that occurs in non-randomized experiments that use matching. Simulations were created so that the true effect size was zero, but the treatment and control groups were biased so that the control group appeared to perform better than the treatment group. The regression artifact was created by adding random error to each observed variable. A three-factor design examined the effects that type of matching, proportion of available matches, and the number of variables used to compute propensity scores had on reducing the regression artifact. It was expected that (a) full matching would be more effective than paired matching; (b) having more control units available to match to treatment units would be more effective than matching equal numbers of control and treatment units; and (c) matching on propensity scores created from several covariates would be more effective than single covariate matching.

Propensity Score Analysis versus Traditional Regression Adjustment for Covariates: A Monte Carlo Study Jason Luellen,  Vanderbilt University,  jason.luellen@vanderbilt.edu David S Cordray,  Vanderbilt University,  david.s.cordray@vanderbilt.edu With regard to adjusting for confounding in nonexperimental studies, recent review papers have questioned the utility of propensity score analysis relative to conventional regression adjustment for covariates, reporting that effect estimates were often comparable for studies that presented both types of adjustments on the same data. However, these findings were tempered by the fact that many published propensity score analyses were not well implemented. This paper briefly explores the rationale behind each of the adjustment methods and presents the findings from a Monte Carlo study investigating their relative performance in terms of bias reduction and precision. The authors simulated a nonequivalent control group design and performed thousands of trials adjusting the outcome estimates for propensity scores and for the same raw covariates using regression.

Examining the Effects of Communities In Schools Inc (CIS) on School-Level Outcomes with the Use of Propensity Score Analysis Methods: A National Application Katerina Passa,  ICF International,  kpassa@icfi.com Heather Clawson,  ICF International,  hclawson@icfi.com Susan Siegel,  Communities In Schools Inc,  ssiegel@cisnet.org As a component of the national evaluation of Communities In Schools, Inc. (CIS), our team sought to quantify the impact of the CIS network on several academic and behavioral outcomes across elementary, middle, and high schools in seven states. Using propensity score matching techniques, we matched CIS schools with non-CIS schools on several characteristics, including attendance rates, racial/ethnic composition, dropout rates, and test scores, and then compared outcomes. The study examined schools served by CIS for 3 to 5 consecutive years between the 1998-99 and 2003-04 school years, and those that did not have exposure to the CIS program during the same period. Our findings indicate that programs in schools using the CIS model demonstrate positive outcomes for students over a three-year period. Specifically, our presentation will focus on the methodology and use of propensity score analysis in a quasi-experimental evaluation of a national program model.

Controlling for Endogeneity Bias in Evaluating Alcoholics Anonymous’ Effect on Drinking Stephen Magura,  Western Michigan University,  stephen.magura@wmich.edu Evaluation studies consistently report correlations between Alcoholics Anonymous (AA) participation and less drinking or abstinence. Randomization of alcoholics to AA or non-AA is impractical and difficult. Unfortunately, non-randomization studies are susceptible to artifacts due to endogeneity bias, where variables assumed to be exogenous (“independent variables”) may actually be endogenous (“dependent variables”). Such common artifacts are selection bias, where different types of people choose to participate or not participate in AA, and reverse causation, where reducing or stopping drinking may lead to increased or sustained AA participation. The paper will present a plan for using three specific statistical techniques to control for possible selection and/or reverse causation biases in a national alcoholism dataset - propensity score matching, instrumental variable analysis and structural equation modeling with cross-lagged panel analysis. This presentation will be primarily conceptual, not mathematical or statistical, and thus accessible to evaluators and others without advanced statistical training.

Session Title: Theory-Based Evaluation and Cultural Contexts

Multipaper Session 422 to be held in Room 108 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Program Theory and Theory-driven Evaluation TIG

Chair(s):

Katrina Bledsoe,  Walter R McDonald and Associates Inc,  kbledsoe@wrma.com

Experiences with Theory-Based Evaluation: Views from Evaluators in Denmark Presenter(s): Peter Dahler-Larsen,  University of Southern Denmark,  pdl@sam.sdu.dk Abstract: A discrepancy may exist between the text book version of an evaluation model and how it works under real-life conditions. Evaluators who have worked with a model constitute an important source of information about the latter. A web-based survey was conducted among evaluators in Denmark who had done theory-based evaluation. The survey results decribe why that model is chosen, how long time it takes, how the program theory is constructed, what the consequences of theory-based evaluation are, and what evaluators view as the major strengths and weaknesses of theory-based evaluation as they understand it.

Using the System of Care Practice Review (SOCPR) as a Quality Improvement Tool in a System of Care Community in Broward County Florida Presenter(s): Keren S Vergon,  University of South Florida,  vergon@fmhi.usf.edu Norin Dollard,  University of South Florida,  dollard@fmhi.usf.edu Linda Ross,  University of South Florida,  lross@fmhi.usf.edu Abstract: One Community Partnership (OCP), along with the other CMHS-funded programs, embraces a system of care philosophy in which a comprehensive, coordinated, community-based system of care brings together all agencies and resources needed to provide services to children with SED. OCP completed three consecutive System of Care Practice Review (SOCPR) studies from 2004 through 2007. Scores from 2004 to 2007 showed improvement with the overall case score increasing from the neutral implementation range to the enhanced range. Similarly, all subdomain scores improved between 2004 and 2007. The greatest improvement was in cultural competence, followed by impact, and child-centered, family-focused. Community-based showed the least change, but was in the enhanced range in 2005. OCP has been successful at improving its inclusion of SOC values and principles in service practices across all domains. Strengths include the agency continuing to be accessible to families, providing services in the least restrictive environment, and case management.

The Iraq War Described in Logic Models Presenter(s): Uda Walker,  Gargani and Company Inc,  udaowalker@yahoo.com John Gargani,  Gargani and Company Inc,  john@gcoinc.com Abstract: The purpose of this presentation is to explore how we think about programs. In particular, I focus on how we represent our thinking with logic models. I present logic models and other diagrams used to plan the Iraq War in order to expose the cognitive traps that await us when we set out to develop and evaluate programs. I argue that programs are only as good as the thinking that goes into them, and that representations such as logic models are only as good as the thinking we get out of them. I suggest concrete ways that evaluators can strengthen their thinking and their representations in order to improve programs and policies, punctuating my suggestions with provocative examples.

Session Title: Research and Development Challenges to Meeting Government Performance Requirements

Panel Session 423 to be held in Room 110 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Government Evaluation TIG and the Research, Technology, and Development Evaluation TIG

Chair(s):

Kathryn Law,  National Institutes of Health,  lawka@od.nih.gov

Discussant(s):

Deborah Duran,  National Institutes of Health,  durand@od.nih.gov

Abstract: The federal government seeks to fund effective programs that are able to achieve long term goals. However, current performance reporting methodologies are not sufficient to assess many programs, especially research and development (R&D) programs. This panel will discuss challenges faced by the federal government in assessing and managing R&D programs. These programs often conduct high risk, high reward research that may not achieve all of its proposed outcomes, but do yield unplanned results that guide the discovery process. Large scientific research initiatives also require new ways of thinking about evaluation. Aggregating the evaluation results of individual components does not yield a sound assessment of the entire program. These programs require new methodologies for adaptive and system assessments that can more appropriately capture the value of these initiatives. Finally, the challenges of incorporating assessment results into planning and decision making activities require the development of structures which encourage positive adaptive change.

High Risk, High Reward Research at the National Institutes of Health (NIH) Goutham Reddy,  National Institutes of Health,  reddygo@mail.nih.gov High Risk/High Reward programs are challenged to meet the many performance reporting requirements. Current methodologies are unable to assess the expected adaptations needed to advance science. Initially, high risk innovative projects struggle to meet the planned goals; yet, properly adapt to the scientific discovery process by dynamically following the direction of good scientific discovery. Unplanned results emerge that guide the discovery process. There are no current assessment approaches that can properly determine project performance. Under the current approach of setting a planned annual milestone, then assessment of met / not met is inadequate. Furthermore, high reward projects are only determined after impact can be assessed, which can only be assessed post the end of the project. Current requirements do not enable follow-up reporting. Until these methodologies and policies can be developed, R&D programs must be allowed to use alternative strategies, such as adaptive annual measures with sound scientific justifications.

Evaluation Policy and Evaluation Practice for Large Scientific Research Initiatives William Trochim,  Cornell University,  wmt1@cornell.edu The Clinical and Translational Science Awards (CTSAs) initiative is one of the largest scientific research efforts funded by the National Institutes of Health. It is designed to transform how clinical and translational research is conducted, ultimately enabling researchers to provide new treatments more efficiently and quickly to patients. The CTSA consortium currently includes 24 academic health centers (AHCs) located throughout the nation; by 2012, about 60 institutions will be linked together nationally. The CTSA has from its inception integrated evaluation into its efforts at multiple levels. This presentation describes the different types of evaluation policies that have been instituted or created - including requirement of a separate evaluation proposal in their center grant proposal; use of logic modeling; development of a national cross-center evaluation steering committee; and integration of performance and outcome evaluation - and considers the implementation and practice challenges of evaluating this type of large complex and adaptive research endeavor.

What's the Use of Studying Science: Case - Using Profiling Analysis to Inform Science Management Decision Making Ken Ambrose,  National Institutes of Health,  ambrosek@mail.nih.gov Required performance reporting often begins with self assessment. The results of these assessments can give support to current reporting and decision-making structures. However, if the conclusions present challenges to current practices, then such research becomes a disruptive influence in the social, political, and economic system of an organization. The challenge becomes one of creating safe harbors for change that help communities recognize existing structures, address the complexities of organizational change, and incorporate stakeholders values and concerns. This presentation uses the case of an analysis of scientist profiles in funding decisions to explore the role of infrastructure and incentives in managing change based on internal assessment. It also discusses the factors for communities to develop structures which encourage positive adaptive change.

Session Title: The New Era of Performance Evaluation in Korean Evaluation System

Multipaper Session 424 to be held in Room 112 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Research, Technology, and Development Evaluation TIG

Chair(s):

Young-Wha Cho,  Korea Institute of Science & Technology Evaluation and Planning,  yhcho@kistep.re.kr

Abstract: There has been increased requirement on the accountability and efficiency of national R&D programs in according to the increased R&D investment within the limited national resources. To cope with these issues, each government is trying to promote its R&D capability by implementing S&T basic plans and Korea is also trying to upgrade national S&T capability under the guidance of NSTC (National S&T Committee). With the limited government resource and increasing demand for the accountability, performance assessment of government R&D for public accountability is increasingly gaining attention. To increase the efficiency and effectiveness of national R&D programs, Korean government has actively been evaluating over two hundreds national R&D programs funded by the government every year, through performance evaluation system. In this session, Korea's expertise and experience in R&D performance evaluation, the relation between performance evaluation and national budget and new approach of evaluation system in Korea are being presented and discussed.

Reformation of Research and Development Program Performance Evaluation System

Jong Sung Im,  Ministry of Strategy and Finance,  jsim114@mosf.go.kr

Apart from general finance program, the unique performance evaluation at R&D program has been conducted since 1999. In Feb. 2008, the function of performance evaluation at R&D program transferred from National Science and Technology Council (NSTIC) to Ministry of Strategy and Finance (MOSF) due to amendment of reformation of the government organization law and related laws. Upon this opportunity, MOSF lays and promotes a scheme of reorganizing R&D program performance evaluation system of ‘Practical R&D program performance evaluation’ through ‘Choice and Concentration’. In the meantime, the investment of R&D program has been increased, on the other hand, it has been being pointed out continuously that the performance evaluation system is inefficient and ineffective. From this situation, responding to the R&D investment expansion, MOSF is planning to reform the R&D performance evaluation system in order to support the R&D investment efficiency through simplification of evaluation, reinforcement of department self-evaluation, in-depth analysis and the strong connection between evaluation result and setting budgetary.

Use of Performance Evaluation Information of Research and Development Program in Budgeting Process in Korea

Boo-jong Kil,  Korea Institute of Science and Technology Evaluation and Planning,  kbjok@kistep.re.kr

Su-dong Park,  Korea Institute of Science and Technology Evaluation and Planning,  triznik@kistep.re.kr

In-ja Kim,  Korea Institute of Science and Technology Evaluation and Planning,  kij@kistep.re.kr

The introduction of performance information into the budget processes is an important initiative that is widespread across OECD countries (OECD, 2007). Some people believe that the use of performance information in budgetary decision making can contribute to improving program management and efficiency. Recently, Korean government (especially, Ministry of Science and Technology, Ministry of Planning and Budget) is trying to use performance evaluation information of government R&D program in budgeting process. In this paper, we will divide Korean government's efforts to link performance evaluation and budget of R&D program into 3 phases and analyze this evolutionary process, and give some recommendations to improve this linking system.

Performance Evaluation of National Research and Development in the Field of Environmental Technology With Respect to Science and Technology Classification

Noeon Park,  Korea Institute of Science and Technology Evaluation and Planning,  ecoenv@kistep.re.kr

HyunJung Cho,  Korea Institute of Science and Technology Evaluation and Planning,  chohj@kistep.re.kr

Seung Jun Yoo,  Korea Institute of Science and Technology Evaluation and Planning,  biojun@kistep.re.kr

Yule Shin,  Korea Institute of Science and Technology Evaluation and Planning,  shinyule@kistep.re.kr

Moon Jung Choi,  Korea Institute of Science and Technology Evaluation and Planning,  mjchoi@kistep.re.kr

Sang-youb Lee,  Korea Institute of Science and Technology Evaluation and Planning,  sylee@kistep.re.kr

Various research and development (R&D) classification systems in Korea have been used to plan and evaluate programs efficiently. Two kinds of classification systems (national and institutional) are mostly used to investigate national R&D in the field of environmental technology. Although both systems have been applicable for performance evaluation in environmental technology, they consist of different categories and scope. It is known that it is difficult to classify environmental technology in terms of division, discipline, and subject because a half of classified research area is overlapped. Until now, environmental technology R&D programs in Korea have simply been evaluated by numbers of each output, for example, 'how many papers were published?' In this study, firstly, performances in the R&D are further qualitatively analyzed in terms of two kinds of classification systems. Secondly, new classification system will be provided to be helpful to evaluate environmental technology R&D programs with customized classification system.

Evaluation of Technology Level Using a Dynamic Method on the Critical Technologies in the 2nd Science and Technology Basic Plan of Korea

Soon Cheon Byeon,  Korea Institute of Science and Technology Evaluation and Planning,  sbyeon@kistep.re.kr

Jiyeon Ryu,  Korea Institute of Science and Technology Evaluation and Planning,  jiyeon@kistep.re.kr

Moon Jung Choi,  Korea Institute of Science and Technology Evaluation and Planning,  mjchoi@kistep.re.kr

Pyengmu Bark,  Pukyong National University,  barkpm@pknu.ac.kr

Hyuck Jai Lee,  Korea Institute of Science and Technology Information,  hlee@kisti.re.kr

It is important to evaluate the exact technology level for the establishment of strategic national R&D policy. In the 2nd S&T basic plan of Korea, there are 90 critical technologies which is urgent for Korea to expand its S&T capability. But an implementation plan such as technology roadmap is yet to be prepared. As a first step for this process, Korean government evaluated its critical technologies and set up development strategy for these critical technologies. In this research, a dynamic method was applied to evaluate the exact technology level. In addition to the common evaluation of the relative position of Korea’s technology level compared to the world best level, the possible maximum level of the technology and the rate of technology changes were measured to propose possibilities of catching-up, if any. In this paper the dynamic evaluation process and the result with the optimal R&D strategy are described.

Session Title: Structural Equation Modeling: Applications and Issues

Multipaper Session 425 to be held in Room 103 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Chair(s):

Karen Larwin,  University of Akron,  drklarwin@yahoo.com

Discussant(s):

Kristi Lekies,  The Ohio State University,  lekies.1@osu.edu

Using Path Analysis to Evaluate Multidimensional Models: Developing a Program Success Theory Model for Microenterprise Development Programs Presenter(s): Michele Cranwell-Schmidt,  University of Vermont,  mschmidt@uvm.edu Jane Kolodinsky,  University of Vermont,  jkolodin@uvm.edu Abstract: This presentation will review the methodology and outcome of an evaluation that focused on factors that lead to client success in microenterprise development (MED) programs. Using a path regression analysis of data from the Vermont Micro Business Development Program, this evaluation examined the relationships between client characteristics, program activities, interim outcomes, and longer term impacts to develop a model of MED program success theory. The presentation will highlight these results and include a group discussion on the study findings, implications on program practice and public policy, and advantages and limitations of the evaluation methodology used. Statistics demonstrated excellent model fit to the data. Overall, the model demonstrates that client success in MED is influenced by a variety of interdependent factors, with access to more financial resources being the most prominent factor. The results suggest implications for business training and access to capital as well as methods to evaluate complex models.

Utilization of Structural Equation Modeling Techniques in Real-World Program Evaluation Presenter(s): Greg Welch,  University of Kansas,  gww33@ku.edu Bruce B Frey,  University of Kansas,  bfrey@ku.edu Jessica Oeth,  University of Kansas,  jessoeth@ku.edu Chris Nileksela,  University of Kansas,  chrisn@ku.edu Abstract: Program evaluation can be enhanced by a data collection process which lends itself to statistical analysis via Structural Equation Modeling (SEM) techniques. The attractiveness of this family of statistical methods lies in their ability to address a variety of correlational questions. Unfortunately, many program evaluations cannot take advantage of SEM techniques. This, in part, is because of the lack of information available to evaluators about the advantages of applying SEM techniques, but also because SEM has fairly stringent data requirements. This presentation describes the challenges, advantages and disadvantages of utilizing SEM in an evaluation setting. An ongoing evaluation of a Kansas early childhood school readiness system will be used to provide a framework for this presentation and to illustrate the difficulties one can encounter when either the design of a particular evaluation or unplanned obstacles to data makes the use of SEM easier said than done.

Understanding Consumer Self-Perceptions Regarding Mental Health Recovery: A Structural Equation Modeling Evaluation Presenter(s): Christopher McKinney,  Mental Health Center of Denver,  christopher.mckinney@mhcd.org Kathryn DeRoche,  Mental Health Center of Denver,  kathyrn.deroche@mhcd.org Antonio Olmos,  Mental Health Center of Denver,  antonio.olmos@mhcd.org Abstract: A consumer’s perception of his/her own mental health recovery is considered important in treatment selection, implementation, engagement, and other items related to mental health treatment. Utilizing structural equation modeling (SEM), the current study explores the relationships among five areas regarding consumers’ perceptions of mental health recovery; including hope, symptom interference, social networks, active growth/orientation, and perceived safety. A model of consumers’ perceptions of mental health recovery is presented, along with discussion of the implications regarding mental health treatment. Furthermore, the advantages of using SEM in exploring relationships among complex behaviors are discussed.

Structural Equation Model Evaluating Students’ Self-Perception Regarding Graduate Level Statistics Coursework: Demonstration of a Jackknife Approach for the Purpose of Item Reduction Presenter(s): Karen Larwin,  University of Akron,  drklarwin@yahoo.com Abstract: The intention of the present investigation was to demonstrate the causal link between student’s prior mathematics preparation and Statistics Self-Perception, using structural equation modeling techniques. It is theorized that many graduate students continue to struggle with required statistics coursework due to poor preparation/minimal prerequisites in mathematics, even after many statistics instructors have shifted to more conceptual presentations of statistical concepts. A third-order factor structure of Statistics Self-Perception demonstrates the link between prior mathematics experience and students statistics-related attitudes/anxieties/self-efficacy. The number of observed items (k = 65) in this original third-order factor model was reduced (k = 40) arriving at a model that is parsimonious, while measurement and structurally invariant relative to the original model. Item reduction has been attempted in previous research using SEM, however these studies do not present clear evidence that model changes resulted in models that were measurement invariant while maintaining the integrity of the structural model.

Session Title: TIG Business Meeting and Think Tank: The Role of the Nonprofit and Foundations TIG as a Forum for Dialogue Among the Membership and AEA as well as Nonprofits and Funders

Business Meeting Session 426 to be held in Room 105 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

TIG Leader(s):

Lester Baxter,  Pew Charitable Trusts,  lbaxter@pewtrusts.org

Lorna Escoffery,  Escoffery Consulting Collaborative Inc,  lescoffery@stanfordalumni.org

John B Nash,  Open Eye Group,  john@openeyegroup.com

Teresa R Behrens,  WK Kellogg Foundation,  tbehrens@wkkf.org

Presenter(s):

Stanley Capela,  HeartShare Human Services,  stan.capela@heartshare.org

Discussant(s):

Saumitra SenGupta,  APS Healthcare Inc,  ssengupta@apshealthcare.com

Charles Gasper,  Missouri Foundation for Health,  cgasper@mffh.org

Abstract: The TIG plays an important role within the American Evaluation Association although it may not always be clear to its membership. The purpose of this session is to start a conversation on the TIGs' role within the AEA. Specifically, we will explore the following questions. Is the TIG a forum for dialogue among its membership and the AEA? Does it have a role in bringing funders and nonprofits together on how to meet each others needs as well as how to make effective use of program evaluation among nonprofits and funders? During the session we will have the current membership committee chair share key findings from a recently conducted TIG study followed by a funder and nonprofit perspective on how to make more effective use of the TIG to assist its membership.

Session Title: The Impact of Quality Assurance and Evaluation; Stakeholder Perceptions

Multipaper Session 427 to be held in Room 107 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Evaluation Use TIG

Chair(s):

Edward McLain,  University of Alaska Anchorage,  ed@uaa.alaska.edu

Can Participatory Impact Pathway Analysis be Used by Evaluators to Engage Stakeholders While Increasing Cultural Competencies of Evaluators? Presenter(s): Alice Young-Singleton,  University of Southern California,  youngsin@usc.edu Abstract: One prevalent foundational principle in the development of logic models (whether to facilitate program design, program implementation, or to report program outcomes) is the collaborative and participatory process required when seeking to visually depict a program’s theory (Kellogg Foundation, 1998; Gasper, 2000; (Mayeske, 2002). A large body of literature on logic models purports that evaluator’s involvement of key stakeholders to help articulate programmatic assumptions and priorities impacts logic model utility, ownership, and ultimately programmatic success (Mayeske, 2001; Cooksy, 2001; Renger, 2006; Kirkpatrick, 2001; Greene, 2001; Hopson, 2003). However, logic models are often designed by the evaluator with little input and interrogation by principle stakeholders thereby prompting the application of the tool to often be only associated with: (1) the evaluation phase of a program and (2) the evaluator (Coffman, 1999). Yet, evaluations conducted using participatory impact pathway analysis, a derivation of the logic model, suggests that evaluators may employ the logic model to not only engage stakeholders in a participatory process that ascertains a program’s theory but also to increase their own cultural competency for evaluation practice. Hence, this paper will draw on literature and evaluative studies utilizing participatory impact pathway analysis to examine how it has been used to engage stakeholders while increasing cultural competencies of evaluators in order to inform evaluation approach and practice. Through this process, this paper also seeks to focus/refocus the attention and purpose of evaluation systems on utility, employing methods that are responsive to the diversity of organizations effectively addressing “the needs of the full range of targeted populations.”

High Quality Program Evaluation with Unrealistic Outcome Expectations Presenter(s): Courtney Brown,  Indiana University,  coubrown@indiana.edu Mindy Hightower King,  Indiana University,  minking@indiana.edu Marcey Moss,  Indiana University,  marmoss@indiana.edu Abstract: Evaluators are increasingly faced with evaluating programs with unrealistic but expected outcomes. How and what evaluators evaluate is directly related to the funding agency’s expectations, whether these are realistic in the time allotted or not. This is true of federally and privately funded evaluations, especially those focused on student achievement. This emphasis on accountability is generally tied to large-scale education reform efforts as well as state and federal legislation. However, it is often unrealistic to evaluate real changes in achievement in a three-year grant. This paper provides practical, realistic solutions to the following challenges: (1) short-term evaluations with expected outcomes more appropriate for long-term projects and (2) projects with program outcomes already determined and expected (i.e., GPRA measures). Solutions to these challenges include: building a logic model prior to program implementation; synthesizing prior research and evaluations; and looking for realistic short- or mid-term outcomes related to the expected long-term outcome.

How Danish Teachers Experience the Impact of Quality Insurance and Evaluation Initiatives on their Teaching Practices Presenter(s): Carsten- Stroembaek Pedersen,  University of Southern Denmark,  csp@sam.sdu.dk Abstract: The results from a nationally representative survey of teachers in public mandatory schools are presented in this paper. This work explores how Danish teachers experience the impact of different quality assurance and evaluation (QAE) initiatives on the quality of their teaching and on their autonomy. In Demark, the public debate has mostly concentrated on the negative impact of QAE on teacher practices such as loss of autonomy and effects like teaching-to-the-test, tunnel vision, etc. Despite this, we still know very little about how teachers in general view the impact of QAE initiatives on their teaching practices. This survey provides us with an extensive data set which allows for a general description of the impact of QAE as perceived by teachers. The survey is part of an international research study of the impact of QAE processes on improving the quality of education in Denmark, England, Finland, Scotland and Sweden.

Session Title: Methods and Models in Evaluating Educational Technology

Multipaper Session 428 to be held in Room 109 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Distance Ed. & Other Educational Technologies TIG

Chair(s):

Saul Rockman,  Rockman et al,  saul@rockman.com

Improving Websites with Usability Testing Presenter(s): Michael Lambur,  Virginia Polytechnic Institute and State University,  lamburmt@vt.edu Abstract: Usability testing is a means for determining how well people use something, a website in this case, for it intended purpose. This evaluative process involves observing people using a website in as realistic a situation as possible to discover errors and areas of improvement. Conducting a usability test of a website can be invaluable in improving its functionality and ultimately its purpose for users. This presentation will walk participants through the process of usability testing and will focus on: 1) deciding what needs to be tested, 2) determining how many users to involve in the test, 3) identifying tasks that will be performed by the user in the test, 4) developing questions that will be asked of the users before and/or after the test, 5) deciding whether to use unobtrusive or obtrusive observation, and, 6) preparing observer guidelines. Experience gained from website usability testing with the eXtension initiative (http://www.extension.org) will also be shared.

Some Reasons for Incorporating Mixed-Methods Designs when Evaluating the Efficacy of Educational Learning Tools Presenter(s): Dane Christian,  Washington State University,  danechristian@mail.wsu.edu Michael Trevisan,  Washington State University,  trevisan@wsu.edu Angela Oki,  Washington State University,  info@currentconceptions.com Phil Senger,  Current Conceptions Inc,  info@currentconceptions.com Abstract: An evaluation was undertaken to test the efficacy of a 3D animated instructional video on reproductive physiology. A four-factor one-way analysis of variance (ANOVA) between groups experimental design was employed. Six universities from across the country participated in the study with subjects being randomly assigned to control or experimental conditions. The results of this initial study were favorable with an effect size greater than one standard deviation unit. This paper highlights various elements of the design used in the study. After accounting for potential areas of weakness in the design, a rationale for the use of qualitative methods in subsequent studies is provided. Specifically, during-treatment observations, and post-treatment interviews and focus groups are suggested. The benefits of this paper add toward the growing belief in the evaluation and social research communities that mixed-methods designs, when correctly utilized, can add valuable information to a study’s purpose.

Evaluation Models for Educational Technology Projects Presenter(s): Michael Coe,  Northwest Regional Educational Laboratory,  coem@nwrel.org Abstract: This presentation reports findings from an NSF-funded project intended to develop improved models of evaluation research for educational uses of electronic technologies. The premise is that the evaluation of educational technology applications is hampered by oversimplified, underspecified models of both the project designs and the project evaluations. Important aspects of the project designs are often left out of the evaluation designs, and the relationships between project components may be misrepresented in the evaluation designs. These issues can lead to unproductive evaluation questions and research methods. Many of these problems could be solved, at least in part, by applying program theory and basic causal modeling concepts. The presentation will include the rationale for the project, brief examples of work we have done over the past few years, and findings from the current study.

Evaluating Educational Technology: Approaches to Collecting Meaningful Data Presenter(s): Shani Reid,  Macro International Inc,  shani.a.reid@macrointernational.com Abstract: In 2005 Macro International embarked into fairly uncharted territory when they were contracted to evaluate an online learning game being developed by Maryland Public Television. The game, which is being developed with funds received from a federal Star schools grant, aims to enhance the pre-algebra and literacy skills of middle school students. The final game will consist of 9 puzzles, each with 3 levels of difficulty. During the development phase, Macro’s responsibility has been to provide stakeholder feedback on the game to project staff during various stages of development. How do you collect formative evaluation data on educational gaming technology? In fact, how do you collect quality feedback from a target audience who are between the ages of 11 and 13? In this session we will discuss the various methodologies employed (including the use of embedded game features) to obtain valid and reliable data from students and educators.

Session Title: Measuring Cultural Issues in Multiethnic Evaluations

Multipaper Session 429 to be held in Room 111 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Multiethnic Issues in Evaluation TIG

Chair(s):

Nicole Bowman,  Performance Consulting,  nbowman@nbowmanconsulting.com

Discussant(s):

Emiel Owens,  Texas Southern University,  owensew@tsu.edu

Measuring Racism: The Brooks Equity Typology (BET) Presenter(s): Pauline E Brooks,  Brooks Cross Cultural Evaluation/Research Consulting,  pbrooks_3@hotmail.com Abstract: Racism makes the world a more dangerous and unsettled place. It adds to the violation of the human spirit, creates and operates on falsity, misguides and weakens social harmony and stability. In nations with a long history of unequal treatment based on race, like the United States, racism continues to contribute to the gapping racial disparities that persist in virtually every sector of contemporary life: in education, health, economics, the judicial system, etc. This paper presents a typology of equity that evaluators/researchers can use to more clearly identify, frame and measure multiple avenues of racism’s influence in their own work and in the everyday contexts of societal programs, policies, etc. The typology can be applied in many sectors of society, is interdisciplinary, and can be modified for use in investigating other forms of inequities (e.g., inequities based on gender, socioeconomic class) and intersections. Measurement examples and research strategies are presented.

Operationalizing Culturally Competent Evaluation Presenter(s): Arthur Hernandez,  University of Texas San Antonio,  art.hernandez@utsa.edu Abstract: The notion of Culturally Competent Evaluation has been around for some time and is currently an important aspect of professional practice as described in the AEA professional standards. Although there is clear discussion of the need for and implications of such practice in the literature, there seems to be very little in the way of particular practice, instrumentation or activities designed to examine or provide evidence of the cultural relevance and/or appropriateness of any given evaluative activity or approach. This presentation will provide a general conceptual overview and suggestions for a “Cultural Competence Rubric” a report of a preliminary examination of its implementation and provide an opportunity for those interested in such practice to discuss approaches, experience and outcomes.

The Impact of Racial Identity Development on African American Female Academic Achievement Presenter(s): Charles Glass,  Texas Southern University,  crglass1@juno.com Abstract: The purpose of this study will be to examine the effect of racial identity/consciousness (RIC) on the academic achievement of African American female college freshmen. This causal-comparative study is intended to provide research based information concerning the impact of racial identity/consciousness development on the academic achievement of African American female freshman students. Research on female identity has focused on White American women at the exclusion of African American women (Hoffman, 2006). The racial identity construct for this overlooked ethnic group is derived from the racial self-identification or definition and racial self-consciousness or acceptance. According to Hoffman (2006), racial self-consciousness affects a person’s confidence as that individual evaluates and accepts who she is.

What’s in the Cultural Competence Toolbox? Presenter(s): Elise Arruda,  Brown University,  elise_arruda@brown.edu James Griffith,  Claremont Graduate University,  james.griffith@cgu.edu Abstract: Discussion regarding culture in evaluation takes many forms. This paper aims to focus the culture dialogue on the needs of evaluators in the field. Working evaluators need: (1) research methods that can incorporate culture without making culture the focus of the study, and (2) procedures to determine whether culture has been given appropriate attention in an evaluation study. Evaluation clients often dictate many of the evaluation parameters (i.e., evaluand, questions, participants, etc.) leaving little control to the evaluator. Nonetheless, evaluators often have some control over the approach and methods used for the evaluation. The question remains: Are there methods, approaches, or procedures that evaluators can add to their repertoire of skills to incorporate culture without unnecessarily shifting the focus of the evaluation? This paper explores research methods from a range of disciplines to assemble a set of approaches with the potential to enhance evaluators’ cultural competence or responsiveness.

Re-Envisioning, Engaging and Advancing Diversity in the Evaluation Community: Premises and Possibilities Presenter(s): Jeehae Ahn,  University of Illinois Urbana-Champaign,  jahn1@uiuc.edu Abstract: This paper explores ways of engaging and increasing diversity in the evaluation field. First, it works to understand diversity not so much in terms of racial/ethnic representation as the wonderfully varying ways of knowing, understanding and valuing those racially/ethnically diverse individuals bring into the field. Framing diversity and its meaning in this way, the paper then envisions ways to create a dialogic space within the evaluation community in order to invite interested individuals, especially those from racially/ethnically underrepresented groups, to share their unique perspectives, aspirations, needs and challenges as individual evaluators (in the making), while at the same time engaging them in broader and ever-evolving conversations about such fundamental issues as who we are, what we (should) value and aspire to accomplish and advance in our work collectively as a profession, as a way to encourage their membership, increase their participation and/or support their advancement in the field.

Session Title: Drawing Logic Models Using DoView - Logic Model and Evaluation Planning Software

Demonstration Session 430 to be held in Room 113 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM

Sponsored by the Integrating Technology Into Evaluation

Presenter(s):

Paul W Duignan,  Parker Duignan Consulting,  paul@parkerduignan.com

Abstract: DoView is a new type of logic modeling software designed for evaluators to draw and amend logics in real-time during evaluation stakeholder meetings. The software creates 'models' of logics rather than just creating 'drawings' of them. Participants will be shown how they can use this approach to build, structure and work with large and small logic models. The way these models can be used for providing a rich description of a program, for documenting a model, and for identifying which links are supported by evidence will be discussed. In addition, ways of using the models for planning and implementing evaluations as a 'living evaluation plan' by inserting indicators and evaluation questions onto the logics will be demonstrated. Lastly, ways that HTML versions of the models can be used to front-end web-based evidence-based practice evidential summaries will be briefly demonstrated.

Return to Evaluation 2008