Search Results for All Sessions

Return to search form

Session Title: Strategies for Reconstructing Baseline Data

Think Tank Session 248 to be held in Capitol Ballroom Section 1 on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the International and Cross-cultural Evaluation TIG

Presenter(s):

Michael Bamberger, Independent Consultant, jmichaelbamberger@gmail.com

Discussant(s):

Jim Rugh, Independent Consultant, jimrugh@mindspring.com

Samuel Bickel, United Nations Children's Fund, sbickel@unicef.org

Abstract: Evaluators are frequently asked to assess the impacts of projects that are already underway before the evaluation is commissioned. Often no baseline data has been collected, or available baseline data is inadequate. The purpose of the Think Tank is to exchange experiences on approaches that participants have used in their own evaluations, both overseas and in North America to estimate the conditions of the project and/or comparison groups at the time of project launch. The workshop facilitators will draw on their extensive experience in conducting and teaching impact evaluation overseas to review commonly used quantitative, qualitative and mixed-method approaches for reconstructing baseline data and to present a framework for assessing the strengths and weaknesses of different approaches. Participants will break into groups to share and assess the different methods they have used and will then reconvene to discuss guidelines for strengthening estimates of baseline conditions.

Session Title: Expanding the Independent Evaluator's Tool Box: First Forays Into Using Cost/Economic Data, Social Network Analysis, Reflective Assessment, and GIS Data

Multipaper Session 250 to be held in Capitol Ballroom Section 3 on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Independent Consulting TIG

Chair(s):

Sally Bond, The Program Evaluation Group LLC, usbond@mindspring.com

Abstract: As evaluators, we are always looking for ways to strengthen our findings of the 'value' of a program. In some cases, even where budgets may limit us, there may be ways to incorporate aspects of cutting-edge analyses that provide a better context for program findings, without conducting full-scale studies. In this session, four evaluators will share ideas about including aspects of the following 1) cost/economic analysis, 2) social network analysis, 3) reflective assessment, and 4) GIS data. The presenters will provide information about the 'full-blown' method, describe the modifications they made, and share the impact that inclusion of such analyses had on their understanding of the program outcomes and impact as well as on the funder's understanding of his or her program. The presentation will also provide time for the audience to ask questions or brainstorm other ways in which such analyses can be incorporated.

Making Use of Economic Data: Cost Study of a Teacher Induction Program

Amy Germuth, Compass Consulting Group LLC, agermuth@mindspring.com

As part of a study of a district-level teacher induction program, the evaluator gathered economic data from the program director regarding the program budget and costs. Additionally the evaluator sought out comparison economic data in order to better understand these costs in context. Without conducting a true cost-benefit analysis, this analysis still allowed for a more refined examination of the costs of implementing the program and how these costs compared with costs associated with teacher attrition, transfer, and recruitment. This approach provided a clearer picture of costs by category and the costs required for successful retention and placement of lateral entry and new teachers. It also allowed one to estimate the cost offsets associated with such an intervention. This analysis provides information that is useful for assessing the value and importance of program outcomes relative to the cost of program implementation.

Using Concepts of Social Network Analysis: A Study of Collaboration in a Media and Technology Initiative

Sally Bond, The Program Evaluation Group LLC, usbond@mindspring.com

From 2003-2006, the state of North Carolina funded a number of schools to implement its IMPACT model for Media and Technology Programs. Collaboration between teachers, the school library media coordinator, and a technology facilitator is key to the success of the model. In order to assess change in collaboration at one of these schools, the evaluator developed a pre/post Collaboration Wheel exercise for participating staff members. With participating school personnel identified as a social network, the Collaboration Wheel was used to capture connections between actors in the network (who interacts with whom, and how often) as well as the nature of interactions between actors. Through the use of this method, the evaluator was able to better understand how program participants defined collaboration. Program participants were then able to consider ways to deepen their collaboration beyond simply sharing information or providing collegial support to one another.

Reflective Assessment: A Streamlined Evaluation Technique

Dawn Hanson Smart, Clegg and Associates Inc, dsmart@cleggassociates.com

Reflective Assessment is a structured, streamlined form of evaluation which captures data on program operations and achievements and generates information on improvements to service delivery. Widely used in education, the evaluator has applied the method in a variety of housing-related evaluations. RA provides an opportunity for multiple stakeholders within an organization to carefully think about a program, assess progress, document accomplishments, and consider the choices ahead before continuing with implementation. It is conducted in 'real time' during implementation. It is the immediacy of the learning that makes RA valuable ' changes are identified and incorporated more quickly than is often true in other evaluation approaches. Involving multiple stakeholders ensures that various points of view are considered. With clients, staff, management, and board members at the table, there is greater understanding of what is needed and why, and greater buy-in to the recommendations developed.

Improving Findings through Mapping: First Steps in Using Geographic Information Systems

Sarah Heinemeier, Compass Consulting Group LLC, sarahhei@mindspring.com

Geographic Information Systems (GIS) analyses are popular for the visual expression of data, especially when that data focuses on the spatial distribution of clients, landmarks, programs, etc. There is an ever-growing field of study behind the use of this approach with specialized training and software applications dedicated to the processing of the data and production of maps and visual exhibits. This presentation will present several techniques for creating relatively simple maps that can be incorporated into evaluation reports and presentations, as was done within the context of an early childhood systems evaluation. These maps may prove to be useful heuristics in presenting and explaining patterns in data, if not triggers for the identification of patterns or trends that were otherwise unnoticed.

Session Title: Perspectives on Using Stakeholder Judgment in Evaluation

Multipaper Session 251 to be held in Capitol Ballroom Section 4 on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the AEA Conference Committee

Chair(s):

Nicole Vicinanza, JBS International, nvicinanza@jbsinternational.com

Evaluating Training Efficacy

Presenter(s):

Yvonne Kellar-Guenther, University of Colorado Denver, yvonne.kellar-guenther@uchsc.edu

Abstract: Education through training is only useful if the student is able to transfer what they learn to their job. Kontoghiorghes (2004) stated that training investments often fail because it is difficult to transfer what is learned in the training environment to the workplace environment (Subedi, 2006). This transfer of learning between the two environments is called the training transfer. In my quest to evaluate an on-going training, I collected data after the trainees had been back at their jobs for awhile; an approach is supported Santos and Stuart (2006). While there have been some studies that have used a similar approach, the follow-up data relies on student self-assessments of their own learning. To test the validity of this approach, I have collected objective data that was independently scored to assess change and compared it to the subjective measures to see if there is a correlation between the two scores.

Estimating Program Impact Through Judgment: A Simple But Bad Idea?

Presenter(s):

Tony Lam, University of Toronto, tlam@oise.utorfonto.ca

Abstract: Estimating program impact is complicated, multifaceted, time consuming, labor intensive, and costly. To overcome these technical and logistic difficulties, evaluators, especially training evaluators, have resorted to relying on judgments to derive program impact estimates. After collecting the outcome data, evaluators ask various stakeholders to determine the degree to which the observed outcomes are attributable to the program, and sometimes to also report confidence of their estimates. Apparently, using self-reporting to estimate program impact is more efficient than deriving such estimates empirically through experimental and quasi-experimental procedures. Unfortunately and expectedly, the judgment-based impact estimates are susceptible to both intentional (e.g., self-serving bias) and unintentional biases (e.g., recall errors). In this paper, I will review the literature, describe and critique the process of using judgments to determine program impact, present the various sources of biases, and propose some strategies to overcome these biases and to incorporate self-reporting data in program impact assessments.

Using Self Assessment Data for Program Development Dialogues: Lessons Learned from Assets Coming Together (ACT) for Youth

Presenter(s):

Amanda Purington, Cornell University, ald17@cornell.edu

Jennifer Tiffany, Cornell University, jst5@cornell.edu

Jane Powers, Cornell University, jlp5@cornell.edu

Abstract: Self-assessments aid in gathering data, engaging participants, and fostering discussion about improvements in practices and policies. We report on a New York State project that conducts self-assessments to promote program, organizational and coalition development, particularly around integrating Positive Youth Development practices and principles. Our presentation will focus on the process of developing and administering self-assessments, as well as data analysis and disseminating findings. We will also discuss challenges encountered including: reporting findings and data in ways that are easily grasped by the diverse groups doing self-assessments, strategies for using self-assessment findings more effectively to inform practice and policy, and using self-assessments from multiple groups to inform �big picture� analysis of systems and initiatives.

Session Title: Ethical Evaluation From Transformative, Transnational, and Public Interest Perspectives: An AEA Ethics Committee Panel Discussion

Panel Session 252 to be held in Capitol Ballroom Section 5 on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the AEA Conference Committee

Chair(s):

Valerie Caracelli, United States Government Accountability Office, caracelliv@gao.gov

Discussant(s):

Scott Rosas, Concept Systems Inc, srosas@conceptsystems.com

Hazel Symonette, University of Wisconsin Madison, symonette@bascom.wisc.edu

Abstract: The expanded view of the social justice functions of evaluation within rapidly shifting cultural and geographical settings has lead to a corresponding need to consider the ethics of evaluation practice at individual and public levels. In this context, critical questions are raised regarding the applicability and appropriateness of currently accepted frameworks for ethical decision-making. During this session, members of the AEA's Ethics Committee and three evaluation ethics scholars will discuss present and future challenges for ethical evaluation practice. The panel will focus on ethical considerations and decision-making related to evaluation within transformative, transnational, and public interest perspectives. Each presenter will raise important issues for evaluation practice, professional expectations, and evaluation's role in larger society. The structure for this session is brief presentations followed by comments from two discussants and closing with an interactive dialogue with the audience and panel members.

Program Evaluation Ethics from a Transformative Stance

Donna Mertens, Gallaudet University, donna.mertens@gallaudet.edu

Heidi Holmes, Gallaudet University, heidi.holmes@gallaudet.edu

Raychelle Harris, Gallaudet University, raychelle.harris@gallaudet.edu

Program evaluators work in culturally complex contexts with challenging agendas to provide information to stakeholders who are situated in positions associated more or less unearned privilege. The ethical assumptions associated with the transformative paradigm prioritize the furtherance of social justice and human rights. This ethical ground leads to questions about the ethical practice of evaluation, given the potential inequities that exist when the full range of stakeholders are considered. What are the ethical implications of stakeholder involvement if the goal of the program and the evaluation is increased social justice? How do issues of power differential complicate ethical decision making? To what extent should evaluators be responsible for furthering a social justice agenda? Examples of ethical practice will be drawn from communities such as the deaf community, African peoples, the lesbian, gay, transsexual, bisexual and queer communities, and indigenous communities that shed light on ethical frameworks to guide practice in transformative program evaluations.

Program Evaluation Ethics in Transnational Spaces

Robert Stake, University of Illinois, stake@uiuc.edu

Fazal Rizvi, University of Illinois, frizvi@express.cites.uiuc.edu

Here we reflect upon the difficulties of pressing for ethical conduct in evaluating social programs in transnational spaces. First we separate transnational space from globalization, highlighting a range of problems, both theoretical and practical. We then examine the question of whether the traditional ways of addressing ethical issues are adequate still, in problems, for example, such as privacy. We conclude that tomorrow's ethical problems of tomorrow will be, at bottom, little different from yesterday's, but that they come packaged (situated) in unexpected ways. We consider the problems of monitoring ethical conduct through training, codes of ethics and institutional reviews, arguing that these are insufficient, as are traditional political, economic, cultural and geographical constraints, in what Wittel (2000) calls, a 'crisis in objectification.' We conclude that ultimately ethical responsibility will reside with the evaluators themselves, individually and interactively.

The Fifth Guiding Principle: Beacon, Banality, or Pandora's Box?

Michael Morris, University of New Haven, mmorris@newhaven.edu

Although all of AEA's Guiding Principles for Evaluators present evaluation professionals with the challenge of how to translate general guidelines into concrete practice, the principle of Responsibilities for General and Public Welfare can be especially demanding, given that it implicitly asks: What is the 'good society'? Indeed, it is not surprising that the authors of the original Guiding Principles (1995) observed that 'the key debates in the task force concerned evaluation and the public interest.' In this paper I examine the ethical implications of the fifth Guiding Principle for our profession. From an ethical perspective, is there more -- or less -- to this principle than meets the eye? What contribution can a principle focusing on the 'general and public welfare' make to AEA's attempts to influence evaluation policy at various levels? To what extent might such a principle represent a 'cautionary tale' for these attempts?

Session Title: Integrating Evaluation Education With Theory: A Practice-Based Approach

Multipaper Session 253 to be held in Capitol Ballroom Section 6 on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Teaching of Evaluation TIG

Chair(s):

Sherri Lauver, University of Rochester, slauver@warner.rochester.edu

Abstract: Working evaluators must have knowledge of the evolving theoretical knowledge of the field and practical knowledge of appropriate methodologies, ultimately relating it all to their real-world evaluation practice (Lee, Wallace, & Alkin, 2007). Yet coursework often involves mostly direct instruction of evaluation concepts and theory (Alkin & Christie, 2002; Patton & Patrizi, 2005). This session will focus on the benefits of participating in real-world evaluation activity first, followed by evaluation theory coursework. First, the instructors will present the strengths and challenges of teaching graduate students using a client-requested evaluation. Second, students will share their experiences as novices who must 'learn by doing' in a local evaluation context. The presentation will focus on the positive learning experience of addressing unexpected issues related to last minute obstacles and stakeholder concerns. Students will reflect on their personal experiences and discuss the merits of teaching evaluation theory after an introductory program evaluation course.

The Benefits of Teaching Program Evaluation Through Real-World Experiences: The Instructor Perspective

Sherri Lauver, University of Rochester, slauver@warner.rochester.edu

Tia Neely, Pacific Research and Evaluation LLC, tia@pacific-research.org

Many evaluation programs teach evaluation via direct instruction. We will argue that experiential learning activities are an effective and appropriate means to help students understand the real world of evaluation practice. Evaluations do not occur in a rigidly controlled environment, and scaffolded, authentic experiential learning activities offer students the opportunity to address the common hurdles we face every day - convincing stakeholders to allow to use suitable methodological designs or types of data collection, working on tight timelines, presenting results that may be controversial, and working with difficult partners. By involving students in real-world evaluation, student interest and engagement remains high. Yet real-world evaluation is demanding on an instructor's time and requires the evaluation client to be especially flexible and understanding of the novice evaluators' needs. We will discuss how to maximize the use of a classroom of student evaluators and minimize the burdens on the instructor and the evaluation client.

The Benefits of Teaching Program Evaluation Through Authentic Experiences: The Student Perspective

Gail Evans, University of Rochester, gaile810@gmail.com

Lisa Hiley, University of Rochester, lmelsemo@orion.naz.edu

Students gain a deeper understanding of evaluation research by participating in an authentic evaluation, rather than simulated scenarios. This method of instruction allows students to construct knowledge from contextualized experiences in order to gain deeper understanding. The unique nature of this approach to learning evaluation requires consideration for feasibility, which includes time pressure and previous knowledge, as well as collaboration, which includes the interactive nature of authentic evaluation. Authentic evaluation situates student learning in a real-life context where students must address unexpected issues related to human error, last minute obstacles, stakeholder concerns and emotional responses. This method for teaching evaluation allows students experience situations that one could not simulate in a case-study approach (e.g., stakeholder withholding information from students). The skills to navigate these situations are typically not taught, but are only learned through the experience. Dealing with these factors makes the authentic evaluation experience worthwhile from the student perspective.

'But is it Authentic?': Evaluation Education Praxis and the Mythic Split Between Theory and Practice

Alfred Vitale, University of Rochester, avitale@warner.rochester.edu

Amy Sherma Steed, University of Rochester, asteed@warner.rochester.edu

For students of Program Evaluation, commencing coursework with immersion in an actual evaluation provides an anchor point for subsequent study of Evaluation theories and helps avoid the conflation of evaluation theory as a toolkit. Theory without practice may constrain the possibility for students to gain their own understandings of evaluation or develop novel and innovative solutions to evaluation issues encountered in the field. The value of a theory-first approach is also questioned by the fact that theories of evaluation are often highly contested within the field itself, and without field experience students may have a more difficult time assessing the merits of different evaluation theories. In addition, a theory-first approach presupposes homogeneity in student learning styles and presumes theories will be retained weeks or even months after introduced. A practice-first approach gives substance to theories studied introduced along the way and studied in depth later, providing critical opportunities for reflection.

Session Title: Peeling Back the Layers From Policy to Practice: The Impact of National Measures on Local Substance Abuse Consumers

Panel Session 254 to be held in Capitol Ballroom Section 7 on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG

Chair(s):

Robert Hubbard, National Development and Research Institutes Inc, hubbard@ndri-nc.org

Abstract: To respond to accountability demands and enhance the effectiveness of Substance Abuse and Mental Health Services Administration (SAMHSA) funded programs, SAMHSA implemented a national, performance-based, outcome-driven measurement system, the National Outcomes Measures (NOMs) in 2007. These federal measures establish the domains and frame the outcomes that are assessed for evaluating substance abuse consumers' progress and program performance. This panel will discuss the multi-layered implications of implementing a national policy in terms of the often distinct information needs and data requirements at each level of the service system: federal, state, and local. Using North Carolina as a case study, this panel will explore the challenges of moving NOMS from policy to practice at a local and state level.

Exploring the Core: The Ongoing Challenges of Developing National Measures for Substance Abuse Treatment Program Outcomes

Deena Murphy, National Development and Research Institutes, Inc, murphy@ndri-nc.org

Robert Hubbard, National Development and Research Institutes Inc, hubbard@ndri-nc.org

Robert Hubbard is Director of the Institute for Community-Based Research of the National Development and Research Institutes, Inc. (NDRI-NC) and over the last 30 years has played a key role in the research, development, and utilization of systems and tools to improve outcomes and enhance program performance as part of a total quality improvement approach. Deena Murphy is a Principal Research Analyst at NDRI-NC and her research focuses on understanding how to improve local and state use of "top down" evaluation systems. Based on this research and Robert Hubbard's extensive experience in the substance abuse field, the ongoing challenges related to developing, implementing, and using national outcomes measures will be discussed.

Framing the Layers of Policy Around Substance Abuse Outcome Measurement

Marge Cawley, National Development and Research Institutes Inc, cawley@ndri-nc.org

Marge Cawley will provide a structural framework within which outcomes evaluation is pursued for substance abuse consumers in the state of North Carolina as guided by federal policy. She will explore the elements that impinge on the success of data collection and its relevance to the stakeholders at each level. These will include addressing the usefulness of the data to all levels of participants, the methods of collecting the data, the measurement options as well as the design and instrument review. This presentation will explore the process of data interpretation for all levels of stakeholders.

Revealing the Contradictory Goals in Evaluating Outcomes Within the Layers of the Substance Abuse Treatment System in North Carolina

Mindy McNeely, North Carolina State University, mindy_mcneely@ncsu.edu

Mindy McNeely will focus upon the data and evaluation needs pertinent at the federal, state, provider, and consumer levels of the North Carolina substance abuse service system. This presentation will concentrate on the often confounding process of providing outcomes that are useful and yet serve many purposes. Data gathered using North Carolina's Treatment Outcomes and Program Performance System' (NC-TOPPS) outcomes tool will be used to examine consumer outcomes and stakeholders needs. The challenge of translating the relevance of outcomes dictated from a higher level of bureaucracy is often a barrier to the broad utilization of outcomes data for evaluating program success and consumer progress. The elements that influence this process in North Carolina will be delineated here.

Roundtable: Obtaining Parsimonious Data: Questions to Avoid Too Much Data

Roundtable Presentation 255 to be held in the Limestone Boardroom on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the AEA Conference Committee

Presenter(s):

Michael Burke, RTI International, mburke@rti.org

Abstract: How much data is too much data? In this roundtable we will discuss questions and approaches that can be used to create an evaluation decision tree to obtain more parsimonious data. While the nature of each evaluation determines whether evaluation questions, stakeholder concerns, planning needs, or some other issue takes precedence, asking a series of questions can help evaluation planners and managers determine what data is essential and what data is nice but not really needed. Sharing this information with clients and staff helps to build common consensus as to where resources should be allocated.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Conducting Cost-Efficient Evaluations That Produce Credible Results

Roundtable Presentation 256 to be held in the Sandstone Boardroom on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Costs, Effectiveness, Benefits, and Economics TIG

Presenter(s):

Nadini Persaud, University of the West Indies, npersaud07@yahoo.com or npersaud@uwichill.edu.bb

Dennis W Rudy, Lakehouse Evaluation Inc, drudy@lakehouse.org

Abstract: Trying to conduct a rigorous evaluation amidst budget constraints is a common problem faced by most evaluators. Confronted with this challenge, many evaluators ignore cost analysis completely�a vital component of professional evaluation. This presentation will discuss how a shoestring budget can be used to produce a rigorous evaluation. First, Nadini Persaud will introduce a cost analysis checklist methodology that can be used by the evaluator to conduct a cost study when resources are not sufficient to hire a cost analysis expert. Next, Dennis Rudy will share with attendees successful strategies that he has utilized amidst budget constraints. The facilitators will then pose questions to the audience to generate dialogue so that attendees and the facilitators can learn from each other about successful strategies and personal experiences. This presentation is important to the evaluation profession because budgets determine the scope of an evaluation.

Roundtable Rotation II: Evaluating Impact of Regional Policy: How do we Handle a Complex Evaluation Task?

Roundtable Presentation 256 to be held in the Sandstone Boardroom on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Costs, Effectiveness, Benefits, and Economics TIG

Presenter(s):

H�kan Gadd, Institute for Swedish Growth Policy Studies, hakan.gadd@itps.se

Jonas M�nsson, Institute for Swedish Growth Policy Studies, jonas.mansson@itps.se

Abstract: Regional policy is a area, where policy measures can target regions, firms and/ or individuals and often have an implicit built in expectation of spillover on other areas of policy. For example it is common that outcome variables for government support of firms is expected to both have an impact on unemployment (individual level) and regional economic growth (regional level). This creates a situation where different authorities running different types of measures, with partly the same goals. Therefore evaluators of regional policy have to deal with complex social systems, context factors, and several levels and systems. The purpose of the roundtable is to specifically discuss approaches and methods to identify strategies for evaluating the impact of regional policy and in general evaluating complex systems. As an introduction the Swedish institute for growth policy studies will present key points/problems from an ongoing study on goal attainment and impact of regional policy.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Evaluating Online Learning: Focus on Usability, Instructional Design and Adult Learning Theory

Roundtable Presentation 257 to be held in the Marble Boardroom on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Distance Ed. & Other Educational Technologies TIG

Presenter(s):

Julie Pekowski, Hanssen Consulting LLC, juliep@hanssenconsulting.com

Carl Hanssen, Hanssen Consulting LLC, carlh@hanssenconsulting.com

Abstract: This work is set in the context of the five-year evaluation of the Life Sciences for a Global Community (LSGC) teacher institute funded by the National Science Foundation Math Science Partnership program. LSGC consists of two three-week summer sessions and six online courses administered during the academic years following the summer sessions. An external review of the online courses and tools from the perspective of Web-based training usability and adherence to adult learning theory principles is a key component of the overall program evaluation strategy. The evaluation was conducted virtually and the evaluation team was provided access to these courses and tools. A review of the specific components was conducted, and a report was prepared for the project team at Washington University that highlighted potential issues and provided an assessment of ways that the courses and tools may be enhanced to provide the user with the best educational experience possible.

Roundtable Rotation II: Regional Education Master�s Online Training in Evaluation (REMOTE): Lessons, and Questions

Roundtable Presentation 257 to be held in the Marble Boardroom on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Distance Ed. & Other Educational Technologies TIG

Presenter(s):

Charles Giuli, Pacific Resources for Education and Learning, giulic@prel.org

Abstract: A partnership between the University of Hawaii and the Pacific Resources for Education and Learning (PREL), sponsored by the National Science Foundation (NSF), offers a distance learning master�s degree to improve the evaluation capacity of the Pacific region. Nineteen professionals from the region are participating in the course. An inaugural 2-week introduction held in Honolulu was the only time participants were together. The rest of the course is delivered electronically using synchronous and asynchronous connections. The 30 credits of graduate-level work, the demands of students� professional lives, and the obligations of community and family they encounter pose challenges to course completion. The presentation will enumerate and discuss these challenges and possible responses. Among the challenges are competition for time, obsolete technology, and lack of personal interaction with fellow students. Possible responses include role-playing and rehearsal of likely challenges, a buddy system, and more time to complete the program.

Session Title: Growing Evaluation Leaders

Think Tank Session 258 to be held in Centennial Section A on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

Presenter(s):

Allison Titcomb, University of Arizona, atitcomb@email.arizona.edu

Discussant(s):

Deborah Loesch-Griffin, Turning Point Inc, trnpt@aol.com

Hallie Preskill, Claremont Graduate University, hpreskill@ca.rr.com

Abstract: AEA has a number of structures and opportunities designed to provide leadership in evaluation at the national level. Increasingly, some TIGs, like the Evaluation Managers and Supervisors and the Organizational Learning and Capacity Building TIGs, as well as the Local Affiliate Collaborative (LAC) have focused on the challenges of leadership in evaluation at local and state levels. An issue of 'New Directions in Evaluation' (Don Compton and Michael Baizerman) is forthcoming and promises a synthesis of the leadership literature and how it applies specifically to the field of evaluation. This Think Tank will invite participants from diverse leadership positions within AEA and the field of evaluation more broadly to consider, via small group discussions and large group reflection, questions and leadership models that hold promise for increasing leadership capacity at the local, state and national levels and provide opportunities for pathways for leadership development across these various levels.

Session Title: Quantitative Methods: Theory and Design TIG Business Meeting and Panel - Revisiting Validity: Implications for Theory, Practice, and Policy

Business Meeting with Panel Session 259 to be held in Centennial Section B on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

TIG Leader(s):

Patrick McKnight, George Mason University, pmcknigh@gmu.edu

George Julnes, Utah State University, george.julnes@usu.edu

Frederick Newman, Florida International University, newmanf@fiu.edu

Karen Larwin, University of Akron, drklarwin@yahoo.com

Chair(s):

Frederick Newman, Florida International University, newmanf@fiu.edu

Abstract: The quantitative traditions in evaluation are based on well-established views of validity. Indeed, many debates among quantitative evaluators are based on different views of validity. This panel will examine different aspects of how we view validity and suggest implications for how we think about validity, how we safeguard it in practice, and how we can use elaborated views of validity to improve evaluation policies.

Reframing Validity for Fun and Profit

George Julnes, Utah State University, george.julnes@usu.edu

In their seminal paper on developing our understanding of use or utilization of insights due to evaluation, Mark and Henry (2003) refer to the prior accretion of theory as being both 'overgrown and impoverished.' Theirs is a useful model of how we honor best the frameworks we have been bequeathed, taking the accumulated insights and trying to clarify and elaborate the relationships involved. This presentation attempts this with our accumulated concepts of validity, identifying dimensions that differentiate our primary types of validity and suggesting implications for evaluation policy.

Reconsidering Validity

Ernie House, University of Colorado, ernie.house@colorado.edu

An inspection of the drug evaluation literature reveals that many drug studies are biased to produce the findings that the sponsoring drug companies want. Yet reviewers of these studies, while noting the deliberate biases built in, often call these studies 'high quality' because the studies meet certain formal requirements like randomization and double blinding. What can one possibly mean by 'high quality' in such circumstances? Perhaps we should reconsider what the 'validity' of studies means and whether the definition should be differently delineated.

A Critique of the Campbellian Conceptualization of Validity and an Alternative

Charles Reichardt, University of Denver, creichar@du.edu

Validity is a central construct in the methodological literature in both the fields of measurement and experimentation. Within the field of experimental (and especially quasi-experimental) design, the most prominent conceptualization of validity is the Campbellian approach which categorizes validity into the four types of internal, external, construct, and statistical conclusion validity. The distinction between internal and external validity was first introduced by Campbell (1957) and subsequently immortalized in Campbell and Stanley's (1963, 1966) seminal treatise on quasi-experimental design. Cook and Campbell (1976, 1979) revised Campbell and Stanley, and expanded the original bipartite categorization of validity into the current four-fold version. In turn, Cook and Campbell (1979) has been revised by Shadish, Cook and Campbell (2002), who further explicated the logic behind the four-part typology. The purpose of the present paper is to detail inconsistencies and limitations in the Campbellian conceptualization. In addition, the present discussion provides an alternative conceptualization of validity which avoids the flaws in the Campbellian approach. My intent is not to replace the Campbellian conceptualization, at least not as it is used in practice. The Campbellian approach has become so firmly entrenched over the past half century that it is difficult to imagine how any critique, no matter how compelling, could dislodge it from its deeply embedded position within the consciousness of social scientists. Nonetheless, methodologists need to be cognizant of the flaws in the current zeitgeist if rigorous theories of experimentation are to be built upon a firm foundation.

Session Title: Benchmarking the United States' Program Assessment Rating Tool: Looking Back at Federal Program's PART Experiences and Getting Glimpses of the Future of Federal Evaluation Policy

Panel Session 260 to be held in Centennial Section C on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Presidential Strand and the Government Evaluation TIG

Chair(s):

Katherine Dawes, United States Environmental Protection Agency, dawes.katherine@epa.gov

Abstract: This session features a series of studies related to the Program Assessment Rating Tool (PART) assessments reported by the US Office of Management and Budget (OMB) as part of the federal budget process (See http://www.whitehouse.gov/omb/part/index.html and www.expectmore.gov). Being assessed under the PART system has proved challenging for many Federal programs, and the use of measures and program evaluation has frequently been a 'point of friction' between agencies and OMB. Several federal agencies have taken in-depth looks at published PART assessments to 'benchmark' their programs PART scores, learn about best practices from other agencies' PART assessments, and to better understand OMB expectations for performance measurement and program evaluation. The three presentations in this panel illustrate the challenges of PART, and consider the effect of agencies' PART experience on the future of federal evaluation policy.

Reflections on Environmental and Natural Resource Programs' Experiences with the Program Assessment Rating Tool and What It Means for Future Federal Environmental Evaluation Policy

Michael Mason, United States Environmental Protection Agency, mason.michael@epa.gov

The presentation will provide an overview of the PART experiences for federal environmental and natural resource programs from 2002 to 2008. Agencies included in the overview will be the Environmental Protection Agency (EPA), the Department of the Interior, the Department of Agriculture, and the National Oceanic and Atmospheric Administration. The presentation will provide a statistical summary of PART ratings and scores with a breakdown of section-by-section and question-by-question results. Trends will be noted and their implications will be discussed. In-depth analysis will also be provided on EPA's experience based on interviews with program PART coordinators. Environmental and natural resource PART results will be compared with a random selection of other federal programs in an effort to detect trends and differences. The presentation will provide a set of recommendations or options for evaluating federal programs with a focus on how environmental and natural resource programs could benefit from regular evaluations.

Program Assessment Rating Tool Scores and Budgetary Reactions from the Office of Management and Budget (OMB) and Congress: An Analysis of the Impact on Energy Efficiency and Renewable Energy Programs

David Shen, Consultant for the United States Department of Energy, david.shen@ee.doe.gov

The Program Assessment Rating Tool (PART) is linked to the Federal budget process and is used as a factor in determining if programs should receive greater or lesser amounts of funding. This paper reports on the findings from a statistical analysis of PART scores for 955 Federal programs and 340 Federal programs in a cohort analysis. It also examines program PART results for eleven programs in the Office of Energy Efficiency and Renewable Energy (EERE) in the Department of Energy (DOE). The analysis examines the first and second order changes in program budgets that may be associated with their PART scores. Although much statistical variance makes it difficult to claim strong significance, a pattern of budget increases with positive scores and budget cuts with lower scores was nevertheless found across all different types of Federal programs.

Research Evaluation and Performance Measurement: New Directions from the National Academy of Sciences

Phillip Juengst, United States Environmental Protection Agency, juengst.phillip@epa.gov

This presentation focuses on the Environmental Protection Agency's (EPA) success in initiating dialog with other agencies to improve the performance measurement and evaluation of research programs. EPA initially struggled with the Office of Management and Budget's (OMB) Program Assessment Rating Tool (PART), and subsequently examined and implemented best practices from around the government that significantly improved PART scores. EPA found wide variation in the types of measures used by Federal agencies and engaged the National Academy of Sciences (NAS) to conduct a study of the issue. NAS' report includes broad recommendations for research programs and changing OMB's PART process. EPA and OMB are leading discussions with other agencies to review these recommendations. The NAS report and cross-agency discussions are providing new directions for how and to what extent research programs should focus on intermediate outcomes, conduct evaluations that incorporate assessments of 'investment efficiency,' and develop meaningful 'process efficiency' measures.

Session Title: Using Developmental Evaluation in Complex Dynamic Systems

Panel Session 261 to be held in Centennial Section D on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Systems in Evaluation TIG and the Evaluation Use TIG

Chair(s):

Michael Quinn Patton, Utilization-Focused Evaluation, mqpatton@prodigy.net

Abstract: Developmental evaluation is defined as an evaluation approach that includes asking evaluative questions and applying evaluation logic to support program, product, staff and/or organizational development. The evaluator is part of a team whose members collaborate to conceptualize, design, and test new approaches in a long-term, on-going process of continuous adaptation, intentional change and transformation. The evaluator's primary function on the team is to elucidate team discussions with evaluative questions, data and logic, and to facilitate data-based decision-making in the developmental process. This session focuses on how developmental evaluation methods are being used to facilitate change in two dynamic, complex environments - healthcare and education. This panel discussion begins with an overview of developmental evaluation by Michael Q. Patton, who developed the approach, followed by a discussion of how four projects are using these methods to facilitate the development of both domestic and international education and healthcare systems.

Case Study: Developing the Ryan White HIV/AIDS Program�s Client-Level Data System

Margaret Hargreaves, Abt Associates Inc, meg_hargreaves@abtassoc.com

Meg Hargreaves, a Senior Associate at Abt Associates, will describe her work on the Health Resources and Services Administration (HRSA) HIV/AIDS Bureau (HAB) Ryan White HIV/AIDS Program's Client-Level Data System (CLD) Vetting Project. Abt worked with HAB to vet the proposed system with over 500 grantees and stakeholders, using a developmental evaluation design to facilitate the national data collection and reporting system's development.

Case Study: Evaluating USAID's EQUIP 2 Program for Education Policy, Systems and Management

John Gillies, Academy for Educational Development, jgillies@aed.org

John Gillies, a Project Director at the Academy for Educational Development, will describe his work on EQUIP 2, a USAID Program for Education Policy, Systems and Management, using the practice and logic of developmental evaluation to engage project teams and national stakeholders in an analysis of the status of education systems in foreign aid projects.

Case Study: Improving USAID's Healthcare Quality Assurance Project

Mary Gutmann, EnCompass LLC, mgutmann@encompassworld.com

Tessie Catsambas, President at EnCompass, will describe her work on a USAID Quality Assurance Project to adapt in developing countries healthcare quality improvement methods that were developed by the Institute for Healthcare Improvement. She used systems thinking and developmental evaluation methods to identify and document an emerging model of rapid collaborative improvement of healthcare quality in eleven developing countries.

Case Study: Fostering Multi-level Systems Change in a Community Mental Health Center in West Suburban Denver

John Mahalik, Jefferson Center for Mental Health, johnma@jcmh.org

John Mahalik, Director of Program Evaluation and Research at Jefferson Center for Mental Health, a community mental health center in West suburban Denver, will describe his systemic evaluation approach to actively engage participants in fostering multi-level change (individual, group, organization and community) for internal and external consumers of evaluation and mental health services.

Session Title: A Step-by-Step Model for Collaborative Evaluations

Skill-Building Workshop 262 to be held in Centennial Section E on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Graduate Student and New Evaluator TIG

Presenter(s):

Liliana Rodriguez Campos, University of South Florida, lrodriguez@coedu.usf.edu

Abstract: This workshop provides effective tools to help master the mechanics of collaboration. The presenter explains a model for collaborative evaluations (MCE) and shares her experiences after implementing it in the private sector, nonprofit organizations, and higher education. Participants will learn to: review the factors that influence the success of collaboration; capitalize on others' strengths to encourage feedback, clarify interpretations, and resolve misunderstandings; select appropriate methods and/or tools in order to respond to and minimize resistance to collaborative initiatives.

Session Title: Interviews With Exemplary Evaluators: Evaluation in Action

Panel Session 263 to be held in Centennial Section F on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Research on Evaluation TIG

Chair(s):

Melvin Mark, Pennsylvania State University, m5m@psu.edu

Discussant(s):

Ross Conner, University of California Irvine, rfconner@uci.edu

Abstract: Through interviews concerning an evaluation, the panel will add to our knowledge about practice by illustrating the choices that exemplary evaluators make in a particular evaluation. Building on the Exemplars interview columns in American Journal of Evaluation, the two interviewers from the Exemplars column will each introduce an evaluation and then interview the evaluator on key components of that particular evaluation. Each interviewer will select two areas of the interview that provide insights into how evaluators make choices in a particular setting. Areas to be probed include establishing cultural competency, evaluator's roles in working with stakeholders, and methodological choices. Following the interview/dialogue, the evaluators interviewed will expand on their role in the evaluation. Finally, the interviewers will comment on what can be learned about practice through these case studies.

A Dialogue with Katrina Bledsoe: Establishing Cultural Competency and Use in a Community-Based Evaluation

Jody Fitzpatrick, University of Colorado Denver, jody.fitzpatrick@cudenver.edu

Katrina Bledsoe, Walter R McDonald and Associates Inc, kbledsoe@wrma.com

Fitzpatrick will begin by summarizing Katrina Bledsoe's evaluation of the Fun with Books (FWB) program in Trenton, New Jersey. Fitzpatrick's interview with Bledsoe will then focus on how Bledsoe recognized that she needed to obtain cultural competence and how she attempted to do so. Fitzpatrick will also focus on Bledsoe's actions to facilitate use. Bledsoe is hired to evaluate what program administrators view as their 'flagship' program, their most successful. They want the evaluation to demonstrate their success. Yet, Bledsoe identifies a number of problems. Ultimately, the administrators accept the evaluation and make many of the program changes that she recommends. Thus, the second part of Fitzpatrick's questions to Bledsoe on the panel will concern how she defined her role to get administrators to accept disappointing results and move toward improvement. The case is a successful example of use in what might have been a contentious situation.

A Dialogue with Ross Conner: An Evaluator's Role and Approach to a Complex Evaluation

Christina Christie, Claremont Graduate University, tina.christie@cgu.edu

Ross Conner, University of California Irvine, rfconner@uci.edu

Christie will summarize Conner's evaluation of the Colorado Healthy Communities Initiative. When Conner began his work, the Initiative was by design undefined. Conner provided an overarching initiative level evaluation of the overall program process. Christie's interview with Conner will focus on several elements of Conner's work including his role in the evaluation, involvement of stakeholders, and how he facilitated use. Christie will focus on how Conner's evaluation approach (collaborative, community-based) allowed him to be responsive to the complexities of the evaluation and provided him with the flexibility necessary for examining program processes at both the community and initiative levels. Conner's evaluation was recognized for its overall impact, and so Christie will question Conner about how his approach helped to facilitate both process and findings use. Related to this, Christie will ask Conner to expound upon his self-described role as a 'story teller' and how this influenced his overall evaluation approach.

Session Title: Improving Youth Development Practice through Program Evaluation

Multipaper Session 264 to be held in Centennial Section G on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Extension Education Evaluation TIG

Chair(s):

Mary Arnold, Oregon State University, mary.arnold@oregonstate.edu

So You Want to Evaluate a Community-Based Youth Program?: Strategies for Success

Presenter(s):

Lynne Borden, University of Arizona, bordenl@ag.arizona.edu

Joyce Serido, University of Arizona, jserido@email.arizona.edu

Christine Bracamonte Wiggs, University of Arizona, cbmonte@email.arizona.edu

Abstract: Evaluating community-based youth programs is an expanding and evolving science. The need for high quality data to ensure that programs are meeting the needs of the young people is critical. Valid and reliable data is essential to program�s long term success. Thus, evaluation practices must rise to the occasion and provide this much needed data. There are many challenges when evaluating youth programs, youth programs have voluntary participation, diversity of age, and often exist for differing lengths of time. A Participatory Action Research (PAR) model provides a set of strategies that can assist in this work. PAR can be defined as �a participatory, democratic, practical knowing in the pursuit of worthwhile human purposes, grounded in a participatory worldview (p. Reason & Bradbury, 2001, p. 1). Given that youth development programs are often fluid with unique challenges; it is important to see young people as active participants in the evaluation process.

Learning-by-Doing Evaluation with Youth Professionals

Presenter(s):

Ben Silliman, North Carolina State University, ben_silliman@ncsu.edu

Abstract: This paper describes the process and outcomes used in an evaluation study of a 4-H district public speaking contest. A utilization-focused evaluation process engaged 13 4-H professionals in planning and implementing the evaluation and using reports for accountability and marketing. Working with an evaluator (the author), youth professionals contributed critical insights on the content, format, and timing of evaluation, recruited data collectors and gained parent consent, managed data collection, and helped in dissemination of evaluation results. As a result, over 90% of presentation contest participants completed the survey and 100% of randomly selected youth completed in-depth interviews. Participating youth reported improvements in confidence, subject knowledge, and life skills that were subsequently applied to 4-H, school, and community settings. Collaborators reported gains in confidence and skills for evaluation as well as insights on programming and marketing of youth programs. Discussion focuses on evaluation capacity-building and use of results to improve youth programs.

Understanding, Evaluating and Elevating Youth Development Practice

Presenter(s):

Kate Walker, University of Minnesota, kcwalker@umn.edu

Abstract: In the field of youth development, there has been increased attention to determining how best to equip those who work with young people. A number of efforts aim to clarify and come to consensus around the essential knowledge and core competencies that are central to youth work practice. Recent research suggests that a central element of youth worker competence resides in their abilities to appraise and respond to the dilemmas of daily practice (Larson & Walker, 2008). To understand how practitioners are effective, we must understand how they deal with the conflicts, challenges and crises of everyday practice. This study explores and assesses how novice and experienced practitioners approach these dilemmas. To evaluate practitioner training and practice, evaluators need to be able to assess effective reasoning and responses to practitioner dilemmas. This paper describes our applied research efforts aimed at conceptualizing and developing a measure of youth development practitioner reasoning.

Session Title: Using Logic Models to Guide Program Implementation and Evaluation: Examples From the Centers for Disease Control and Prevention (CDC)

Panel Session 265 to be held in Centennial Section H on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Program Theory and Theory-driven Evaluation TIG

Chair(s):

Thomas Chapel, Centers for Disease Control and Prevention, tchapel@cdc.gov

Abstract: The starting point of meaningful evaluation is a strong program description. Logic models have been the chosen tool for depicting programs in succinct, graphic formats. Logic models, once elaborated serve as program theories that can guide program planning, implementation, and program evaluation. This presentation discusses selected CDC programs, all of have used their program theories to guide diverse aspects of program implementation and program evaluation. In all cases, program staff present their initial thinking, how they identified and selected their program theory or frameworks, how the underlying logic of their programs has been influential in making choices in implementation and evaluation.

Using a Logic Model to Guide Evaluation Capacity Building

Maureen Wilce, Centers for Disease Control and Prevention, mwilce@cdc.gov

Heather Joseph, Centers for Disease Control and Prevention, hbj7@cdc.gov

Logic models are widely used tools in planning and evaluation. Typically, logic models are created to help guide program evaluations, but may also be used as planning tools to reflect program capacity building processes. In 2002, the Division of TB Elimination and its partners decided to expand our evaluation efforts. From the literature and experience, we recognized the need to develop logic models for common TB program components. As we considered our strategy for evaluation capacity building (ECB), we realized that a different kind of logic model would be useful. This logic model delineated essential activities of the ECB process, identified interrelationships of our activities, and defined expected outcomes. It enabled us to focus our efforts, articulate the concept of ECB to external partners, and recruit participants to assist us in defined tasks. The logic model effectively guided the ECB process and provided a mechanism to track successes.

Logic Models as Multi-Faceted Resources to Advance Evaluation Policy

Marlene Glassman, Centers for Disease Control and Prevention, mglassman@cdc.gov

By requiring the health departments and community-based organizations (CBOs) it funds for HIV prevention to participate in its national monitoring and evaluation system, CDC's Division of HIV/AIDS Prevention (DHAP) puts forth policy -- that program monitoring and evaluation (M&E) should take place as elements of a comprehensive HIV prevention program. The three major components of M&E in HIV prevention are process monitoring, process evaluation, and outcome monitoring. Logic models can inform M&E as well as the design and implementation of evidence-based interventions. More specifically, logic models can help identify variables for process monitoring, process evaluation, and outcome monitoring and the development of SMART (specific, measurable, appropriate, realistic, and time-phased) objectives for each of these M&E activities. By tracking progress in the achievement of objectives grounded in behavior change and implementation logic, CBOs advance DHAP policy and the likelihood of intervention success.

Evaluability Assessment: Case Management for Completion of TB Therapy

Judy Gibson, Centers for Disease Control and Prevention, jsd0@cdc.gov

Maureen Wilce, Centers for Disease Control and Prevention, mwilce@cdc.gov

Tuberculosis (TB) treatment requires long term but finite adherence by patient and provider. Case management for treatment completion is done by nurses, but the value added by the nurse not well defined. Patient outcomes can be measured, but there is no system to link the nurses' performance with these results. The Division of TB Elimination (DTBE) recognized the need to clarify goals and objectives and develop a theory of how nurses' performance affects patient outcomes. DTBE undertook an evaluability assessment (EA) with partners in state and local TB programs. The assessment began by engaging stakeholders on an EA team, reviewing literature, and exploring practice. The team collaboratively developed a logic model and used it to develop evaluation tools and establish a performance measurement system. Initial applications of this system have shown the value of the EA for improving public health programs by enhancing capacity for self-evaluation.

Session Title: Evaluation Practice and Policy in Social Service Systems

Multipaper Session 266 to be held in Mineral Hall Section A on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Health Evaluation TIG

Chair(s):

Dale Stratford, Centers for Disease Control and Prevention, bstratford@cdc.gov

Abstract: National health promotion and disease prevention programs require comprehensive monitoring and evaluation (M&E) plans to ensure quality service delivery, assess progress towards goals and objectives, and promote accountability. The Centers for Disease Control and Prevention (CDC) has implemented a multi-method M&E strategy to strengthen the Counseling, Testing, and Referral (CTR) component of CDC's national HIV prevention program. This session describes methods used to implement three HIV prevention M&E activities, and their applicability to other national programs: 1) development of quality assurance standards aimed at improving CTR data collection, management, and reporting; 2) use of national HIV prevention program performance indicators for monitoring progress in achieving CTR goals and objectives, and; 3) assessment of the impact of CDC's new recommendations for routine HIV testing in health care settings. Lessons learned about the collection, reporting, and utilization of M&E data for program improvement will be highlighted.

Program and Data Quality Assessment and Standards for HIV Counseling, Testing and Referral (PADQUA): Implications for Centers for Disease Control's National HIV Prevention Monitoring and Evaluation Efforts

Renee Stein, Centers for Disease Control and Prevention, rstein1@cdc.gov

Denise Duran, Centers for Disease Control and Prevention, dduran@cdc.gov

Jane Mezoff, Centers for Disease Control and Prevention, jmezoff@cdc.gov

Frances Belcher, Macro International Inc, frances.e.belcher@macrointernational.com

As part of the Division of HIV / AIDS Prevention's (DHAP) national monitoring and evaluation plan, client and test-level data are required to be collected at CDC-supported HIV Counseling, Testing and Referral (CTR) sites to describe the population receiving CTR services and to inform HIV prevention planning. CTR data are submitted by service providers to health departments where they are cleaned and analyzed for state-level reporting and prepared for submission to CDC for national-level analysis and reporting. Each step in the multi-level CTR data life cycle introduces the opportunity for data error and would benefit from the implementation of national quality assurance standards and monitoring procedures. This session will provide an overview of PADQUA, a project initiated by DHAP's Program Evaluation Branch to support the national investment in high quality monitoring data by developing and piloting a set of national data quality assurance standards for CTR data.

Using Performance Indicators to Monitor National HIV Prevention Programs

Barbara Maciak, Centers for Disease Control and Prevention, bmaciak@cdc.gov

Tamara Lamia, Macro International Inc, tamara.l.lamia@macrointernational.com

David Cotton, Macro International Inc, david.a.cotton@macrointernational.com

Pankaja Panda, Centers for Disease Control and Prevention, ppanda@cdc.gov

Marla Vaughan, Centers for Disease Control and Prevention, mvaughan@cdc.gov

Dale Stratford, Centers for Disease Control and Prevention, bstratford@cdc.gov

Performance indicators provide an important tool for monitoring national prevention programs, improving service delivery, and ensuring accountability. In 2003, the CDC's Division of HIV/AIDS Prevention (DHAP) developed National HIV Prevention Program Performance Indicators addressing program objectives and desired outcomes in critical areas of HIV prevention planning, service delivery, and evaluation. Since 2003, fifty-nine CDC-funded health department jurisdictions have submitted baseline, annual target, and five-year goal measures for each of 23 performance indicators. This presentation describes CDC's rationale for developing HIV prevention performance indicators; strategies for engaging stakeholders in indicator development and implementation; lessons learned regarding indicator data collection, calculation, and reporting, and; the utility of indicator data in the context of a national monitoring and evaluation plan. Sample indicators, calculation methods, and implementation challenges are presented for Counseling, Testing, and Referral (CTR).

Evaluating the National Health Policy of Routine HIV Testing in Health-care Settings in the United States

Denise Duran, Centers for Disease Control and Prevention, dduran@cdc.gov

Renee Stein, Centers for Disease Control and Prevention, rstein1@cdc.gov

Frances Belcher, Centers for Disease Control and Prevention, frances.e.belcher@macrointernational.com

John Beltrami, Centers for Disease Control and Prevention, jbeltrami@cdc.gov

Dale Stratford, Centers for Disease Control and Prevention, bsratford@cdc.gov

In 2006, the Centers for Disease Control and Prevention (CDC) released the revised HIV testing recommendations for private and public health-care providers in U.S. health-care settings. In 2007, a new program announcement (PA) funded 23 health departments to expand and integrate HIV testing in clinical settings for populations disproportionately affected by HIV. Short-term and long-term outcomes of the CDC recommendations and objectives of the PA include increasing the number of people diagnosed with HIV that are unaware of their infection and decreasing HIV morbidity and mortality, respectively. The Assessment of HIV Testing in Clinical Settings (AHITS) is a multi-methods evaluation project focusing on estimating the baseline of HIV testing prior to the release of the CDC recommendations and assessing challenges, strategies, and successes of grantees funded under the PA. AHITS includes process and outcome evaluation methods, and uses population-based surveys and in-depth interviews.

Session Title: Using Pictorial Survey Methods in Program Evaluation to Reach Adults With Lower-Literacy Skills and Students in Special Education

Demonstration Session 282 to be held in Room 104 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Special Needs Populations TIG

Presenter(s):

Deborah Mattila, The Improve Group Inc, deborahm@theimprovegroup.com

Rebecca Stewart, The Improve Group Inc, becky@theimprovegroup.com

Abstract: Creating data collection instruments that allow those with low literacy levels or learning abilities to self-report about program impacts presents a challenge for evaluators. In this session, presenters from the Improve Group, Inc. will share instruments they have developed to work with (1) children in special and general education settings and (2) adults of mixed and low literacy levels. Using pictorial tools, respondents draw, create collages or select images to depict feelings and reflect on change. Presenters will describe the goals and parameters of the evaluations for which these tools were developed and lessons learned from their development. In addition, the kinds of analysis and findings that these tools can support will also be explored. Presentation and discussion will focus on the opportunities, challenges, considerations and limitations associated with these tools. Attendees will be invited to share experiences, observations and suggestions on the use of pictorial instruments in evaluation.

Session Title: Gauging the Impact of a National Literary Reading Initiative: The Goals, Challenges, and Findings of The Big Read Evaluation

Panel Session 283 to be held in Room 106 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the AEA Conference Committee

Chair(s):

Kay Sloan, Rockman et al, kay@rockman.com

Discussant(s):

Saul Rockman, Rockman et al, saul@rockman.com

Abstract: In 2004 the National Endowment for the Arts published Reading at Risk: A Survey of Literary Reading in America, which found that literary reading was on the decline, especially among the young. The NEA responded with The Big Read, a grassroots literary program designed to bring American communities together to read a single book and celebrate great imaginative literature. By 2008, over 300 communities had received grants and held local reading programs. The Big Read also included a national evaluation to study the program's implementation and impact. This panel, with presenters from the evaluation team and the National Endowment for the Arts, will look at the evaluation goals, challenges, and findings. Panelists will describe the methodological and logistical challenges of a study conducted in 300 different communities, and efforts to make the study rigorous and the data reliable, but still keep data collection sensitive to local needs and nuances.

The Research Behind the Big Read

Kay Sloan, Rockman et al, kay@rockman.com

This presentation provides background on The Big Read, discussing its impetus and NEA research that was a reference point for the evaluation. Reading at Risk presented results of the literature segment of the Census Bureau's 2002 Study of Public Participation in the Arts, conducted with 17,000 adults. The report indicated that reading was declining in the U.S. and that declines were steepest among young adults. To address and reverse these trends, the NEA launched The Big Read in 2006 as a pilot in 10 communities. It has grown steadily, and by 2009 some 400 communities will have held Big Reads. An important question for the NEA, one the evaluation could help determine, was 'Is The Big Read making a difference?' That question, and evaluation's focus on teens and young adults, intensified with a second report, To Read or Not To Read, which presented even more troubling news about reading declines.

Assessing Implementation and Impact: Relative Challenges and Successes

Kay Sloan, Rockman et al, kay@rockman.com

The Big Read evaluation was designed to explore and assess both implementation and impact. This presentation will describe the methodological and logistical challenges we faced in the two and a half years of the study, the success of our solutions, and the major findings. Assessing implementation was relatively uncomplicated because grantees took part in surveys and interviews and eagerly shared information on partnerships, programming, and the value of the program. Many findings were compelling and unambiguous: the numbers of events and attendees were high, and the program built capacity and unprecedented partnerships for organizations, increased status and visibility, and laid the groundwork for future collaborations. Although there were many pieces to the site data population served, funding, prior experience, numbers of events data were easy to tabulate, aggregate, and analyze. The impact on reading habits and attitudes was far harder to assess and quantify.

Telling the Story of The Big Read: Findings from Qualitative Studies

Michelle Honeyford, Rockman et al, michelle@rockman.com

A complement to the quantitative data collected for The Big Read evaluation was the qualitative case study and focus group data. This presentation will share goals, challenges, and findings from these studies, which were designed to tell the story of The Big Read, in a local context, looking at unique partnerships and programming and the program's impact on local communities and readers. Although cases provided some of the most compelling and uncomplicated evidence of impact, this evaluation component had its own challenges how to chose a handful of sites with the appropriate geographical, demographic, and organizational range and variety, and how to be there at the right time to capture that rich and varied story. Focus groups with teens and young adults, conducted during the third cycle of The Big Read, presented a unique set of challenges but yielded valuable findings.

Session Title: Challenges in Evaluating Work of International Nongovernmental Organizations

Multipaper Session 284 to be held in Room 108 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Barry B Cohen, Rainbow Research Inc, bcohen@rainbowresearch.org

Evaluating Sustainability in Non-Governmental Organization (NGO) Programs: Findings, Challenges and Lessons Learned From an Ex-Post Field Study

Presenter(s):

Joe Dickman, Mercy Corps, jdickman@bos.mercycorps.org

Gretchen Shanks, Mercy Corps, gshanks@mercycorps.org

Sandy Sheard, University of California Berkeley, sandy_sheard@berkeley.edu

Brandy Westerman, Mercy Corps, brandy@tj.mercycorps.org

Abstract: In 2007, Mercy Corps, an international Non-governmental Organization (NGO), undertook a research study to analyze lasting behavior change and project sustainability stemming from two large multi-year, multi-site programs funded by the U.S. Agency for International Development (USAID) in Central Asia. Implementation of program activities in the target communities had finished one to five years prior to conducting the study. The research team analyzed the programs� impact in communities according to three themes: participation, accountability and collective community action. The study found significant evidence of sustainability both in terms of project outputs and lasting behavioral change, while noting key differences among sub-groups and project types and acknowledging various methodological constraints. The paper provides a brief overview of the study�s key findings and recommendations, and discusses the methodological challenges and lessons learned for evaluating program impact and sustainability ex-post in dynamic, transitional environments.

An Analysis of the Efforts to Improve the Quality of International Aid Evaluations

Presenter(s):

Thomaz Chianca, COMEA Communication and Evaluation Ltd, thomaz.chianca@gmail.com

Abstract: This paper analyzes the 16 most prominent and documented efforts in place to improve the quality of aid evaluations until 2007. The efforts were classified into five groups: (i) consortia of organizations (e.g., 3IE and NONIE); (ii) multilateral and bilateral organizations (e.g., OECD/DAC and UNEG); (iii) International Nongovernmental Organizations (e.g., InterAction); one had INGOs as the leading agencies; (iv) professional associations or networks (e.g., IDEAS and IOCE); and (v) international development research groups and think tanks (J-PAL and SEGA). While most of the reviewed efforts propose more holistic approaches to move the field forward, at least six of them were solely focused on improving the quality of impact evaluations including a few advocating strongly for the use of randomized control trials to improve aid evaluation. This paper describes and provides a critical review of those efforts, indicating opportunities for advancing the field.

Session Title: Decolonized Evaluation in the Real World: Two Projects Walk the Talk in Native Hawaiian Communities

Multipaper Session 285 to be held in Room 110 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Indigenous Peoples in Evaluation TIG

Chair(s):

Alice J Kawakami, University of Hawaii, alicek@hawaii.edu

Discussant(s):

Karen E Kirkhart, Syracuse University, kirkhart@syr.edu

Jennifer Greene, University of Illinois Urbana-Champaign, jcgreene@uiuc.edu

Abstract: The process of decolonizing evaluation often leads to an empowerment model for indigenous communities and other 'colonized' populations conducted by and for its members. This two-paper session describes projects, in two Native Hawaiian communities, that were designed and implemented by Native Hawaiian professionals. The first paper describes the evaluation of an early childhood advocacy and networking program. The second paper describes an assessment and development of support initiatives in a program for recruitment, development, and retention of teachers from the community. Both presentations provide examples of community-based evaluation that honors and incorporates indigenous perspectives in the process of community development resulting in program features that perpetuate the culture and values of the communities for young children and adults.

How a Decolonized Evaluation Differed From Mainstream Evaluations and Better Protected Young Children

Morris Lai, University of Hawaii Manoa, lai@hawaii.edu

In an evaluation of an early childhood education (ECE) initiative, we found specific differences in practice between our decolonized, indigenous approach and more mainstream approaches. For example, we regarded interview input from parents regarding effects on children as more valid than 'objective' information from tests developed by strangers from far away. We viewed randomized controlled trials that include control groups receiving no preschool as ethically and methodologically inappropriate; accordingly we obtained comparison groups without using random assignment to treatment. Although it was initially through our indigenous lenses that we noticed problems with the evaluation practices used in widely acclaimed ECE studies, we also realized that those mainstream ECE studies also violate our Western standards. We provide other examples on how a decolonized, indigenous evaluation in practice (a) is distinctly different from mainstream evaluations and (b) better protects the children in the evaluation.

Kukuluao (Building Enlightenment): Support for Teachers from a Native Community

Alice J Kawakami, University of Hawaii, alicek@hawaii.edu

Kukuluao, a teacher support program, was funded in June 2007 to identify existing support and needs and to design assistance for community members along the continuum of the teacher career path to encourage them to become trained and to persist in the teaching profession. The assessment was conducted by staff from the community, resulting in a high return rate on surveys. Results of the surveys were validated by a community advisory board and initiatives designed and implemented by community members in ways that perpetuate the culture of the community through education and professional development for teachers. The presentation will describe ways that the development of capacity and inclusion of the native communities' voices is fundamental to building a stable core of Native Hawaiian teachers for this community.

Session Title: Bridging Gaps between basic and applied Research and Development and the 'Valley of Death'

Multipaper Session 286 to be held in Room 112 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Research, Technology, and Development Evaluation TIG

Chair(s):

Yongsuk Jang, Science and Technology Policy Institute, jang@stepi.re.kr

Abstract: There is a big gap in applying knowledge produced by basic research to innovative products and services. This gap is often referred to as 'Valley of Death.' Due to many reasons including insufficient financial availability to small and medium enterprises, major actors converting basic knowledge to specific innovations, bridging this 'valley of death' increasingly becomes one of primary foci of science, technology, and innovation policies in almost all countries. Each country however adopts quite different policy measures to bridge the basic and applied research and development. This session first examines the theoretical background of 'valley of death' mainly from the economic perspective to identify various factors that cause this gap. Then it is followed by three country studies that focus on country conditions in terms of 'valley of death' and policy measures to bridge the gap. The selected countries for case studies are Korea, Japan, and Mexico.

A Valley of Death in the Innovation Sequence: An Economic Investigation

George S Ford, Phoenix Center for Advanced Legal and Economic Public Policy Studies, george.ford@phoenix-center.org

Thomas M Koutsky, Phoenix Center for Advanced Legal and Economic Public Policy Studies, tom@koutsky.com

Lawrewnce J Spiwak, Phoenix Center for Advanced Legal and Economic Public Policy Studies, lspiwak@phoenix-center.org

Innovators and investors alike routinely claim that a 'Valley of Death' exists at intermediate stages of the innovation sequence. Rationalization of this valley is typically limited to the standard explanations for underinvestment in research. But the image of a valley suggests that funding is more readily available for 'early-stage' and 'late stage' research (the 'peaks') than the intermediate stages (the 'valley'). Our economic model implies that this nonlinearity occurs only in the presence of 'noneconomic' investments (such as government expenditures) at early stages, where insufficient attention is given to decisions at later stages. We argue the standard explanations for reduced investment in research cannot produce a valley. We also consider the effect of noneconomic investments on the location of lenders and the risk premium. Our research suggests technology policymakers should study the optimal mix of government support for early-stage and intermediate-stage R&D projects.

Organizational Barrier in Bridging the 'Valley of Death' in Korea

Yongsuk Jang, Science and Technology Policy Institute, jang@stepi.re.kr

During the past few decades, Korea has been quite successful in commercializing the imported technologies primarily through Chaebol companies. As its technological capability has ripened into the advanced level, Korea has invested more resources in basic sciences for endogenous technologies. This progress has introduced the deepening gaps between basic and applied R&D - difficulties in linking university researches to industrial developments especially through SMEs. This paper overviews the Korean 'valley of death' through various statistics and major policy measures to bridge this gap. With emphasis, it also introduces the recent governmental reform - grouping several innovation-related agencies into two major ministries - that has taken place under the new administration and discusses its impact on policy efforts for various innovators to overcome such 'valley of death.'

'Full Research' to Overcome 'Valley of Death' in National Institute of Advanced Industrial Science and Technology (AIST)

Osamu Nakamura, National Institute of Advanced Industrial Science and Technology, osamu.nakamura@aist.go.jp

In 1990s, the Japanese industries had lost its competitiveness and Japanese economy was in slump in spite of the increased funding in R&D both in private and in public sectors. To overcome these situations, METI has been publishing the 'Strategic Technology Roadmap'. This is designated to show strategies formulated to develop new industries and measures to stimulate demand for a variety of products and services. The missions of AIST are contribution to 1) a sustainable society, 2) industrial competitiveness, 3) local industrial development, and 4) industrial technology policies. For these missions, AIST advances 'Full Research' which is a unique and original research methodology developed by AIST to cover basic researches through applied researches to overcome 'Valley of Death' by creating the multilayered network among industry, academia, and government. Under this concept, selection, integration, and application of knowledge accumulated by discovery and invention have been undertaken to promote product realization.

From S&T to Innovation: A Challenge to Mexico

Enrique Campos Lopez, Center of Investigation and Assistance in Technology and Design Diseno del Estado de Jalisco, enriquecampos@mac.com

Mexico�s Science and Technology policy was established in 1971. Its main objective and policies were to improve human capital, geographical decentralization and physical infrastructure. During the first three decades efforts to bring private sector participation were weak and generally unsuccessful. Mexican S&T remain fragmented with weak links between innovation actors. Indicators such as R&D investment are the lowest within the OECD�s countries. From the eighties Mexico started to open its traditionally closed and protected economy, and to face the competitiveness challenge. To move from a S&T conglomerate to an innovation system became explicit and urgent. The first steps came in 2002, with a new Law of Science and Technology several policies and reinforcing mechanisms (as in tax incentives) appeared to induce private sector participation and creating new mechanisms for innovation at a regional perspective. This paper discuss challenges to transit to an innovation system and the role that regional policies could play.

Session Title: Techno-babble: Evaluating Programs That Focus on Participant Use of Unique Technologies

Think Tank Session 287 to be held in Room 103 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Integrating Technology Into Evaluation

Chair(s):

Emma Norland, e-Norland Group, enorland2@comcast.net

Presenter(s):

Susan Foutz, Institute for Learning Innovation, foutz@ilinet.org

Irene Porro, Massachusetts Institute of Technology, iporro@mit.edu

Abstract: As technology-focused programs become more and more common, evaluators are required to investigate the use of the technology and how it relates to the learning outcomes. Participants in this Think Tank will discuss the implications of evaluating a program that focuses intensely on a unique technology. An example from the facilitators' experience evaluating an after-school astronomy program will be discussed and the software in the case, MicroObservatory, will be demonstrated briefly. Participants will be invited to share examples from their practice with the entire group. Small groups will then discuss key questions: How much of an understanding of the technology should evaluators have? How is it possible to separate out learning to use the technology from learning the subject matter? Other questions generated by participants will also be discussed. Each group will share key points from their discussions and, led by the session chair, all participants will identify implications for practice.

Session Title: Growth Charts: The Monitoring and Evaluation of Capacity Building Efforts

Panel Session 288 to be held in Room 105 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Chair(s):

Matt Reeves, Pact Inc, mreeves@pacthq.org

Abstract: Capacity building, cited as the 'missing link' for achieving the Millennium Development Goals, has been a key focus for funders and non-profits in recent years. Today, there are increasing demands, by donors and beneficiaries alike, to demonstrate the value of capacity building initiatives, linking them to community level impact. This panel aims to add to the body of knowledge on what makes capacity building successful, how to build the capacity for evaluation in nonprofits, and primarily to focus on evaluating the successes and failures of capacity building projects in international and domestic contexts. Panelists represent USAID, Pact and First 5 Monterey County.

Evaluation of Capacity Building Efforts, The Donor Perspective

Annie La Tour, United States Agency for International Development, latouram@state.gov

Annie La Tour, Strategic Information Officer at the Office of the Global AIDS Coordinator will present to the perspective of an international donor organization. USG has come under increasing pressure to demonstrate impact in order to secure further funding for international development activities. As a result, much effort has gone into developing indicators and reporting mechanisms to which grantees are held accountable. The international donor's perspective is key to understanding how development policy has catalyzed greater accountability and learning through the international development sector.

Evaluation of Capacity Building Efforts, The International Perspective (Field)

Lynn McCoy, Pact Inc, lmccoy@pacthq.org

Lynn McCoy will present to the perspective of an International NGO operating with partners in developing countries. She is Pact's Global Director of Monitoring and Evaluation and has been involved in the development of numerous training materials used by Pact with partner organizations in developing countries. Since the majority of Pact's programs involve capacity building, the organization is working to develop and refine tools and methodologies for monitoring the impact of its interventions. These tools and methodologies must gather key information for funders as well as assisting local partners to learn more about their work.

Evaluation of Capacity Building Efforts, The International Perspective (HQ)

Matt Reeves, Pact Inc, mreeves@pacthq.org

Matt Reeves will speak to the perspective of a headquarters based staff person in an International NGO. As part of his work his has researched and applied a number of innovative tools for monitoring and evaluating the success of capacity building interventions. He will speak to the need to align research and practice - empowering local actors while gathering the information necessary for improving practice and reporting to donors.

Evaluation of Capacity Building Efforts, the Local Perspective

David Dobrowski, First 5 Monterey County, david@first5monterey.org

David Dobrowski, Evaluation for First 5 Monterey County, will present a perspective of implementing organizational assessment and capacity building activities based on the findings in a semi-rural county in California with 20 small local nonprofits and county agencies. Methods for evaluating success, failure and recommending improvements in capacity building activities include program coordinator feedback, changes in program services and/or outcomes, and completion of sustainability plan activities. His perspective allows parallels and differences with international development work to be highlighted.

Session Title: Improving Global Results: Getting the Most out of Evaluations of Global Partnership Programs - Through Intentional Planning, Policies, and Quality Standards

Multipaper Session 289 to be held in Room 107 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Evaluation Use TIG and the International and Cross-cultural Evaluation TIG

Chair(s):

Uma Lele, Independent Consultant, umalele.orgemail

Abstract: The presentations and discussion will familiarize evaluators (and users of evaluations) with the challenges involved in evaluating Global and Regional Partnership Programs (GRPPs), and equip them with emerging consensus standards and guidelines for application to development programs. These standards and guidelines were developed by the Independent Evaluation Group (IEG) of the World Bank under the auspices of the OECD/DAC Evaluation Network, with input and validation from a wide range of stakeholders from both developed and developing countries. GRPPs are complex to evaluate because they (a) often involve provision of global public goods (GPGs); (b) are usually funded by multiple donors, requiring complex governance and collective decision-making processes; (c) evolve over time as the scale and specific strategies change with donor mix, resources, or a shifting external environment; and (d) have a global community clientele, making planning, quality, and transparency particularly important.

Intentional Evaluation Policy Development, Planning and Budgeting for Global and Regional Partnership Programs: Lessons Learned

Dale Hill, World Bank Group, dhill@worldbank.org

Intentional Planning for high-quality evaluations of Global and Regional Partnership Programs (GRPPs) should begin soon after inception ideally including the intention or requirement in the charter. A framework for such evaluations may also be defined in advance. Once the Governing Body is assembled, an evaluation policy should be established, which covers roles and responsibilities, principles on independence and participation, evaluator selection processes, transparency and disclosure, and processes for acting on findings. In its ongoing review of Monitoring and evaluation (M&E) processes for a sample of GRPPs, the independent Evaluation Group (IEG ) of the World Bank has found that such evidence of intentionality and sound planning are often lacking. This presentation will cover the ingredients of intentional planning for M&E that provide the foundation for subsequent high-quality evaluations, and help ensure their productive use, and will present the range of experience found and some examples of good practice.

Adapting Standard Evaluation Criteria to evaluation of Global and Regional Partnership Programs

Chris Gerrard, World Bank Group, cgerrard1@worldbank.org

Standard Evaluation Criteria can be applied to Global and Regional Partnership Programs, but several features pose complexities: (a) They often involve provision of global public goods; (b) They are usually funded by multiple donors, requiring complex governance and collective decision-making processes; (c) They evolve over time as the scale and specific strategies change with donor mix, resources, or a shifting external environment; and (d) They finance interdependent activities at different levels (global, country, local). This presentation will present some examples of analytical tools and sub-criteria which have been developed to apply to evaluation of these GRPPs. For example, a framework for assessing relevance will be presented which analyzes both the vertical relevance (subsidiarity issues) and horizontal relevance (alternative sources of supply). Also, sub-criteria for assessing effectiveness of governance, such as legitimacy, fairness, and accountability will be described.

Expanding Financial Services to the Poor in a Rapidly Changing Technological Environment: the Case of the Consultative Group to Assist the Poorest (CGAP)

Khalid Siraj, Independent Consultant, ksiraj@cox.net

Building on the successful model of Bangladesh's Grameen Bank, the Consultative Group to Assist the Poorest (CGAP) was founded by multiple donors in 1995, with the mandate to promote and support expansion of microfinance for the poor. The program has since been evaluated three times, and each time, management has acted on the findings and adjusted its strategy, clientele, and product mix to better serve its objectives. The program has recently received a large contribution from the Gates Foundation to explore new ways to make use of technology to improve effectiveness. This presentation will first review briefly the findings of the three evaluations and the follow-up actions taken. Then it will illustrate through reference to the most recent evaluation, how some of the recently agreed principles and guidelines for GRPP evaluation were applied.

Measuring and Aggregating Achievement of Global Environmental Benefits From Dispersed Small Grants: The Case of the Critical Ecosystems Partnership Fund

Lauren Kelly, World Bank Group, lkelly@worldbank.org

The Critical Ecosystem Partnership Fund (CEPF) is a global partnership housed in Conservation International (CI) between CI, the World Bank, the Global Environment Facility, the MacArthur Foundation, the Government of Japan, and the French Development Agency that provides grants to NGOs and other private sector partners to protect critical ecosystems (ecological hotspots) in developing countries. During its first phase (December 2000 to June 2007), CEPF awarded approximately 1,000 grants to more than 600 NGOs, community groups, and private sector organizations in 33 countries. Averaging $16-17 million a year, CEPF grants have supported the expansion and enhanced management of protected areas, the promotion of alternative sustainable livelihoods in production landscapes, and environmental education, awareness and capacity building. This presentation will review the arrangements under which CEPF manages, administers, monitors and reports on the achievements of its small-grants program in an effort to understand how the evaluation of a global program must differ from the evaluation of a project administering small grants for the purpose of achieveing global environmental gains. Then it will illustrate through reference to the most recent evaluation, how some of the recently agreed principles and guidelines for GRPP evaluation were applied.

Session Title: Evaluation Capacity: What it is and How to Grow it!

Multipaper Session 290 to be held in Room 109 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

Chair(s):

John Ehlert, University of Minnesota, jehlert@comcast.net

Discussant(s):

Jean King, University of Minnesota, kingx004@umn.edu

Evaluation as a Key Capacity-Building Tool: Sharing Outcomes from a 10-Year Review of New Jersey�s Faith-Based Initiatives

Presenter(s):

Anne Hewitt, Seton Hall University, hewittan@shu.edu

Abstract: Officially launched in 1998, New Jersey�s Office of Faith-Based Initiatives (NJ-OFBI) was one of the first in the nation and has subsequently awarded in excess of $10 million dollars to over 300 faith-based agencies. The NJ-OFBI systematically offers grantees technical assistance (TA) opportunities in order to facilitate successful agency and community outcomes. The primary purpose of this mixed-method evaluation was to establish which types of TA activities were capable of producing desired capacity-building outcomes in faith-based organizations over time. Grant recipients and technical assistance providers responded to a survey or participated in a focus group session. State agency personnel completed individual interviews. Based on survey responses, a continuum of TA opportunities was developed and then aligned with capacity �building activities and agency outcomes. Preliminary results indicate a correlation between amount and scope of TA activity and positive impact on program outcomes and sustainability.

Understanding the Dimensions of Evaluation Capacity

Presenter(s):

Isabelle Bourgeois, University of Ottawa, isabelle.bourgeois@nrc-cnrc.gc.ca

Abstract: The question of evaluation capacity in service organizations has been studied empirically in only a limited fashion in recent years. The actual dimensions of evaluation capacity, or what evaluation capacity might look like in an organization, have not previously been identified systematically in the literature. This study, conducted for the purposes of the author�s doctoral dissertation, sought to identify the key dimensions of evaluation capacity in Canadian federal government organizations as well as the manageable steps required to move from low to high capacity for each of these. The study concluded that evaluation capacity in Canadian federal government departments and agencies can be described through six main dimensions, each one broken down into further sub-dimensions and four capacity levels: exemplary, intermediate, developing and low. These dimensions are fully described in a framework of evaluation capacity, based on three series of interviews with evaluation experts and other government evaluation stakeholders.

�If You Build It They Will Come�: Context, Design, Outcomes, and Lessons Learned from an Internal Evaluation Capacity Building Initiative

Presenter(s):

Stacey Farber, Cincinnati Children's Hospital Medical Center, stacey.farber@cchmc.org

Britteny Howell, Cincinnati Children's Hospital Medical Center, britteny.howell@cchmc.org

Daniel McLinden, Cincinnati Children's Hospital Medical Center, daniel.mclinden@cchmc.org

Stacy Scherr, Cincinnati Children's Hospital Medical Center, stacy.scherr@cchmc.org

Abstract: The purpose of this presentation is to share the context, design, activities, outcomes, and lessons learned from an evaluation capacity building initiative within the Education and Training department at Cincinnati Children�s Hospital Medical Center. As those who highly value the Evaluation Impact standard of the Program Evaluation Standards, we seek to enhance the influence of evaluation, expand client understanding and use of evaluative thinking and techniques, and improve client perceptions of the process and utility of evaluation. This presentation will offer thoughtful strategies (including the constraints and affordances that affect the success of these strategies) for building client capacity to better perceive, understand, and use evaluation for programming and overall business results.

From Supervisors to Staff: Building Capacity for Evaluation and Data-Driven Program Planning in Afterschool Programs

Presenter(s):

Laurie Van Egeren, Michigan State University, vanegere@msu.edu

Heng-Chieh Wu, Michigan State University, wuhengch@msu.edu

Megan Platte, Michigan State University, plattmeg@msu.edu

Nai-Kuan Yang, Michigan State University, yangnaik@msu.edu

Chun-Lung Lee, Michigan State University, leechunl@msu.edu

Beth Prince, Michigan State University, princeern@msu.edu

Celeste Sturdevant Reed, Michigan State University, csreed@msu.edu

Abstract: In the Michigan 21st Century Community Learning Centers (21st CCLC) afterschool programs, a concerted effort has been made to increase program quality by building programs� capacity to make informed, data-driven decisions. Organizational values communicated from the top down create shared goals and visions for a program (Gowdy & Freeman, 1993). Within the context of evaluation, this suggests that staff attitudes about evaluation use will be affected by their supervisors� support for evaluation efforts. In this study, 21st CCLC staff (N = 302) and supervisors (N = 56) completed surveys assessing perceptions of the importance of evaluation and data-driven program planning. Multilevel analyses confirmed that supervisors who were more positive proponents of evaluation and data use had staff who were more likely to embrace data-driven program planning. Moreover, staff who reported greater exposure to evaluation data were more likely to use data-driven program planning and higher quality planning strategies.

Session Title: Addressing Complexity: Conducting Evaluation of a Comprehensive Math-Science Partnership (MSP)

Panel Session 291 to be held in Room 111 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Cynthia Tananis, University of Pittsburgh, tananis@education.pitt.edu

Abstract: The Math Science Partnership of Southwest Pennsylvania (MSP) is a comprehensive partnership funded by the National Science Foundation (NSF) in 2003 to address K-12 science and mathematics reform. This project includes 53 school districts, four institutions of higher education (IHE's), and four regional educational service agencies known as Intermediate Units (IU's). The evaluation investigates the effectiveness of this partnership, its impact on institutional practices and policies at partner educational institutions, changes in math and science instruction, and changes in student course taking and achievement outcomes. This panel offers four distinct but inter-related presentations that describe various aspects of the evaluation and the strategies used to guide the discussion of key elements related to reform sustainability. These papers include a focus on the partnership of three evaluation agencies (through higher education, a private research firm, and a K-12 public entity) to provide a diverse array of evaluation strategies, evaluation planning and responsiveness through logic modeling, the exploration of various individual educator characteristics that are descriptive of high levels of reform implementation, and the variables that appear connected to broader sustainable reform at a systemic level.

Evaluation as a Collective Effort: Diversity to Best Serve Client Needs

Cynthia Tananis, University of Pittsburgh, tananis@education.pitt.edu

More commonly today, funding agents expect potential grantees to partner with other organizations in a common mission and agenda of activity. As the resultant partnerships and collaboratives grow in complexity of interaction and implementation, so too, does the evaluation agenda and the need to manage a vast array of data. Our approach to address the program (tm)s complexity and its need to be responsive to evaluation demands included the development of an evaluation team, drawing on expertise from three different evaluative organizations and perspectives. As a result of this approach, we are able to offer depth of experience and analytical expertise in numerous methodological approaches including case study, interviews, survey research, and extensive quantitative statistical modeling across various participation and student achievement data. While ripe with benefit to and for the client, the inclusion of a large number of evaluators and evaluation perspectives and methods also presents challenges in practice. This paper identifies and explores some of the promising approaches we have used to develop a methodological diversity and appreciation of disparate expertise as well as some of the dilemmas and challenges we have faced as we seek to provide high quality, useable evaluative knowledge for our client and the field at large.

The Continuing Evolution of Logic Modeling for the Math Science Partnership of Southwest Pennsylvania

Valerie Williams, RAND Corporation, valerie@rand.org

Cynthia Tananis, University of Pittsburgh, tananis@education.pitt.edu

John Pane, RAND Corporation, jpane@rand.org

Stuart Olmsted, RAND Corporation, olmsted@rand.org

A critical component of our evaluation of the MSP has been the development of a logic model to communicate the project's operations, theory of action and intended outcomes. Our efforts to develop a logic model for the MSP began in the early proposal stage and have continued to evolve over the first five years of the project. The logic model has not only been a tool for communication, helping to provide a common language and write our annual reports but more importantly, a tool to clarify our evaluation activity. It has served as a mirror to reflect our changing conceptualization of the project, and in so doing, has challenged our understanding of the MSP, requiring us to deal with the difficult "if-then" questions that are inherently embedded in any logic model. As the project has moved into its last funded year (though it expects to operate for at least one additional year), issues related to scaling up and sustainability have moved into a more prominent focus. This focus has implications in prioritization of both program and evaluation activity, potential indicators of outcomes, and how the culminating evaluation report will be organized. This paper explores these latter developments.

A Comparative Study of Individual-Level School Reform Implementation

Keith Trahan, University of Pittsburgh, kwt2@pitt.edu

Cara Ciminillo, University of Pittsburgh, ciminill@education.pitt.edu

Cynthia Tananis, University of Pittsburgh, tananis@education.pitt.edu

The key reform strategy of the Math Science Partnership (MSP) of Southwest Pennsylvania is the professional development (content knowledge, pedagogical knowledge and practice, and leadership skills) of a cadre of teacher leaders (TL's) who are charged with further developing and supporting professional learning communities (PLC's) in their schools to implement math and science reform. These key actors' perceptions of themselves as a competent education reformers are essential to the ultimate scaling up and sustainability of the reform initiative. This presentation reports findings from a research study of educational leaders' perception of self as an educational reformer. Individual characteristics of successful reform include: increased content and pedagogical knowledge, developed leadership skills, improved decision-making, and expanded collaboration. Data are collected through a variety of methods including open-ended interviews, a sustainability survey and the MSP Management Information System database. The study uses a visual heuristic to analyze participants' perception of self as a program reformer. Drawing from individual data, we create a composite of a high fidelity implementer. Findings may shed light on implementation characteristics that support educational reform initiatives, helping to target activities and resources on participants that are more likely to have a greater impact on program implementation.

Exploring Collective Variables Related to Systemic Educational Reform Sustainability

Cynthia Tananis, University of Pittsburgh, tananis@education.pitt.edu

Cara Ciminillo, University of Pittsburgh, ciminill@education.pitt.edu

Keith Trahan, University of Pittsburgh, kwt2@pitt.edu

The Math Science Partnership (MSP) of Southwest Pennsylvania focuses its reform efforts on both teachers and administrators in the K-12 sector. The project assumes the need for both individual agency among educators as well as collective action to provide levers for sustainable and extended systemic change. As the project has moved into its last funded year (though it expects to operate for additional years with extended and additional funding), issues related to scaling up and sustainability have moved into a more prominent focus. The evaluation activities associated with the project are seeking to address these issues through clustering findings from various evaluation activities and strategies to inform the development of a model of interaction. Findings may shed light on implementation characteristics that support educational reform initiatives, helping to target activities and resources on strategies and system pressure points that are more likely to have a greater impact on program implementation.

Session Title: Culturally Competent Organizations: What Are They and How Do You Measure Them?

Panel Session 292 to be held in Room 113 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM

Sponsored by the Multiethnic Issues in Evaluation TIG

Chair(s):

Kien Lee, Association for the Study and Development of Community, kien@capablecommunity.com

Abstract: Presenters will discuss their frameworks for defining and measuring organizational cultural competency and how to use the findings to inform strategies to increase an organization's cultural competency. Three efforts will be presented: an initiative aimed at addressing health disparities by building the cultural competency of a variety of organizations; the development and validation of an instrument to assess the cultural competence of community organizations that serve people with disabilities; and the implementation of a survey and process that benchmarked changes in a foundation's philosophies and underlying assumptions about diversity grantmaking and culturally competent grantmaking practices. The presenters intend to engage the audience in a discussion about the following: What makes an organization culturally competent? What are the best measures of an organization's cultural competency? How do you use the evaluation findings to enhance an organization's structure, internal processes, and effectiveness? What are the challenges of such evaluations?

Lessons Learned From An Organizational Cultural Competency Building Initiative

Chris Armijo, Colorado Foundation for Families and Children, carmijo@coloradofoundation.org

Erica Baruch, Colorado Foundation for Families and Children, ebaruch@coloradofoundation.org

The Colorado Foundation for Children and Families (CFFC) serves as the coordinating agency for The Colorado Trust's Equality in Health initiative. As such, CFFC provides capacity building and evaluation support to the 26 grantees participating in the initiative. CFFC works closely with the initiative's evaluator (Association for the Study and Development of Community) and The Colorado Trust to ensure a seamless support system for the grantees. In this presentation, CFFC staff will share the framework it uses to define cultural competency and to guide its capacity building strategies; how it works with the evaluator to use the evaluator's findings to inform its strategies; and some of the opportunities and challenges encountered so far during the capacity building process.

Evaluating Organizational Cultural Competency: Approach and Lessons Learned

LaKeesha Woods, Association for the Study and Development of Community, lwoods@capablecommunity.com

Kien Lee, Association for the Study and Development of Community, kien@capablecommunity.com

The presentation by the Association for the Study and Development of Community complements the previous one by CFFC. ASDC is the evaluator for The Colorado Trust's Equality in Health initiative. In this presentation, ASDC staff will describe the design and methods used for evaluating the extent to which 26 grantees' organizational cultural competency changed over time, and how the change affected the grantees' capacity to reduce racial and ethnic health disparities. In particular, we will describe how we developed an instrument and a process to assess the grantees' organizational cultural competency. We also will share some of the challenges encountered during the assessment process and how we addressed these challenges, as well as other lessons learned.

Evaluation of a Foundation's Multicultural Transformation

Hanh Cao Yu, Social Policy Research Associates, hanh_cao_yu@spra.com

What does it mean for a foundation to become 'culturally competent'? Social Policy Research Associates (SPRA) explores this question through an organizational learning and evaluation framework that includes a focus on a shared vision, an authorizing culture to support multiculturalism, and appropriate organizational policies, procedures and systems to implement diversity-related grantmaking principles. Based on this framework, SPRA used mixed-methods to track the institutional culture and evolution of a major foundation over a five-year period. In this session, SPRA will present a survey that benchmarked changes in the foundation's philosophies and underlying assumptions about diversity grantmaking, the foundation's commitment to a diversity focus, and culturally competent grantmaking practices. SPRA will share measures that were developed to assess foundation staff's comfort-level in explicitly discussing and promoting diversity; their perceived level of support at different levels of the organization; and their sense of reward and accountability for diversity-related work.

Development, Validation and Use of the Cultural Competence Assessment Instrument for Evaluating Organizational Cultural Competence

Yolanda Suarez-Balcazar, University of Illinois Chicago, ysuarez@uic.edu

Fabricio Balcazar, University of Illinois Chicago, fabricio@uic.edu

Tina Taylor Ritzler, University of Illinois Chicago, tritzler@uic.edu

Human service organizations are under increasing pressure from various stakeholders to become culturally competent, especially in the fields of psychology, counseling, rehabilitation, and health care (e.g., APA Guidelines on Multicultural Education, Training, Research and Practice, 2003; Code of Professional Ethics for Rehabilitation Counselors, 2001). Researchers and providers in multiple disciplines have proposed various skills, knowledge, and abilities that can increase cultural competence in systems, organizations, and programs. Although several conceptual frameworks and models of cultural competence have been suggested in a variety of fields (e.g., nursing and counseling psychology) only a few experimentally validated instruments to assess cultural competence are available. In this presentation, we describe the development and validation of the Cultural Competence Assessment Instrument as well as describe how we use it and a goal setting and follow along process to evaluate and increase the cultural competence of community-based organizations that serve people with disabilities.

Return to Evaluation 2008 Search Results for All Sessions