| Poster 1: Illustrating the Value of McNemar Test of Dependence in Evaluation Studies |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Omolola Adedokun, Purdue University, oadedok@purdue.edu |
| Kasey Goodpaster, Purdue University, scott66@purdue.edu |
| Loran Parker, Purdue University, carleton@purdue.edu |
| J Paul Robinson, Purdue University, jpr@flowcyt.cyto.purdue.edu |
| Wilella Burgess, Purdue University, wburgess@purdue.edu |
| Abstract: This poster provides a step-by step illustration of McNemar Test, using data from the evaluation of an electronic fieldtrip. Program evaluation often involves the examination of pre-post differences in dichotomous items (e.g., yes or no, correct or incorrect, etc.). Although the McNemar Test is often the most appropriate tool for analyzing this type of data, many scholars have noted the inappropriate use of Pearson's Chi-square Test by researchers, including social scientist and evaluators, for the analysis of related or dependent dichotomous variables. The goal of this poster presentation is to promote the use of McNemar Test among evaluators. Using data from a Howard Hughes Medical Institute (HHMI) funded electronic fieldtrip, Purdue zipTripsTM, this presentation will employ screenshots from SPSS to illustrate this statistical method. We will also explain measures of effect sizes for the results and discuss how to interpret and present results. |
| Poster 2: Internal Evaluation of the NIH Interdisciplinary Research Consortium Program |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Irina Agoulnik, Brigham and Women's Hospital, irina@syscode.med.harvard.edu |
| Abstract: Federal funding agencies use several mechanisms to sponsor extramural research projects for collaborative teams of scientists. The NIH Common Fund, for example, supports a series of short-term, exceptionally high impact, trans-NIH programs known collectively as the NIH Roadmap for Medical Research. The Interdisciplinary Research Consortia (IRC) is one such crosscutting NIH program that includes self-assembled teams of investigators involved in integrated interdisciplinary research projects, and supported by a coordinated administrative structure. To assess the promise of this NIH initiative, both National and local evaluations were proposed and performed. Here we discuss the design, methods and outcomes of the local evaluation for one of the nine IRCs, and illustrate the value of this type of evaluation for team science in biomedical research. Moreover, we demonstrate that basic training of the Consortium's administrator in at least one of the Consortium's disciplines benefits both the successful administration of team science at the IRC and the successful completion of the local evaluation. |
| Poster 3: Evaluating the Implementation and Long Term Outcomes of a Government-funded Distributed Medical Undergraduate Program |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Peggy Alexiadis Brown, Dalhousie University, palexiad@dal.ca |
| Abstract: In 2007, Dalhousie University contracted with the Government of New Brunswick and the University of New Brunswick to establish the New Brunswick Medical Education Program (DMEP-NB). This would provide Dalhousie with two 4-year MD campuses, with the first class entering in August 2010. The basis for distributing the program was to develop the education program for NB students in NB; the long-term aim is to increase retention of graduates within their local communities. Additionally, an extensive curriculum renewal process was initiated in 2009 where a number of innovations were integrated into the medical program . Both DMEP-NB and the new curriculum were launched in September 2010. An evaluation framework proposal was developed to address: 1. Evaluation of implementation; 2. Evaluation of the program; and 3. Evaluation of project results. effect academic, administrative, financial, resource and physical plant changes required to accommodate students at all sites. Implementation of these two major curricular innovations - delivery and content of the program- concurrently, has presented a number of challenges. Logic models were used as a way for program developers, evaluators and researchers to coordinate efforts, reduce duplication and ensure program evaluation needs are prioritized and met. |
| Poster 4: Moderators of Intervention Effectiveness: A Systematic Review of Research-based Practices |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Aaron Alford, Battelle, alforda@battelle.org |
| Abstract: Each test of the effectiveness and efficacy of an intervention is a unique intersection of people, activities, history, and setting. These factors may influence both the amount of change one might expect from an intervention and the amount of change observed as a result of that intervention. Because these factors exist simultaneously in an intervention, the estimated impact of that particular implementation of the intervention can be thought of as the pooled impact of the intersection of program, sample, history, setting and service delivery practices. Distinguishing the unique contribution of each input from the effectiveness of the intervention is difficult, if not impossible, within a single test. This review of meta-analyses identified and assessed the importance of moderating factors, referred to as practices, on the effectiveness of programs intended to improve behavioral and substance abuse outcomes for children. Findings are presented with a discussion of their application to evaluation practice. |
| Poster 5: The Winston Salem-Forsyth County Schools (NC)/University of North Carolina at Greensboro (UNCG) Teacher Residency Program Evaluation- Project ENRICH at UNCG-Teacher Quality Partnership (TQP) Grant |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lynn Amwake, University of North Carolina, Greensboro, camwake@serve.org |
| Betty Epanchin, University of North Carolina, Greensboro, bcepanch@uncg.edu |
| Christina K O'Connor, University of North Carolina, Greensboro, ckoconno@uncg.edu |
| Abstract: The Project ENRICH (Educational Network for Renewal, Innovation, Collaboration and Help) is a partnership between UNCG and Winston Salem/Forsyth County Schools for the 'purpose of simultaneous improvement of teacher education and PK-12 education.' The Teacher Residency Program is a key partnership component. This teacher residency apprentice model incorporates graduate education (M.Ed. in 18 months) with classroom experience and is designed for individuals with a bachelor's degree and content appropriate to become certified as an ESOL, mathematics, science, or special education teacher. Selection to participate is through an application and interview process and requires a commitment to spending four days a week for one academic year with a master teacher in a high-need school and one day per week engaged in educational seminars, observations and graduate classes. SERVE Center was selected by the UNCG School of Education to serve as the external evaluator for this multi-year, multifaceted Teacher Quality Partnership (TQP). |
| Poster 6: Assessment of Implementation of the Research of AIDS Cases Associated With Blood Transfusion in Brazil |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Walkiria Andreev, Health Secretary of Brasilia, wal.andreev@hotmail.com |
| Marly Cruz, Oswaldo Cruz Foundation, marly@ensp.fiocruz.br |
| Vera Luzia, Oswaldo Cruz Foundation, vera@ensp.fiocruz.br |
| Abstract: This study aimed to evaluate the implementation of the research of AIDS cases associated with blood transfusion in three Federal Units (FU) of Brazil. A normative evaluation was conducted through a multiple case study to estimate the degree of implementation of the research of AIDS cases associated with blood transfusion services in these locations Epidemiological Surveillance. We used an array of trial exploring the concept of quality as a dimension of assessment categorized into levels of compliance, availability and technical and scientific quality. The investigation of cases of AIDS associated with transfusion proved to be partially implemented in Distrito Federal, not implemented in Mato Grosso and Paraná. Thus, it became necessary to revise the technical and operational strategies in order to improve implementation in the Epidemiological Surveillance, particularly in relation to activities related to obtaining more consistent data. |
| Poster 7: Evaluation of the Colorado Oral Health Surveillance System; Data Uses, Data Users, and Future Directions |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Theresa Anselmo, San Luis Obispo Health Agency, tanselmordh@netscape.net |
| Mathew Christensen, State of South Dakota Public Health, mathew.christensen@state.sd.us |
| Abstract: Oral health surveillance is a newer application of traditional disease surveillance rooted in controlling and preventing population-based infectious diseases. However, the application of surveillance to oral disease and the data's usefulness to public health practitioners has not been evaluated. This qualitative evaluation sought to determine basic user characteristics, data uses, and data gaps. Also the evaluation sought determine the usefulness of the Colorado Oral Health Surveillance System and how effectively oral health surveillance data are being used for the prevention and control of oral disease in Colorado. Two concepts 'power users' and 'critical mass' emerged to describe groups of data users that have the potential to play a significant role in oral disease prevention and control in Colorado. The identity of primary data users and their respective program actions must be understood to tailor data analysis, data products, and to ensure that oral health data are useful and fully engaged in prevention and control activities. |
| Poster 8: The Rice Breeding Program in Argentina: an Evaluation Challenge |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Patricia Arnozis, Instituto Nacional de Tecnología Agropecuaria, parnozis@cnia.inta.gov.ar |
| Alberto Livore, Instituto Nacional de Tecnología Agropecuaria, alivore@yahoo.com.ar |
| Abstract: The involvement of stakeholders in research programs is rare in Argentina complicating evaluation. In this paper, we present the case of the Rice Breeding Research Program at INTA. In this unique and successful experience a variety of stakeholders were involved in defining, contributing to or funding it. A reverse sequence of involvement took place in which stakeholders had input to the research agenda and execution. Remarkable results followed. The success of the program was due to a very good match of interests and expectations of all parties, both private and public, which grew into a social network of significant size. The Agency now faces a new challenge to assess the full value of these technological and organizational innovations with a holistic view of the whole process the value of which transcends the value of the individual products, as good as these were. |
| Poster 9: Evaluation of Aids Assistance in Brazil |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rachel Baccarini, Ministry of Health of Brazil, rachelbaccarini@hotmail.com |
| Maria Ines Nemes, University of San Paulo, |
| Elizabeth Moreira, Oswaldo Cruz Fondation, bmoreira@ensp.fiocruz.br |
| Abstract: The brazilian response to aids epidemic has been internationally recognized as effective, comprehensive and timing. Universal access to care and treatment, including antirretrovirals and services, has been a milestone of Brazilian National Programme. The number of facilities has increased along the years in all federal states, and nowadays they are more than 700 hundreds outpatient's services, making out a heterogeneous group in size, complexity and number of patients. A significant part of those have less than a hundred patients under accompaniment and recent researches show the tendency toward less quality of assistance in small services. We intend to evaluate the assistance of HIV/aids outpatients' implementation in the Brazilian Public Health System, focusing on structure and process, and caring out a qualitative study in three of those small services through a contrasting case-study with a convenient sample. |
| Poster 10: Developing and Sharpening the Tools in Your Toolbox: Strategies for Creating Tools to be Used in a Collaborative Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Eun Kyeng Baek, University of South Florida, ebaek@mail.usf.edu |
| SeriaShia Chatters, University of South Florida, schatter@mail.usf.edu |
| Abstract: A public, US university requested an evaluation to determine the level of satisfaction of students in a graduate program. The results of the evaluation are utilized to improve the overall quality of the program, course work and facilities of the graduate program. The evaluation team took charge to design an evaluation that defined evaluation questions, designed data collection, and developed data analysis. A critical component of the evaluation design was the development of a tool to measure students' perceived satisfaction, due to the lack of valid quantitative measures available to measure perceived satisfaction within a graduate program. The evaluation team collaborated with students, faculty, and expert personnel to develop an instrument. Key components of the perceived satisfaction were identified and used to develop an instrument. The purpose of this poster is to discuss the development and validation of a measure of perceived student satisfaction to aid in the collaborative evaluation. |
| Poster 11: Assessing Needs of Probationary Students at a Private Liberal Arts University |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Patrick Barlow, University of Tennessee, Knoxville, pbarlow1@utk.edu |
| Jason Kelly, Saint John's University, jkelly@csbsju.edu |
| Abstract: The proposed poster will report on methods and findings from a needs assessment of an academic advising department at a private, liberal arts college. The purpose of the present evaluation has been to develop a profile of the student who is most in danger of going on academic probation at some point during his or her career at the institution. Using notes from advising sessions matched to extensive archival data, the evaluation aims to identify past, present, and expected student behaviors most likely to attribute to future probation. Furthermore, these results will be used to highlight areas in which the department may create or improve upon services to provide a targeted approach towards reducing probation rates. |
| Poster 12: Structural Evaluation of Public Policies: The Case of the Evaluation of Environmental Policy of Municipality of Cabo de Santo Agostinho, State of Pernambuco, Brazil |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Andre Barros, Souza Barros Consultoria em Sustentabilidade, andrebarros@souzabarrosconsultoria.com |
| Karla Silveira, Souza Barros Consultoria em Sustentabilidade, karlasilveira@souzabarrosconsultoria.com |
| Marx Prestes, Universidade Federal de Campina Grande, marx@deag.ufcg.edu.br |
| Vitoria Regia Fernandes Gehlen, Universidade Federal de Pernambuco, vicgehlen@yahoo.com.br |
| Abstract: To analyze a public policy is to establish a process of investigation into what a political system has produced throughout its activities. The objective of this proposal is to present a methodological approach to evaluate the public policies based on essential elements of a public policy: a) the legal base, 2) institutional apparatus, 3) planning and management, 4) social control. This approach allows us to analyze and assess the maturity level that public policy has achieved throughout its evolution. For each element of the analysis are identified evaluation indicators that allow reviewers to gain a contextualized view of the results of the public policy in question. This methodological approach was applied to the analysis and evaluation the Environmental Policy in the municipality of Cabo de Santo Agostinho, Pernambuco, Brazil, and has proved useful as an evaluation method that analyzes public policy in its structural elements. |
| Poster 14: Evaluating Fidelity of Implementation of Response-to-Intervention (RTI) in One Rural Elementary School |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lisa Benjamin, Georgia State University, elissabenjamin@yahoo.com |
| Sheryl Gowen, Georgia State University, sgowen@gsu.edu |
| Abstract: The new IDEIA (2004) mandates regarding the implementation of Response-to-Intervention (RtI) present challenges for general education teachers. The law dictates the implementation of Response-to-Intervention, which requires the application of a pyramid of interventions for students failing to make adequate yearly progress in response to general education programs. Response-to-Intervention regulations redefine general education teacher roles, increase responsibilities regarding instructional interventions for at-risk learners, and change the process used to determine qualification for specific learning disability (SLD). This study evaluates the fidelity of implementation of RtI in a rural public elementary school. The study also examines teacher descriptions of the influence policy implementation has on instructional practices for at-risk students. Data collection methods include structured and unstructured interviews, videotaped classroom observations, Teacher Performance Record data, lesson plans, and relevant RtI artifacts to advance understanding of RtI implementation in relation to the particular research site and study participants. |
| Poster 15: An Evaluation of Strengths, Challenges, and Opportunities for Improvement at a Midwestern Substance Abuse Treatment Facility |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amanda Bolton, University of Missouri, St Louis, amandagbolton@gmail.com |
| LaMarcus Bolton, Saint Louis University, boltonlr@slu.edu |
| Abstract: As a result of drug offenders overcrowding American prisons, criminal justice policies have realigned to a more rehabilitative and reintegration approach to keep such offenders out of prison. Academics have also shifted their attention to the effectiveness of community-based programming. One such Midwestern program provides outpatient community drug treatment to men in the criminal justice system. The purpose of this study was to evaluate the strengths, challenges, and opportunities for improvement in light of program counselors. Data for this project came from 36 qualitative in-depth interviews completed between February and April 2010. Policy implications will be discussed. |
| Poster 16: Incorporating Diverse Stakeholder Values Into an Enhanced Evaluation Design of an Individual Prevention Service Process |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shanelle Boyle, EVALCORP Research & Consulting, sboyle@evalcorp.com |
| Lisa Garbrecht, EVALCORP Research & Consulting, lgarbrecht@evalcorp.com |
| Abstract: As evaluators with a restricted budget, it is challenging to design an initial evaluation that addresses diverse questions posed by multiple stakeholders who value different program components and outcomes. This was the case when EVALCORP partnered with the Riverside County Department of Mental Health - Substance Abuse Program to design and conduct the first formal evaluation of the Brief Risk Reduction Interview and Intervention Model (BRRIIM), an innovative prevention process that screens and educates individuals at high risk for alcohol and other drug problems. The program's context and original budget constraints made it difficult to develop an evaluation design that met the varying needs and expectations of stakeholders. This presentation will provide an overview of the strategies EVALCORP used to resolve contextual issues and help the program secure additional funding for an enhanced evaluation plan that incorporates a more rigorous design and broader range of stakeholder values. |
| Poster 17: Using Indicators for Research Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michael Braun, Vietnam Science & Technology Evaluation Center, michael@most.gov.vn |
| Neville Reeve, European Commission, neville.reeve@ec.europa.eu |
| Abstract: Research and technology indicators can provide substantial information about an Innovation System's or program's status and performance for evaluations. There is no lack of such possible indicators. But still, many evaluations don't use indicators consistently and don't exploit their potential because auf limited data availability, methodological and behavioral restrictions. Unlike traditional indicators, evaluations often require ad-hoc indicators which provide evidence for the assessment of complex phenomena. These combine different data and suitable proxies related to the observed feature for the quantitative measurement of specific features of reality which cannot be directly observed. This imposes limitations on indicator generation and use. Building on recent evaluation experience with Europe's Framework Programmes, this presentation discusses the methodological state-of-the-art, proposes an indicator framework and identifies success factors for indicator application. http://ec.europa.eu/research/evaluations/index_en.cfm Braun et al., Tools and Indicators for Community, Research Evaluation and Monitoring, Study for the European Commission, Brussels, 2009, http://ec.europa.eu/research/evaluations/pdf/archive/fp6-evidence-base/evaluation_studies_and_reports/evaluation_studies_and_reports_2009/tools_and_indicators_for_community_research_evaluation_and_monitoring.pdf |
| Poster 18: Evaluation Capacity Building in Slovenian Schools |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Mateja Brejc, National School for Leadership in Education, Slovenia, mateja.brejc@solazaravnatelje.si |
| Nada Trunk Sirca, University of Primorska, nada.trunk@guest.arnes.si |
| Abstract: The aim of the paper is to provide an insight to training program for self-evaluation capacity building in Slovenian schools. In pilot phase of developing and implementing the training program 84 primary and secondary schools were involved. During 18 months teachers participated in 7 one day seminars covering topics such as: students' achievement as a self-evaluation focus, improvement planning, defining »measures« and gathering data, the role of stakeholders in self-evaluation, reflecting and reporting on school self-evaluation process. Between each seminar intermediate activities (applying theory into practice) were taking place in schools. A »train the trainers« model has been developed by which self-evaluation teams were trained to run seminars and self-evaluation processes in their own schools. Different aspects of training program have regularly been monitored. Evaluation at the end of the program has been focused on self-evaluation capacity building. At the conference some prevailing issues will be presented in more depth. |
| Poster 19: Using Social Network Analysis to Inform Community Capacity Building in a National Science Foundation Funded Science, Technology, Engineering, and Math Project (NSF GSE/EXT 0832913) |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Donna-Jean Brock, Evaluation Consulting Services Inc, djbrock.ecs@cox.net |
| Sandra B Schneider, Virginia Tech University, saschne3@vt.edu |
| Lydia I Marek, Evaluation Consulting Services Inc, evalconsultecs@yahoo.com |
| Peggy S Meszaros, Virginia Tech University, |
| Abstract: The Appalachian Information Technology Extension Service (AITES) project's overarching goal is to increase girls' interest in jobs requiring information technology (IT) skills through the development of community capacity. Developing community capacity is accomplished by training Community Cohort Teams (CCT) which consist of teachers, school counselors, and Cooperative Extension agents. These team members are tasked to train other key community leaders (Secondary Stokers), who surround the girls (i.e., other teachers, counselors, and parents). Through the transfer of knowledge between CCT Members and Secondary Stokers and subsequently from these Secondary Stokers to the broader community (Tertiary Stokers) community capacity is being built. Theories and tools from Social Network Analysis (SNA) serve to visualize the many vehicles and routes taken in building this community capacity. SNA also provides an additional lens to make sense of project diffusion, as well as identify best practices and appropriate and effective course corrections. |
| Poster 21: Sharing Evaluation Data with Researchers and Evaluators to Inform Decision Making During the Program Planning and Implementation Stages |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Charles Brown, Centerstone Research Institute, charles.brown@centerstoneresearch.org |
| Catherine Sewall-Martin, Centerstone Research Institute, catherine.sewall@centerstone.org |
| Tara Kirkpatrick, Centerstone Research Institute, tara.kirkpatrick@centerstone.org |
| Abstract: This paper discusses the value of sharing data among researchers and evaluators to inform decision making regarding new and existing programs in the field of Substance Abuse and Mental Health. Data sharing is critical in the development of programs and can have important implications for refining strategies during the program implementation stage. This paper describes an example of how a collaborative partnership between the originator of the data and secondary user interface with each other to gain a greater understanding of the target population and developed intervention strategies for program improvement. The pros and cons of data sharing from an evaluator's perspective will be shared and steps evaluators can take to inform decision making with new and existing programs will be presented. |
| Poster 20: Defining Value Through Peer-led In-depth Interviews |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lisanne Brown, Louisiana Public Health Institute, lbrown@lphi.org |
| Mary Freyder, Tulane University, mfreyder@tulane.edu |
| Abstract: CHAT NOLA is a peer-driven HIV prevention program for Black youth 15-22 living in New Orleans aimed at reducing multiple concurrent partnerships, increase condom use, and increase knowledge of HIV. Semi-structured qualitative interviews were conducted by peers in the third person to provide relative anonymity, comfort and trust. Following interviews, peer researchers met with evaluation staff for recall interviews. Storytelling was an important aspect of interviews. In primary interviews, interviewees were encouraged to share stories. In the recall interviews, peer-researchers recount the interview as a story, and the evaluation staff probed for greater depth as needed. Interviews were coded and reviewed for theme analysis with Ethnograph 5.0. Evaluation staff identified several preliminary themes. Based on the preliminary analysis, we designed a three-tiered analysis workshop for the team, which produced results in the form of a Peer Archetype Illustration and Narrative and a Suite of Micro-Narratives. |
| Poster 22: Social Network Analysis and Disaster Resilience in Miami-Dade |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michael Burke, RTI International, burkemic99@hotmail.com |
| Abstract: Social network analysis has been frequently used to identify organizational challenges and constructive approaches to improving organizational processes. Only rarely has social network analysis been applied to disasters and the network of organizations that help communities prepare for, respond to, and recover from a disaster. An example of a project to conduct a needs assessment and examine the network of organizations in Miami-Dade County will be discussed. The basic network structure representing the relationship of Miami-Dade organizations will be presented and methodological challenges will be examined. Subsequently, a report designed to report back to faith-based and community organizations will be presented. Contained within the report are the results of the network analysis and recommendations based on social network and disaster response literature. |
| Poster 23: Ontological and Deontological Continuity Underlying Approaches to Addressing Evaluands |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jason Burkhardt, Western Michigan University, jason.t.burkhardt@wmich.edu |
| Lee Balcom, Western Michigan University, |
| Abstract: Theoretically, many design options are available to the modern practicing evaluator (Bickman & Rog, 2009). These tools can be complementary or adversarial in their relationship to each other. How can evaluators in contemporary practice understand how these choices relate to the philosophical, ethical, and practical underpinnings of the Evaluation field? This poster will clarify the relationship between philosophy, ethics, context, theory, and practical applications in evaluation. This poster is a revision of a prior poster presentation that includes feedback given by AEA conference attendees during the poster session in 2010. This poster presentation will be interactive, and viewers will have the opportunity to provide feedback, with their comments gaining acknowledgement in the final product. |
| Poster 24: Overcoming the Scientist-Evaluator Communication Barrier to Improve Data Quality And Improve the Utility of Evaluation Findings |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Erin Burr, Oak Ridge Institute for Science and Education, erinburr1@gmail.com |
| Samuel Held, Oak Ridge Institute for Science and Education, sam.held@orau.org |
| Abstract: Evaluators are often trained in specific fields (e.g., psychology, sociology) which include coursework or training in evaluation; however, programs they are tasked to evaluate do not necessarily match their subject matter expertise. When evaluating a highly technical program, evaluators are likely to encounter a great deal of professional jargon. Unfortunately, budgets don't always allow for subject matter experts on the evaluation team and clients may not have the time to provide assistance, so the accuracy and utility of the evaluation can suffer. This poster addresses challenges we experienced while interacting with scientists during our evaluation of the Oak Ridge Leadership Computing Facility (OLCF), a Department of Energy super-computing facility. We share how we overcame these challenges during each step of the evaluation process and maintained the standards of accuracy and utility. A summary of what did and didn't work for us is provided as well as suggestions for other evaluators. |
| Poster 26: Deconstructing the Culture of Sexuality to Evaluate Sexual Reproductive Health Services in Grenada |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Aaron Buzard, St George's University, aaronbuzard@gmail.com |
| Lynne Fakeye, St Catherine University, lfakeye@sgu.edu |
| Cecilia Hegamin-Younger, St George's University, chyounger@mac.com |
| Jacqueline Lewis-Harris, University of Missouri, St Louis, antjharr@jinx.umsl.edu |
| Christine Richards, St George's University, crichards@sgu.edu |
| Abstract: Grenada Planned Parenthood Association (GPPA) was designed to empower women to exercise their choice and rights in regards to their sexual and reproductive lives. The evaluation of GPPA was threefold: (1) assess the perception of services of existing adolescent and young adult clients aged 15 to 24 years old; (2)understand why adolescents and young adults do not access their services; and (3) evaluating the administrative processes. This presentation will discuss the importance of understanding the culture of sexuality to conduct and interpret the evaluation and evaluation results. |
| Poster 25: Evaluation of Grenada Planned Parenthood Association (GPPA) |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Aaron Buzard, St George's University, aaronbuzard@gmail.com |
| Cecilia Hegamin-Younger, St Catherine University, chyounger@mac.com |
| Abstract: Grenada Planned Parenthood Association (GPPA) was designed to empower women to exercise their choice and rights in regards to their sexual and reproductive lives. The evaluation of GPPA was threefold: (1) assess the perception of services of existing adolescent and young adult clients aged 15 to 24 years old; (2)understand why adolescents and young adults do not access their services; and (3) evaluating the administrative processes. This presentation will discuss the importance of understanding the culture of sexuality to conduct and interpret the evaluation and evaluation results. |
| Poster 27: Proposed Framework To Evaluate Communities of Practice Working With Knowledge Management Approaches |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kemly Camacho, University of Costa Rica, kemly@sulabatsu.com |
| Abstract: Different networks and communities of practice -from academia or from civil society- are using knowledge management approaches to build knowledge, share information and development collaborative actions. There are always questions about how effective the knowledge management is to build communities, to create new knowledge, to impact and change conditions for the final population. The evaluator who is proposing this paper has developed a framework to analyze the results, effects and impacts of the knowledge management approaches to build communities of practice.The framework is based in 5 evaluation case studies developed by the evaluator during the last two years. The evaluation framework analyzes topics like: a. Information management b. Knowledge sharing c. Knowledge management facilitation d. Transformation in the community e. Knowledge production f. ICT uses f. Organizational culture g. Cost |
| Poster 28: A Study of Culturally Integrated Wellness Using Behavioral Trending Analysis |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Scott Carlson, Sonoran Research Group, scarlson@sonoranresearchgroup.com |
| Abstract: For the better part of a century the disease model of addiction has dominated medical and scientific studies in the analysis of addicts and alcoholics. While this model has a place in the overall study of addiction, it fails to address societal factors that a cultural and environmental model can bring to light. Culturally Integrated Wellness seeks to identify behaviors that can lead to dependence on alcohol and other drugs and to encourage addicts to adopt positive life skills by engaging in cultures and environments characterized by non-use. |
| Poster 29: Keeping Educators in the Loop: Developing Resources for College and Career Readiness |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shauna Clarke, ICF Macro, sclarke@icfi.com |
| Amy Newell, ICF Macro, anewell@icfi.com |
| Edith Stevens, ICF Macro, estevens@icfi.com |
| Abstract: Preparing high school students to be college and career ready depends a great deal on educators having the technological skills and resources they need to help students improve their achievement. In Maryland, the College and Career Readiness Grant Consortium is a model for how states can create professional development materials and resources while giving educators the opportunity to play an integral role in each stage of the development process. This approach helps educators to be more effective, which ultimately makes students more successful. As the evaluator of this project, ICF Macro has facilitated this process by providing guidance on program implementation and impact, by being an agent for identifying and communicating stakeholders' needs, and by monitoring how the project addresses those needs. |
| Poster 31: Build it and They May Come: The Struggles of Implementing an Integrated Primary Care Clinic Within A Mental Health Center |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deborah Cohen, Community Mental Health Center Inc, debbie.cohen@cmhcinc.org |
| Abstract: There is an assumption that a program will be successful just because a there is a documented need for the service. The role of the evaluator can be difficult when it is discovered that a program is struggling, and difficult decisions need to be made to ensure that the program has the opportunity to become successful. This poster will display how information collected during the first year of the implementation of an integrated primary care clinic was used to assist program staff in changing the trajectory of the program. The presenter for this poster will display how an internal evaluator utilizes empowerment evaluations approaches to work with program staff to make data informed decisions to improve program implementation (Fetterman & Wandersman, 2004). |
| Poster 30: Empowerment Evaluation: Applicability and Adaptation for the Criminal Justice System |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Derek Cohen, University of Cincinnati, cohendk@mail.uc.edu |
| Abstract: While criminal justice scholars have yet to accept a unified theory of system function, several which have been put forth emphasize the need for criminal justice actors to be locally responsive. 'Empowerment Evaluation,' first laid out by David Fetterman in 1996, is an evaluation paradigm that emphasizes data gathering, analysis, and policy formation at the practitioner/client level. By involving the street-level worker in the evaluation and policy formation process, empowerment evaluation is thought to increase participant buy-in, the adoption of best practices, and the targeting of pragmatic outcomes. This exploratory piece seeks to highlight where in the criminal justice system an empowerment evaluation model may be used, necessary changes to the model that must be made in order to maintain the goals of criminal justice, and potential advantages agencies may enjoy when adopting an empowerment evaluation model. |
| Poster 32: Assessment of Data Quality Control System for Laboratory Tests (SISCEL) as Source for Identification of AIDS in Children, Brazil |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Leidijany Costa Paz, Brazil Federal District Department of Health, leidipaz@gmail.com |
| Vera Luzia, Center for Pharmaceutical Care, vera@ensp.fiocruz.br |
| Carmen Barros Correia Dhalia, Health Department of Pernambuco State, cdhalia@gmail.com |
| Abstract: This study assessed the quality of the data from SISCEL as a source for tracking AIDS cases in children. It focused on processes evaluation, aiming to improve the system and the epidemiological surveillance (ES) of AIDS in Brazil. A cross sectional data collection approach was used. Judgment criteria prioritized five dimensions: accuracy, comparability, use, opportunity, and validity (positive predictive value - PPV). Seven data collection strategies were used in order to calculate the indicators. Data revealed that the system has good accuracy, and that the data are comparable and available to the epidemiological surveillance staff. The PPV of the system, for ES ends, has shown itself on the critical threshold for the direct use of such data in the calculation of epidemiological indicators (50.0%). Its use as a tool to decrease the underreporting, should consider only the laboratory-confirmed HIV infections, which will improve the PPV of the system and consequently its quality for ES. |
| Poster 33: Challenges and Barriers to Tuberculosis Follow-up and Reporting for Immigrants and Refugees |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kendra Cuffe, Centers for Disease Control and Prevention, kkn2@cdc.gov |
| Deborah Lee, Centers for Disease Control and Prevention, dlee1@cdc.gov |
| Awal Khan, Centers for Disease Control and Prevention, aek5@cdc.gov |
| Abstract: In the United States, the tuberculosis (TB) follow-up component of the Electronic Disease Notification (EDN) system serves as a method providing state health departments with an electronic system to record and evaluate the outcome of domestic follow-up, and serves as a surveillance component of the EDN system. Historically, follow-up reporting has been not optimal. During 2010, median reporting was less than 50%. The goal of this project was to perform a formative evaluation in which needs and challenges were determined by interviewing low and high reporting health departments. A gap analysis helped identify challenges and barriers to the follow-up process and to follow-up reporting. Health departments reported several key challenges and barriers including the limitation of resources, absence of a guidance document and data dictionary, and lack of training by the Centers for Disease Control and Prevention (CDC). Based on the results, recommendations are made to increase TB follow-up reporting. |
| Poster 34: Communicating Potential Health Risks to Worker Populations: Lessons Learned from an Evaluation of a Worker Notification Program |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Andrew Davis, Battelle Memorial Institute, davisah@battelle.org |
| Mary Kay Dugan, Battelle Memorial Institute, dugan@battelle.org |
| Amy Mobley, National Institute for Occupational Safety and Health, frj4@cdc.gov |
| Everett Lehman, National Institute for Occupational Safety and Health, exl7@cdc.gov |
| Abstract: Through its worker notification program, the National Institute for Occupational Safety and Health (NIOSH) works to communicate study results to workers and promote occupational health by informing workers of findings from epidemiological studies. Recently, Battelle assisted NIOSH in conducting a formative evaluation of its worker notification program. The evaluation was designed to evaluate (and improve) the content of notification materials and the dissemination strategies used. Focus groups with workers from four occupational cohorts involved in NIOSH studies, as well as interviews with other key stakeholders, were conducted to gather input on both the quality and effectiveness of message content and message delivery modes. Overall, results of the focus groups and interviews indicated that efforts to communicate health risks and target health communication messages to various worker populations require collaboration among scientists responsible for 'translating' the study and risks, workers, and other impacted stakeholders to be more effective. |
| Poster 35: Constructing a Gender Disparities Assessment for Program Planning Purposes |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ann Del Vecchio, Alpha Assessment Associates LLC, ann.delvecchio@gmail.com |
| Justina Trott, Women's Health Services Santa Fe, jtrott@whssf.org |
| Abstract: Coalitions for Healthier Communities (CHC) federal funding has prompted the development of gender disparities data collection, analysis, and report development. This process was designed to provide coalitions with data related to gender disparate health outcomes. Planning a structure / model to address these disparities is based on a comprehensive needs assessment and the gender disparities analysis. The Guidelines for Gender-based Analysis of Health Data for Decision Making developed by the Pan American Health Organization was used by the New Mexico Coalition for a Healthier Community to develop a New Mexico gender disparities report to inform a strategic plan. This gender disparities report is one of 16 reports developed for CHC funding from the Office of Women's Health, Department of Health and Human Services. Qualitative and quantitative data findings indicated that New Mexico women have higher rates of anxiety, depression and domestic violence health related problems than the state's men. |
| Poster 36: Meta-Evaluation: The World Food Programme Evaluation Policy and Practice |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lauren Denofrio-Corrales, University of Illinois at Urbana-Champaign, denofrio@illinois.edu |
| Lopez Cintron Julio, University of Illinois at Urbana-Champaign, jrlopez@illinois.edu |
| Abstract: Using a meta-evaluative framework, we present a comprehensive assessment of the current, normative evaluation strategies of a global food assistance program, the World Food Programme (WFP). The purpose of this meta-evaluation is to clarify the current scope and orientation of and resources devoted to evaluating food assistance programs within WFP. Specifically, this investigation seeks to determine the orientation towards learning and use of evaluations conducted within this field. This meta-evaluation synthesizes findings as to what constitutes best practices for evaluation in global humanitarian programs with diverse constituents seeking to learn about and improve programs and proposes a new set of guidelines to evaluate them. |
| Poster 37: Evaluating the Use of Skilled Volunteers versus Paid Staff in Delivering Direct Services in Nonprofit Organizations |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ellen Denzen, National Marrow Donor Program, edenzen@nmdp.org |
| Katy Engelby, National Marrow Donor Program, kengelby@nmdp.org |
| Tammy Payton, National Marrow Donor Program, tpayton@nmdp.org |
| Kate Pederson, National Marrow Donor Program, kpederso@nmdp.org |
| Abstract: In an effort to be more financially responsible, nonprofits are moving towards incorporating more skilled volunteers into direct client services as the types of tasks volunteers are asked to do are becoming increasingly complex. This paper presents the finding from a pre/post-test evaluation design conducted via internet survey. The objective was to compare the effectiveness of paid consultants versus skilled volunteer ambassadors in an outreach program intended to raise awareness and utilization of patient education materials and resources among medical professionals. In this presentation, we will describe how outcomes and experiences met or diverged from program expectations. We will also describe how participant feedback was incorporated into a volunteer ambassador training program. Finally, we will share lessons learned that could benefit other nonprofits considering or incorporating skilled volunteers in direct service roles. |
| Poster 38: A Meta-Analytic Evaluation of the Effectiveness of Targeting Mediating Factors to Reduce Violent and Aggressive Outcomes |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| James Derzon, Battelle, derzonj@battelle.org |
| Aaron Alford, Battelle, alforda@battelle.org |
| Abstract: Intermediate risk and protective factors are seen as potential targets for reducing violence and aggression in schools. Measures of these intermediate outcomes are often reported as evidence of change in distal outcomes in school-based violence interventions. While this strategy is certainly valid, given strong causal relationships, the strengths of the associations between the mediating variables and the violent behavior are generally poorly understood and based on means driven data or correlational evidence. This presentation will present meta-analytic evidence of how the associations between intermediate outcomes and distal outcomes are reported in school-based evaluations of violence interventions and their relative strengths. This evidence will also be used to further explore the relationship between changes in risk/protective factors and in distal behavioral outcomes with an emphasis on informing study design, analysis and reporting. |
| Poster 39: Evaluating an HIV and Substance Abuse Intervention for Homeless Youth |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Mona Desai, Children's Hospital Los Angeles, mdesai@chla.usc.edu |
| Leslie Clark, Children's Hospital Los Angeles, lclark@chla.usc.edu |
| Abstract: This presentation describes a mixed method evaluation of a youth development approach to substance abuse and HIV prevention among homeless youth ages 18-24. Homeless youth at drop-in centers, shelters and transitional living programs were recruited to participate in the 9 session weekly curriculum that focused on helping them envision a future for themselves and understand how their behavior know can impact that future. Baseline, exit and 3 month follow-up surveys were completed with participants to determine changes in sexual risk and substance use behavior over time. In addition focus groups and interviews were done with participants to further understand the impact of the intervention as well as what changes needed to be made. We will discuss the statistically significant changes from baseline to follow-up and share youth comments about the impact the intervention has had on them. The session will also discuss tracking strategies for highly mobile homeless youth. |
| Poster 40: Child Welfare Cross-Site Evaluation: The Challenge of Assessing Two Dozen Unique Grantees |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jennifer Dewey, James Bell Associates Inc, dewey@jbassoc.com |
| Kate Lyon, James Bell Associates Inc, lyon@jbassoc.com |
| Grace Atukpawu, James Bell Associates Inc, atukpawu@jbassoc.com |
| Chi Connie Vu, James Bell Associates Inc, vu@jbassoc.com |
| Abstract: This paper presents process evaluation results from the cross-site evaluation of 24 Family Connection Discretionary Grants that were awarded in September 2009 with funds authorized by the Fostering Connections to Success and Increasing Adoptions Act of 2008 (Public Law 110-351). These grants support demonstration projects to help reconnect family members with children who are in or at risk of entering into foster care in four diverse program areas: 1) Kinship Navigator; 2) Intensive Family-finding; 3) Family Group Decision-Making; and 4) Residential Family Treatment. Nine grantees are implementing multiple areas. Inputs, activities, outputs and proposed outcomes are documented through grantee applications, reports, and local programmatic and evaluation documents, along with individual and group interviews conducted during site visits to each grantee. The program theory of change structures results by: participant and program characteristics; program design, implementation and maintenance; modifications; and system collaboration. |
| Poster 41: Investigating the Amazing You Through Technology: Evaluation of Informal Learning For Health Awareness and Education within a Working Museum Environment |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Wendy Dickinson, Ringling College of Art and Design, wdickins@ringling.edu |
| Bruce Hall, University of South Florida, bwhall@tampabay.rr.com |
| Dave Conley, Museum of Science and Industry, dconley@mosi.org |
| Abstract: Informal learning environments encourage learning in ways that traditional, classroom-based environments may not: different learning styles are accommodated, and learners absorb information at their own pace, constructing their own sense of relevance (Melber & Abraham, 1999). Therefore, experiences at science museums make a profound impact in public education, and these unique environments provide rich opportunities for learning (Abraham-Silver, 2006). Evaluation activities focused on eight integrated, mixed-reality exhibits based on health science and the human body, collectively referred to as Investigating the Amazing You Through Technology (IAYTT), located in a contemporary science museum. We examined characteristics of museum visitors to provide recommendations for more meaningful learning experiences, utilizing unobtrusive observations, demographic/time measures; visitor interviews; and retrospective survey instrumentation to evaluate visitor experiences and learning. By developing appropriate, effective evaluation methods, we crafted recommendations for improvement to ensure the highest quality visitor experience (thus facilitating learning) within the informal learning environment. |
| Poster 42: The Impact of Value Added Tax on Alcohol Consumption |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shabana Din, St Catherine University, sdin@sgu.edu |
| Ywende Oladipe, St Catherine University, yoladipe |
| Cecilia Hegamin-Younger, St George's University, chyounger@mac.com |
| Jacqueline Lewis-Harris, University of Missouri, St Louis, antjharr@jinx.umsl.edu |
| Abstract: Small Island Nations are interested in the impact of Value Added Tax (VAT) on society and alcohol behavior consumption. However, an individual's behavior of alcohol consumption is connected to their culture. Culture defines the structural factors that contribute to the role of alcohol in family roles, masculinity and societal views. Even though patterns of alcohol can be documented through purchasing receipts, the behavior of consumption is less understood. This presentation will focus on the results and lessons learned understanding the role alcohol plays in Grenadian society and the structural factors. |
| Poster 43: A Pilot Study to Evaluate the Role of Education & Hands-on Training in Emergency Preparedness & Response |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Thuy Doan, University of Minnesota, doanx034@umn.edu |
| Debra Olson, University of Minnesota, olson002@umn.edu |
| Abstract: Natural disasters like Hurricane Katrina and man-made catastrophes like the September 11th attacks validated the lack of timely and coordinated response, while reinforcing the urgent need to improve the public health preparedness system in the United States. This is a process and outcome cross-sectional retrospective study aimed to evaluate the effectiveness of the Centers for Disease Control & Prevention Division of Strategic National Stockpile Training Program. Our goal is to examine whether this specific training impacted individual behavior change post participation and/or organizational performance capability in disaster preparedness and response. A 37-question survey adapted the Kirkpatrick evaluation theory, a knowledge test, and a demographic form was designed. Completed online surveys were received from 93 participants. Most respondents tend to agree or strongly agree about their overall satisfaction and relevance of the training course to their response role. The survey is being evaluated for internal consistency and identification of redundant items. |
| Poster 44: Evaluation of State and Territorial Pandemic Influenza Operations Plans (OPLANS) |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stephanie Dopson, Centers for Disease Control and Prevention, sld9@cdc.gov |
| Abstract: The aim of this session is to describe the process of evaluating OPLAN scores in 2007 and 2008 to determine if states and territories improved their pandemic influenza preparedness through activities related to epidemiological investigation and surveillance, community mitigation, medical countermeasures, mass vaccination and communications. The presentation will highlight preliminary findings examining the effect of select awardee activities funded by the Pandemic Influenza Supplemental program that were intended to assist planning at the state, local level and territories. States were required to submit their Pandemic Influenza Plans in 2007 (baseline) and 2008 for the purpose of being scored and determine current preparation and ability to respond to a pandemic. OPLAN data is being analyzed by Public Health objectives (Epidemiology, Laboratory, Public Health Monitoring, Public Health Systems Support and Response and Recovery) and compared with budget allocations by the categories of Personnel, Supplies, Equipment and Contracts. |
| Poster 45: Evaluation of the National Chlamydia Coalition |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sonal R Doshi, Centers for Disease Control and Prevention, sdoshi@cdc.gov |
| Victoria M Beltran, Centers for Disease Control and Prevention, vbeltran@cdc.gov |
| Abstract: Despite the increasing importance of coalitions in public health, few models exist to evaluate the effectiveness of coalitions. In 2008, the Centers for Disease Control and Prevention's (CDC) Division of STD Prevention (DSTDP), in collaboration with Partnership for Prevention, established the National Chlamydia Coalition (NCC). The goal of the NCC is to reduce the rates of Chlamydia and its sequelae among sexually active adolescents and young adults. The NCC is comprised of over 59 national organizations representing the medical community, advocacy groups, and local, state, and federal government agencies working to improve the health of women and adolescents. Evaluating the effectiveness of the NCC, internal operations, impact and outcomes will result in valuable information for all stakeholders and the field of coalition evaluation in general. This presentation will cover the following evaluation components: coalition logic model; methods; and impact and outcomes of coalition activities on CDC/DSTDP and coalition members |
| Poster 46: Measuring the Right Outcomes: How to Select and Create Instruments That Reflect the Values Of Science, Technology, Engineering, and Math (STEM) Education |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anna Douglas, Purdue University, douglask@purdue.edu |
| Melissa Yale, Purdue University, myale@purdue.edu |
| Deborah Bennett, Purdue University, bennett@purdue.edu |
| Abstract: It is important in evaluation to use survey instruments that adequately measure the desired outcomes in a STEM education initiative. However, published options for capturing STEM education outcomes from a quantitative perspective are limited, and may not be appropriate, given the unique goals and objectives of each program. Using two case studies, the presenters will demonstrate how to select an instrument that closely matches with the desired goals and objectives of the program, noting the design and psychometric properties of the survey and the interpretation of results. |
| Poster 47: Research in Process Use Evaluation: Using Evaluation-Focused Discussion Groups to Improve Learning Outcomes |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anna Douglas, Purdue University, douglask@purdue.edu |
| Carrie Wachter Morris, Purdue University, cwachter@purdue.edu |
| Deborah Bennett, Purdue University, bennett@purdue.edu |
| Abstract: As evaluators, we frequently rely on participants' help in order to collect meaningful data for the program or organization being evaluated. In process use evaluation, there is also a concern for how the program and participants are affected by the evaluation process. In this study, graduate students whose coursework involved using an innovative pedagogical tool and participation in an evaluation-focused discussion group about their experience with the learning tool, were assessed for learning improvements in terms of self-efficacy and skill as a result of participation in the evaluation-focused discussion group. |
| Poster 49: Outcome Evaluation of the Portsmouth, Virgina Weed and Seed Program |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Krystall Dunaway, Eastern Virginia Medical School, kdunaway@odu.edu |
| Abstract: This poster focuses on an outcome evaluation of the 5-year Portsmouth, VA Weed and Seed program. Weed and Seed is a community-driven two-prong strategy that combines law enforcement efforts with community-based services and seeks to 'weed' neighborhoods of crime and 'seed' them with community-based programs. The outcome evaluation consisted of community surveys, committee member focus group, and analysis of crime data. Community surveys revealed that residents of the Weed and Seed area typically responded similarly to residents of other areas of Portsmouth, and most responses were positive. Focus group participants reported that the most beneficial aspect of Weed and Seed was the relationships that were formed between law enforcement and community members, and that limited citizen input, loss of momentum during the program's final years, and substantial problems with coordinator turnover were significant milestones. Finally, crime data from before the implementation of the program was compared to current crime data. |
| Poster 50: Using Evaluative Thinking to Improve the Policy Development Process |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diane Dunet, Centers for Disease Control and Prevention, ddunet@cdc.gov |
| Lauren Gase, Centers for Disease Control and Prevention, lgase@cdc.gov |
| Abstract: In policy development, evaluative thinking offers a way of considering problems and potential policy solutions while recognizing diverse political and social contexts. 'Evaluative thinking' is a way of questioning, reflecting, and learning through the use of logical reasoning, explicit criteria, and data. Drawing from evaluation and planning frameworks and our experience with a project for developing a policy to reduce dietary sodium in the County of Los Angeles, we derived six principles of evaluative thinking to improve a policy development process. The principles include engaging stakeholders to understand how they frame a problem and the values important to them to in selecting a policy option. Evaluative thinking also helps to clarify the underlying theory-of-change of a policy solution and looks ahead to plan for implementation of a policy after it is adopted. We conclude that evaluative thinking can strengthen a policy development process and lead to well-conceived policies. |
| Poster 51: Database Management Strategies for Longitudinal Criminal Data |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lindsey Dunn, University of North Carolina, Greensboro, l_dunn@uncg.edu |
| Lauren Fluegge, University of North Carolina, Greensboro, lbfluegg@uncg.edu |
| Abstract: Database management strategies that are efficient and systematic for large volumes of scientific data are an essential part of evaluation given that the quality of the evaluation results can largely be determined by the quality of the data that are analyzed. Technology has allowed evaluators and community alike to collect large amounts of data at a fast rate, but the techniques used to manage the data files sometimes lack the same elegance of the data collection process. Because community program data can be managed by various sources and be kept in different formats, it can be difficult to seamlessly restructure and analyze the data. In this poster session, we intend to present issues encountered in the data management and analysis of multiple, large, longitudinal criminal databases and provide an overview of the strengths and drawbacks of using different programs, including Excel, Access, and SPSS, in managing large datasets. |
| Poster 52: Project ENRICH at the University of North Carolina at Greensboro (UNCG) - Evaluating the Revisioning of the University of North Carolina Greensboro Teacher Education Undergraduate Programs through a Teacher Quality Partnership (TQP) Grant |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pamela B Finney, University of North Carolina, Greensboro, pfinney@serve.org |
| Lynn Amwake, University of North Carolina, Greensboro, camwake@serve.org |
| Betty Epanchin, University of North Carolina, Greensboro, bcepanch@uncg.edu |
| Cheryl Greenberg, University of North Carolina, Greensboro, cgreenberg@uncg.edu |
| Abstract: The SERVE Center entered into an agreement with the University of North Carolina at Greensboro (UNCG) School of Education, to serve as the external evaluator for the Teacher Quality Partnership (TQP) ENRICH Project, a multi-year, multifaceted project. This proposal provides an overview of the evaluation of one component of the ENRICH Project—the re-visioning of undergraduate teacher education across the 22 various undergraduate university programs. This re-visioning includes: Adherence to the NC State Board of Education 2007 mission statement. Incorporating the standards for administrators and teachers developed in alignment with this mission statement, including the NC Professional Teaching Standards (NCPTS). Utilizing the developed NC Teacher Evaluation Rubric which operationalizes NCPTS. Implementing a new electronic teaching portfolio used for program review/accreditation. Following the guidelines for Program Blueprints which provide a consistent outline for programs' re-visioning reports. Blueprints must be approved by the state committee for programs to continue. |
| Poster 53: Revisiting a Statewide Evaluation of Supplemental Educational Services (SES) Programming: Lessons Learned and Changes Made Year 2 Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pamela B Finney, University of North Carolina, Greensboro, pfinney@serve.org |
| Melissa Williams, University of North Carolina, Greensboro, mwilliam@serve.org |
| Ann Poole, University of North Carolina, Greensboro, apoole@serve.org |
| Jane Meli, University of North Carolina, Greensboro, jmeli@serve.org |
| Abstract: In 2010, the SERVE Center at UNCG served as an external statewide evaluator for the Supplemental Educational Services (SES) program which falls under Title I of the Elementary and Secondary Education Act, as reauthorized by the No Child Left Behind Act of 2001. The SES program provides free-of-cost academic assistance for eligible students, over and above regular school hours. This external evaluation examined the performance of the statewide SES providers on the basis of: (1) student attendance, (2) parental satisfaction, and (3) academic achievement of participating students. In 2011, SERVE was contracted to conduct the external evaluation for statewide SES programming for a second year. As such, SERVE implemented strategic changes in the evaluation process which were a direct result of challenges encountered in year one of the evaluation. This proposal provides an overview of the lessons learned and highlights the changes that were incorporated into the 2011 evaluation. |
| Poster 54: Planning for Sustainability: Using Needs Assessment to Develop a Roadmap for the Caribbean Center for Green Technology |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kala Fleming, University of the Virgin Islands, kalafleming@fhwgroup.com |
| Wayne Archibald, University of the Virgin Islands, warchib@uvi.edu |
| Abstract: The University of the Virgin Islands' Caribbean Center for Green Technology was established in 2010 with the vision to become a regionally focused, internationally relevant authority on energy security and environmental sustainability. In developing the Center's program theory, important milestones that would need to be accomplished to make progress toward that vision included creating new tools that improved decision-making and creating a vast partnership network that the Center could leverage for growth. Struck by the Center's limited resources and lofty goals, we recognized that conducting a needs assessment would be a critical program planning step that could clarify program goals and lead to the development of a more focused approach. The presentation will describe how a participatory needs assessment process was used to prioritize the Center's activities. |
| Poster 55: Visualizing Significance |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Steve Fleming, National Center for Educational Achievement, sfleming@nc4ea.org |
| Lynn Mellor, National Center for Educational Achievement, lmellor@nc4ea.org |
| Abstract: The assessment of statistical significance helps researchers summarize the evidence against a null hypothesis. However, it is not always a simple task to discern the true meaning or value of the results of null hypothesis significance testing to a research study. This poster will focus on visual explanations of the meaning of statistical significance and offer guidance on why one might choose a particular significance level. In addition, the relationship between statistical significance and confidence intervals, and the trade-offs inherent in choosing a significance level, power, and sample size in the context of variability and the determined level of practical significance will be illustrated. Examples will be based on a value-added model which attempts to identify schools that stand out on measured student achievement. |
| Poster 56: Assessing Invisible Needs: Uncovering and Exploring the Needs of Young Lesbian, Gay, Bisexual, Transgender and Queer (LGBTQ) Youth Who Have Been Pregnant |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rachel Fletcher, University of Minnesota School of Public Health, rachel.fletcher@rainbowhealth.org |
| Loretta Worthington, Rainbow Health Initiative, loretta.worthington@rainbowhealth.org |
| Abstract: The Rainbow Health Initiative has embarked on a three-year process of assessing the needs of young (18 to 25-year-old) lesbian, gay, bisexual, transgender and queer individuals who have been pregnant. While little is known about this population, we do know that the prevalence of pregnancy among lesbian and bisexual youth has been documented to be twice as high as that of their heterosexual peers . Completing a needs assessment of this population presents unique evaluation challenges, because of the marginalized and invisible place these individuals hold in our society. This poster will present an overview of our three-stage community needs assessment, beginning with the local qualitative face-to-face interviews, the national needs assessment we intend to develop, and finally the focus groups we will use to pilot test program ideas. It will highlight particular challenges we have faced, lessons we have learned, and recommendations we have for working on needs assessments with members of the LGBTQ community. |
| Poster 57: Enhancing Natural Resource Data Collection Accuracy through Positive Partnerships between Undergraduates and Urban High School Students |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Nick Fuhrman, University of Georgia, fuhrman@uga.edu |
| Milton Newberry, University of Georgia, miltron3@uga.edu |
| John Peterson, Virginia Tech, jopeters@vt.edu |
| Abstract: Working collaboratively, undergraduates and urban high schoolers from Atlanta, Georgia and Washington, DC visited two parks to collect forestry data needed by federal and state land managers. In previous years, urban high schoolers worked with their high school teachers to collect tree species and diameter data without the assistance of undergraduate students or university faculty and accuracy of their data suffered. Only 55% of the data collected through student-teacher collaborations matched established answer keys. When high schoolers collaborated with university faculty to collect data, 98% of their data were accurate. However, when high schoolers collaborated with forestry and agricultural communication undergraduates with minimal faculty assistance, 85% of their data were accurate. These results suggest that with less faculty support and trained undergraduates leading the teaching, data accuracy can remain moderately high. Environmental education programs should consider promoting partnerships between undergraduates and high schoolers in the collection of natural resource data. |
| Poster 58: Gender Empowerment in Basic Development Needs (BDN) Initiative |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Salma Galal, Evaluation and Research Network, Egypt, asra78@gmail.com |
| Abstract: WHO /EMRO initiated Basic Development Needs (BDN) in diverse countries of the region following objectives: - Organising the community and building its capacity - Promoting self-management and self-reliance - Building relationships with different stakeholders Partnership was necessary between the community, facilitators, and technical support team. This work addresses: -Process and achievements -Questionnaire for needs assessment -Gender empowerment Community representatives function as - link to families in their street - BDN needs assessment survey - dissemination of health and environmental education - follow up on micro-credit projects, loans Health survey: significantly more female adults (7.3%) were found to have ailment than men (3.7%). The hospital seems to be the most accessed health care service. Process indicators: such as CRs absenteeism, health amd environmental education acquirement, interface between health service and community, attendance of literacy classes and graduation, loans repayment, and legal status (ID, birth certificate) Observations of women and youth empowerment |
| Poster 59: Development and Implementation of a Statewide Database for Child Maltreatment Prevention Outcomes |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rebecca Gillam, University of Kansas, rgillam@ku.edu |
| Karin Chang-Rios, University of Kansas, kcr@ku.edu |
| Mary Morales, University of Kansas, mmorale1@ku.edu |
| Abstract: Data collection and measurement have become priority issues for early childhood and child abuse and neglect prevention programs as funders, program administrators, and evaluators seek common outcomes and indicators. Recent federal funding for State Advisory Councils for early childhood, for example, requires states to move toward a unified data collection system. Kansas has implemented use of a common instrument within prevention programs, the Protective Factors Survey, to help agencies measure changes in protective factors and identify areas where workers can focus on increasing individual family protective factors. This poster presentation provides an overview of a web-based data collection system that was developed for CBCAP programs and has been expanded for use statewide. Evaluators will describe the process used for designing a cost-effective database that monitors data collection and provides immediate access to data reports by both program staff and evaluators. |
| Poster 60: Character Assessment for School Age Children: A formative evaluation tool |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Leslie Grier, California State University, Fullerton, lgrier@fullerton.edu |
| Abstract: The purpose of this research was to examine the psychometric properties of a character assessment scale (i.e., the Character Assessment for School Age Children - CASAC) based on the six pillars of character (Josephson Institute, 2009). Many youth development and character education programs utilize some if not all of the pillars of character explicitly or implicitly (e.g. Prestwich, 2004); yet there are few character assessments that integrate these components in a comprehensive way. Furthermore, as many of these programs have been utilized to augment academic achievement, there is increasing emphasis on the importance of formative evaluation (e.g., Berkowtiz & Bier, 2004; Zins, Bloodworth, Weissberg, & Walberg, 2004). Hence a brief assessment of character that was also predictive of academic achievement would be useful as a formative evaluation tool. Exploratory factor analyses resulted in a two-dimensional scale. These dimensions demonstrated internal reliability and criterion validity. |
| Poster 61: Evaluating Small Online Courses |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Mark Grimes, University of California Educational Evaluation Center, mgrimes@education.ucsb.edu |
| Abstract: A pilot evaluation was conducted for a novel approach to instructing statistics online. A network of online statistics courses was created that allowed to students enrolled in any of the courses to access material from other courses in the network. Data collection methods included student surveys, student interviews, student focus groups, instructor interviews, observations of synchronous lectures, and a review of the course content and operating system. Results showed that this type of instruction is effective for those who enroll and this network helped moderate the different levels of instructor ability in conducting online courses. This project presented a number of issues that are important for evaluators to address. Are there ways to conduct a meaningful impact evaluation when there are barriers such as these? Or would it be better to focus more resources on a thorough process evaluation? General recommendations are made for future evaluations of online course evaluations. |
| Poster 62: Corporate Sustainability Reporting: Driving Behavior Change |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shari Grossarth, United States Environmental Protection Agency, grossarth.shari@epa.gov |
| Abstract: Corporate sustainability reporting involves a company's measurement, management and public disclosure of information related to the company's environmental, social and governance performance. The measurement, management and disclosure process has the potential to increase awareness and drive changes in behavior that could ultimately lead to more sustainable operations in companies. The US Environmental Protection Agency is exploring how sustainability reporting might drive improved corporate sustainability practices, with an eye toward achieving environmental outcomes. This poster presents the logic behind corporate sustainability reporting in driving behavior change and invites discussion around how sustainability information disclosure might be used as a policy tool. |
| Poster 63: Lessons Learned from Evaluating the Texas College and Career Readiness Website |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Leslie Grunden, Edvance Research Inc, lgrunden@edvanceresearch.com |
| Lauren Decker, Edvance Research Inc, ldecker@edvanceresearch.com |
| Sarah Braisel, Edvance Research Inc, sbraisel@edvanceresearch.com |
| Eric Rolfhus, Edvance Research Inc, erolfhus@edvanceresearch.com |
| Abstract: The Texas College and Career Readiness website—a one-stop place for students to complete specific tasks necessary to enter college—is currently being piloted in 37 schools across the state of Texas, selected because of their high percentage of at-risk students. This paper will present lessons-learned from implementation and external evaluation of the relationship between the website pilot and student task completion. Because it will not be possible to compare student task completion in pilot schools with student task completion in non-participating schools, an alternative method will be employed. Regression models will identify the conditions associated with higher levels of student task completion on the website, adjusted for school-level baseline risk. Findings will provide formative guidance on potential modifications to the pilot in future scale-up, as well as the amount and types of support that should be in place to increase student task completion using the website. |
| Poster 64: Strategies for Increasing the Likelihood of Information Acceptance by Non-Profit Organizational Leaders During Collaborative Evaluation Presentations |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Claudia Guerere, University of South Florida, cguerere@mail.usf.edu |
| Debra Thrower, Metropolitan Ministries, dthrower@tampabay.rr.com |
| Abstract: Collaborative evaluation was used to observe the effectiveness of a permanent supportive housing program managed by a non-profit organization in the southeastern United States. Strategies used by facilitators in the presentation ensured organizational leadership buy-in of focus group results. These strategies included careful thought of internal and external evaluators strengths to gain the trust of the audience. Recognizing the power of individual cultural diversity afforded opportunities to speak the 'language' of a multicultural audience. Supportive resources, to include industry standards and procedures, established credibility when addressing questions on employed methodology. Acknowledging management's desire to enhance client services, via focus groups, was essential to preparing the audience for unanticipated focus group results. This course of action generated open discussion, which positivity affected the re-development of program goals and objectives for clients. An organizational shift occurred as a direct result of this effective presentation style. |
| Poster 65: Evaluation of Educational Outcomes of Problem-Based Learning in a Hybrid Curriculum |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shyamala Hande, Melaka Manipal Medical College, hande2010@gmail.com |
| Ciraj A Mohammed, Melaka Manipal Medical College, cirajam@gmail.com |
| Ramnarayan Komattil, Melaka Manipal Medical College, kram@manipal.edu |
| Abstract: Problem-based learning (PBL) is a vital component of the medical undergraduate hybrid curriculum at the Melaka Manipal Medical College, Manipal Campus, India. The main objective of this study was to evaluate the educational outcomes of problem-based learning. Medical students in four cohorts were requested to participate n the study by completing an evaluation questionnaire. Qualitative data was collected to support quantitative data. It was a cross-sectional study and 464 students participated. Problem-based learning at the institution improved students' acquisition of knowledge, generic skills and attitudes and showed a positive correlation (0.451-0.72) between scores in all the three domains. Students perceived PBL as a strategy for gaining knowledge, acquiring generic skills and developing favorable attitudes. Qualitative analysis of student responses indicated that PBL as a small group learning strategy provided them a favorable and safer environment to develop necessary skills and attitudes. |
| Poster 66: Hoping and Coping: A phenomenological approach to evaluating a Human Immunodeficiency Virus (HIV) counselor training program |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shayne Harrison, University of Tennessee, Knoxville, sharrison1976@comcast.net |
| Abstract: Limited research has studied emotional experiences of individuals living with the Human Immunodeficiency Virus (HIV). With development of protease inhibitors in 1995, life quality of HIV patients has undoubtedly improved (Hickey et al., 1996). However, fewer individuals are being tested. By identifying barriers, health agencies can strengthen outreach efforts resulting in personalized initiatives. In this study, the student evaluator interviewed five local homosexual men who have lived with HIV for at least fifteen years. The interviews shed light on key themes, including bias and a journey for support. The researcher applied evaluation models (Schulberg & Baker, 1968; Glasgow et al., 1999) to the research design, but incorporated qualitative approaches to explain the phenomena. The researcher used private interviews, resulting in recommendations to the Tennessee Department of Public Health's certified HIV-testing Counselor Program. The researcher found an increase in testing participation, and hopes to continue this process with other high-risk populations. |
| Poster 67: Teacher Quality Partnership Grant Logic Mapping: Models within Models |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tiffany K Harrison, University of North Carolina, Greensboro, tharriso@serve.org |
| Karla C Lewis, University of North Carolina, Greensboro, klewis@serve.org |
| Kathleen Mooney, University of North Carolina, Greensboro, kmooney@serve.org |
| Bernette Kelley, University of North Carolina, Greensboro, bkelley@serve.org |
| Abstract: Logic mapping is often used to help people and programs think through the process of an evaluation or research project. A visual method for understanding the evaluation process, the SERVE Center at UNC-Greensboro helped the East Carolina University (ECU) College of Education (COE) and its partners in the Teacher Quality Partnership (TQP) grant develop a logic map for each component of the project. This poster will highlight the curriculum reform, clinical practice, and induction components of the partnership working in tandem toward the goal of preparing new teachers who have a strong 'knowledge and experience base' in order to improve student achievement (USED 2010). U.S. Department of Education. (2010). Teacher Quality Partnership Grant Program: 2009 Awards. Retrieved March 10, 2010, from http://www2.ed.gov/programs/tqpartnership/2009awards.html |
| Poster 68: A Visual Introduction to the Japanese Local School Evaluation in Practice: Snapshots of the Public School Evaluation Done by an External Visiting Team |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Akihiko Hashimoto, National Institute for Educational Policy Research of Japan, soubun@white.plala.or.jp |
| Abstract: The first purpose of this presentation is to let the audience have a visual understanding of Japanese school evaluation in practice through seeing my personal experiences as an evaluator or a designer of evaluation to public schools in Japan. And the second purpose is to evoke interesting dialogues between the audiences and the presenter, myself, that should yield precious ideas for better practices of school evaluation. This presentation consists of; a) a bird eye view of the Japanese school evaluation system, that was legislated within this decade. b) an instant explanation of the national ?rial project? of the public school external evaluation. c) slideshow and panels of the details of the real public school external evaluation cases. |
| Poster 70: Step-by-Step Strategies for Conducting a Collaborative Evaluation of Teacher Technology Training |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Hibbard, Florida Gulf Coast University, shibbard@fgcu.edu |
| Douglas Spencer, Edu Inc, douglas@eduinc.org |
| Abstract: Teacher professional development plays an important role in ensuring science teachers are kept abreast current knowledge and skills in STEM education. This poster presents step-by-step examples of the strategies used to evaluate teacher professional development training in information and communication technology (ICT). A collaborative evaluation approach was used to evaluate a program designed to train secondary science teachers to teach their students about information and communication technology (ICT). The evaluation used the six steps outlined in Rodriguez-Campos' (2005) step-by-step guide to collaborative evaluation. Strategies and activities used to conduct the evaluation will be presented. Data sources include, pre- and post surveys from teachers and students, teacher interviews, curricular resources, and wikis. Lessons learned and future directions will be discussed. |
| Poster 69: Mixed Methods Designs in Evaluation Studies: Trends in the Published Literature |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Hibbard, Florida Gulf Coast University, shibbard@fgcu.edu |
| Anthony Onwuegbuzie, Sam Houston State University, tonyonwuegbuzie@aol.com |
| David Devine, Lincoln School, djdevine@eagle.fgcu.edu |
| Abstract: To describe the utilization of mixed methods designs in evaluation practice, the authors conducted a content analysis on evaluation studies published in eight evaluation-focused journals for an 8-year period (2003-2010). The purpose of this study was to identify common uses of mixed methods in evaluation practice across a large set of peer-reviewed evaluation studies. These data provide insight into the way practitioners are using mixed methods evaluation designs within the field. Greene, Caracelli, and Graham's (1989) mixed methods conceptual framework of five purposes for using mixed methods designs was used to categorize the program evaluation studies. Results from this content analysis provide empirical evidence of the way mixed methods evaluation designs are used in practice. The use of mixed methods designs in evaluation studies will be discussed as well as the current state of evaluation practice in regards to mixed methods evaluation designs. |
| Poster 71: Redefining Levels of Impact: Introducing a More Comprehensive Framework to Conceptualize and Measure Impact for Individuals, Groups, Organizations and Society Through Development Interventions |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Emily Hoole, Center for Creative Leadership, hoolee@ccl.org |
| Jennifer Martineau, Center for Creative Leadership, martineauj@leaders.ccl.org |
| Abstract: This session introduces a more comprehensive framework for understanding the impact of developmental interventions at the individual, group, organizational and community level. The Center for Creative Leadership has been working with clients and through research to understand how work focused beyond the individual level helps groups, organizations and communities develop new ways of working together to achieve results. This work extends current frameworks for evaluation and outcomes by fully explicating each level of impact within a multi-level system, and differentiating between competencies and capabilities to be developed, and the impact that results from the development process. The framework also takes into account the reciprocal influence between levels. Evaluation methods appropriate for each level will also be presented and discussed, along with challenges and opportunities. |
| Poster 72: Considerations for Educational Evaluations for Children of Military Personnel |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stephani Howarter, University of Kansas, sschmidt@ku.edu |
| Karin Chang-Rios, University of Kansas, kcr@ku.edu |
| Thai Q Le, University of Kansas, le1242@ku.edu |
| Stephanie Diehl, Geary County, stephaniediehl@usd475.org |
| Susan Guinn, Geary County, susanguinn@usd475.org |
| Abstract: From 2004-2011, the Foundations of Literacy Activities for Geary County Children (FLAG) project implemented literacy development interventions for approximately 1,600 preschool-aged children from Geary County , Kansas. Nestled within Geary County is the Fort Riley Military Reservation, a U.S. Army installation with nearly 20,000 active service men and women. Many children of these military men and women attended Geary County preschools and comprised nearly 70% of the FLAG project's sample. The following poster presentation will review the challenges and complexities the FLAG project coordinators and evaluators encountered, including issues of children and staff transiency and protocols for implementing on-site interventions. The presentation will provide insight into the special considerations of implementing and evaluating large-scaled projects within this unique population. Lastly, solutions will be discussed as well as suggestions for future projects with military personnel. |
| Poster 73: High School Robotics Competition: Building More than Robots |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Douglas Huffman, University of Kansas, huffman@ku.edu |
| Anita Welch, North Dakota State University, anita.welch@ndsu.edu |
| Abstract: In this project, high school students participated in a robotics competition where they worked with scientists to design and build a robot to perform a specific task. Students' attitudes toward the scientific process and attitudes toward scientists were measured as a means of evaluating the impact of the project on students. Students who participated in the robotics competition were compared to students in the same schools who did not participate in the robotics competition. Students tend to have stereotypical negative views of scientists as 'nerds' or special people who are particularly smart. Results indicated that the project appears to have helped students see science as a normal activity and scientists as a normal people. The theoretical and practical implications of results for evaluation are discussed. |
| Poster 74: Systematic reviews: An Approach for Examining the Existing Literature |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jacqueline Huscroft D'Angelo, University of Nebraska, Lincoln, jhuscrof@yahoo.com |
| Alexandra Trout, University of Nebraska, Lincoln, atorkelson-trout2@unl.edu |
| Kristin Duppong Hurley, University of Nebraska, Lincoln, kdupponghurley2@unl.edu |
| Abstract: A critical component to research in any discipline is an understanding of the existing literature. This knowledge is of particular importance when training doctoral students to aggregate and evaluate findings across a subject area. Because the literature on best practice for conducting comprehensive reviews is sparse, students often leave training programs without the critical skills needed to identify and locate the existing literature or to evaluate methods and results to determine what is and is not known, the quality of the research, and the existing gaps. This poster will present one systematic approach for conducting literature reviews that we have used with a variety of topics to gain knowledge, interpret results, and synthesize findings. An example of this approach will be presented as it was used to review the psychometric qualities of parenting measures assessing parental attitudes from 1960 to 2009. |
| Poster 75: Relationship of Client Engagement and Treatment Alliance in Predicting Youth Mental Health Outcomes |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chrystal Jansz, University of Nebraska, Lincoln, cerj7@hotmail.com |
| Kristin Duppong Hurley, University of Nebraska, Lincoln, kdupponghurley2@unl.edu |
| Justin Sullivan, University of Nebraska, Lincoln, justin.sullivan@boystown.org |
| Abstract: The objective of this NIMH-funded study is to examine the role of common therapeutic factors (e.g. client engagement, therapeutic alliance) on youth mental health outcomes for 145 youth during their stay in residential group homes using a modified version of the Peabody Treatment Progress Battery (PTPB). The audience will learn about how the PTPB can be used to routinely monitor common therapeutic treatment factors in a residential care setting, examine how these common process factors change over time, the impact these factors have on youth outcomes, and the relationship between common therapeutic factors and quality of treatment implementation. |
| Poster 77: Application of Experiential Learning Concepts for Teaching Values and Valuing in Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Koralalage Jayaratne, North Carolina State University, jay_jayaratne@ncsu.edu |
| Abstract: Understanding the pluralistic meaning of 'Values and Valuing' is essential for making valid judgments in evaluation profession. Preparing future evaluators with necessary professional capacity to make meaningful judgments is essential for the continuous growth of the profession. Evaluation graduate courses shape the future evaluators. The purpose of this poster presentation is to demonstrate how to integrate experiential learning concepts for designing instructional activities to teach evaluation concepts including 'Values and Valuing.' This poster presentation is based on the author's experience in teaching an evaluation course and literature. This poster presentation analyzes the concept of values and valuing in an evaluation context; examines the practical means to integrate experiential learning concepts; designs student assessments to determine the effectiveness of experiential learning activities; and provides a framework to integrate experiential learning concepts to enhance the effective learning of evaluation concepts. |
| Poster 76: Issues, Challenges and Strategies to Improve Utilization of Evaluation Results in Extension |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Koralalage Jayaratne, North Carolina State University, jay_jayaratne@ncsu.edu |
| Abstract: With the increased demand for accountability, Extension agents have been compelled to evaluate their programs. In this situation, Extension agents tend to view evaluation as a responsibility rather than a useful tool. Most of the time, the current utilization of Extension evaluation results is limited to satisfy state and federal reporting requirements. We have not yet explored the potential for using evaluation results to shape the future of Extension. The purpose of this paper presentation is to share results of an online study conducted with Extension agents and county Extension directors to determine their current situation of evaluation results usage, issues associated with utilization, needed assistance in promoting evaluation results utilization and opportunities and strategies useful to promote utilization of evaluation results. This presentation will display the values and valuing of evaluation results by Extension agents and county Extension directors. |
| Poster 78: Randomized Control Evaluation of a home-based youth farm safety intervention |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Hamida Jinnah-Ghelani, University of Georgia, hamidajinnah@gmail.com |
| Zolinda Stoneman, University of Georgia, zo@ihdd.uga.edu |
| Becky Brightwell, University of Georgia, beckybrightwell@ihdd.uga.edu |
| Abstract: Injury and death rates of youth on farms in the U.S. are particularly high. Most approaches in farm injury prevention have focused on knowledge as the primary outcome. Gains in knowledge are minimally effective for changing behavior. Results will be presented from a three-year randomized control trial evaluating the effectiveness of a family-based farm safety intervention that utilizes the Theory of Planned Behavior (Azjen, 1985) as a framework for changing safety behaviors of individuals. Data from about hundred families will be presented. Study focused on children between 10 and 19 years, who were active on the farm. Families were randomly assigned to one of three groups - parent-led, peer led and control group. Analysis of covariance models on the final wave of data collected, controlling for pre-intervention levels of the outcome variables, will be preformed. We postulate that one group will be more effective in positively changing the knowledge, attitudes, intentions, and behaviors of parents than the other. |
| Poster 79: Evaluating Climate Change Adaptation: The Certainty of Uncertainty |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Britta Johnson, United States Environmental Protection Agency, johnson.britta@epa.gov |
| Scott Bowles, United States Environmental Protection Agency, bowles.scott@epa.gov |
| Abstract: While the U.S. Environmental Protection Agency and others are working to help mitigate the impacts of climate change a simultaneous effort by federal, state and local planners is also underway to adapt to future climatic changes. A key to ensuring that climate change adaptation planning is both effective and efficient will be to incorporate performance measurement and program evaluation protocols into these efforts. The poster will share experiences with the development of the U.S. Cross-Agency Framework for Adaptation, that includes program evaluation and EPA's adaptation planning process as well as to provide a forum for the discussion of the unique set of challenges that climate change and climate adaptation present to evaluators. A primary goal of this poster session is to begin these discussions and lay the foundations for the development of a community of practice around climate change adaptation to address these issues as they develop. |
| Poster 80: Parental Involvement in Non-formal Environmental Education Programs |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Mary Kate Jordan, Penn State University, mary.kate.jordan1@gmail.com |
| John Ewing, Penn State University, jce22@psu.edu |
| Rama Radhakrishna, Penn State University, brr100@psu.edu |
| Abstract: Research has long shown that parents' positive involvement with their children's schooling is related to many positive outcomes. Non-formal education provides this opportunity for continued growth of the child. Nationwide, non-formal education centers, such as nature centers, zoos, and aquariums, offer programs for children with and without their parents being present. Many parents recognize the importance of getting their children outside, but are unable or unwilling to do so. With the busy world moving around them, parents have many reasons not to take their children outside. The purpose of this study was to investigate the involvement of parents and their children to attend environmental education programs in a non-formal setting. In addition, barriers to their participation were also examined. Results from this study could help environmental education programmers create new curricula to attract more families to their programs. |
| Poster 81: The Heat is On: Self-Efficacy for Sexual Abstinence and Gender Differences in a Rural Adolescent Population |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Waindim Jude Nsom, University of Dschang, Cameroon, waindim63@yahoo.com |
| Abstract: Although teenage pregnancy rates in Cameroon have declined since 1999, specific geographic areas of the country may be more at risk. Certain risk behaviors are more likely to occur among subpopulations defined by gender, race/ethnicity and age. An integrative review of adolescent sexual decision-making (Ambass 2000) confirmed gender differences as an important variable in adolescent sexual decision-making. This study was part of a larger cross-sectional investigation to evaluate a school-based sexual abstinence curriculum in various groups, including secondary-school and high school populations and an alternative school population in a rural community. Secondary school students (n= 80), high school students (n = 62), and alternative school students (n = 10) in a rural community received a 10 session abstinence-based, character building curriculum. Four demographic questions (age, gender, racial/ethnic background, family composition) were summarized with descriptive statistics. The AFL Core Instrument was used with additional ASESSA scale added for the purposes of this investigation. Previous work on the ASESSA reported concurrent validity and Cronbach's alpha of .91. |
| Poster 82: Self-Evaluation System of Information Technology (IT) Projects in Korean Central Government: Institution and Practices |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| JaeDong Jung, National Information Society Agency, jjdkorea@nia.or.kr |
| Abstract: As one of the best information technology (IT) powerhouses in the world, Korea has undertaken systematic IT project evaluation since 1997 in order to maximize the performance of investment into public IT projects. Currently under the Korea's IT project evaluation method under which one third of the total projects are selected as evaluation targets every year, each organization performs self-evaluation on logic model-based survey indicators on Program Assessment Rating Tool (PART), and the Ministry of Strategy and Finance verifies the result. This system pursues outcome-oriented performance management by deducting performance achievement scores if indicators and goals are inappropriate. Moreover, it holds a strong feedback system, under which more than ten percent of the next-year budget is deducted if the final result of evaluation turns out 'poor'. The 2008 IT project evaluation shows high scores in the stages of planning and execution of IT projects. This reflects, in most part, a fair level of collecting opinions from stakeholders, recognizing laws and regulations, and securing connection with relevant projects during the planning stage. It also signifies that any problems in the execution process are responded to in an appropriate way and that time schedules are well maintained and followed. However, the performance indicators and goals were found to be very inappropriate; the reason being that a majority of projects used output or process indicators rather than outcome indicators which represent final project performance and that the performance goals were set in a way too passive compared to the budget and the past performance. Indeed, such lack of appropriateness in performance indicators and goals was reflected as a deduction factor in measuring the performance achievement. The IT project evaluation needs to be improved in the future. First, a methodology or a model that help setting of appropriate performance indicators for IT projects needs to be developed and a process needs to be adopted, which allows setting of performance indicators and goals in the stage of planning in advance. Second, the improved level of informatization raised interoperability, standardization and security issues as IT projects, which had been carried out by each ministry, are now being integrated. Therefore, it is required to establish an evaluation framework that reflects features of the IT projects in order to assess such issues in a comprehensive way. Third, it is recommended to establish and diffuse an evaluation method that can help not only evaluate the performance of IT projects that had been already invested but also develop new projects or strategy and directions. Fourth, besides the short-term performance evaluation, which takes place about once every three years and measures only the process from planning to execution, performance and feedback during the year, it is also necessary to deploy mid- and long-term impact evaluation. |
| Poster 83: Preparing for a Portfolio Comprehensive Examination in an Evaluation Doctoral Program |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susanne Kaesbauer, University of Tennessee, Knoxville, skaesbau@utk.edu |
| Thelma Woodard, University of Tennessee, Knoxville, twoodar2@utk.edu |
| Jason Black, University of Tennessee, Knoxville, jblack21@utk.edu |
| Abstract: This poster presentation will highlight an Evaluation, Statistics and Measurement doctoral program's novel portfolio comprehensive examination option. As a portfolio showcases students' entire relevant learning experiences and work products, the development of a portfolio can be a challenging and stressful task. This presentation will discuss lessons learned by students who have completed portfolio comprehensive examination. Furthermore, suggestions to other students considering a portfolio option will be discussed. Lessons learned and suggestions will focus on topics, such as creating file organization systems, managing large amounts of reflections and creating detailed outlines. This topic will also be of value to other novice evaluators and faculty, as the lessons learned and suggestions can be applied directly to showcasing one's skills and work products to potential employers and evaluation clients. |
| Poster 84: Formative Evaluation Along the Science, Technology, Engineering, and Mathematics Pipeline |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jason Katz, University of South Carolina, jaskatz@gmail.com |
| Michael Schillaci, University of South Carolina, schillam@mailbox.sc.edu |
| Abraham Wandersman, University of South Carolina, wandersman@sc.edu |
| Susan Hudson, University of South Carolina, hudsons@mailbox.sc.edu |
| Pearl Fernandes, University of South Carolina, pefernan@mailbox.sc.edu |
| Briana Timmerman, University of South Carolina, timmermb@mailbox.sc.edu |
| Timothy Mousseau, University of South Carolina, mousseau@mailbox.sc.edu |
| Abstract: Our presentation will describe the use of FORmative Evaluation Consultation And Systems Technique (FORECAST) in programs that are contextualized within the Science, Technology, Engineering, and Mathematics (STEM) pipeline. FORECAST is an approach to formative evaluation that 1) offers specific models and tools, 2) promotes integration of process evaluation and short-term outcome evaluation, and 3) allows for real-time feedback to improve a program's theory and its implementation (Goodman & Wandersman, 1994; Katz et al., submitted). As a case example application, we will draw on the use of FORECAST in STEPs to STEM -- a National Science Foundation-funded undergraduate talent expansion program at the University of South Carolina for promoting the recruitment and retention of transfer students in STEM majors. We will describe how FORECAST has facilitated improvements in STEPs to STEM's theory of change and its implementation. |
| Poster 85: Evaluation of Study Abroad Programs for Science and Engineering Graduate Students |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Nicola Kernaghan, University of Florida, nikkik@ufic.ufl.edu |
| Linda Jones, University of Florida, lcjones@coe.ufl.edu |
| Abstract: In our increasingly globalized society, a college education should prepare students to live and work effectively with people from different backgrounds. Developing a diverse, internationally competitive, and globally engaged science and engineering workforce is one of the stated goals of the National Science Foundation (NSF). This study investigated the academic learning, personal growth, and intercultural development outcomes of 50 graduate students in STEM disciplines (science, technology, engineering and mathematics) who participated in four different study abroad programs sponsored by NSF (Chile, China, South Africa and Brazil). Students' pre-trip expectations and post-trip outcomes were measured using electronic surveys. Changes in pre-post levels of academic learning, personal growth, and intercultural development were measured for each student and aggregated and for the entire study abroad group. In addition, analyses included comparisons of the level of congruence between students' expectations and those of faculty members teaching the study abroad courses. |
| Poster 86: Engaging Stakeholders in Review and Recommendations for Models of Outcome Monitoring for Substance Abuse Treatment |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Nooshin Khobzi Rotondi, Centre for Addiction and Mental Health, nooshin_khobzi@camh.net |
| Brian Rush, Centre for Addiction and Mental Health, brian_rush@camh.net |
| Abstract: Aims: To present an example of a collaborative process designed to review models of outcome monitoring for substance abuse services, with a view to assessing the feasibility of implementing different approaches in Ontario, Canada. Methods: A partnership between diverse stakeholders, including managers, policy analysts, researchers, clinicians, and representatives from (non-)governmental organizations, was formed early in the planning phase to ensure greater input and investment in the project. Detailed feedback was obtained from stakeholders during three consultation events about the types of information they would most like to obtain from an outcome monitoring system. Findings and Conclusions: A model of outcome monitoring and a phased approach to its implementation were recommended by the participants. We view our collaborative approach as essential to the successful implementation of a client outcome monitoring system in Ontario. Given adequate resources, this project is expected to benefit clients, clinicians, and program and system planners. |
| Poster 87: Negotiating the Value of Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chris Kirk, Wichita State University, cxkirk@wichita.edu |
| Rhonda Lewis, Wichita State University, rhonda.lewis@wichita.edu |
| Abstract: While evaluation research is clearly established as a vital element of many programs and organizations, some fail to recognize the true value of program evaluation. In these climates, members of the evaluation team must promote the value in their work and negotiate key elements of the evaluation process. This poster will highlight one such relationship in a federally-funded, state-wide program focus sing on college access for youth in foster care. Throughout the multi-year grant, the evaluation team and program personnel conducted a series of conversations to establish common values which reflected both the integrity of the program and the needs of the evaluators. As a result of these discussions, data was obtained which added both to internal program quality and the external research literature. Key lessons learned will be shared, including strategies for negotiating access, validity trade-offs, and creative usage of mixed-methods to enhance evaluation results. |
| Poster 88: Data Sharing Among Evaluators to Improve Substance Abuse Treatment for People who are Involved in the Criminal Justice System: Strategies and Implications for Evaluators |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tara Kirkpatrick, Centerstone Research Institute, tara.kirkpatrick@centerstone.org |
| Catherine Sewall-Martin, Centerstone Research Institute, catherine.sewall@centerstone.org |
| Charles Brown, Centerstone Research Institute, charles.brown@centerstoneresearch.org |
| Abstract: This paper discusses the value of sharing data among evaluators to provide information on substance abuse treatment for individuals involved in the Criminal Justice System. Data sharing can have important implications for refining intervention strategies to meet the needs of criminal justice offenders during service delivery. This paper will explore the benefits of a collaborative partnership among evaluators during program implementation. An example will be presented to show how an evaluator shared outcomes involving adults in state prisons with an evaluator of a school-based program who used the data to make evaluative decisions to improve substance abuse treatment among juveniles. This paper will highlight benefit of shared/common values among evaluators particularly in terms of trust, good communication, and using evaluation findings creatively to improve program implementation. |
| Poster 89: Finding Value in Professional Development for Informal Science Educators |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| James Kisiel, California State University, Long Beach, jkisiel@csulb.edu |
| Sue Magdziarz, Crystal Cove Alliance, smagdziarz@crystalcovealliance.org |
| Maria Grant, California State University, Fullerton, mgrant@fullerton.edu |
| Amy Cox-Petersen, California State University, Fullerton, acox@fullerton.edu |
| Donna Ross, San Diego State University, dlross@mail.sdsu.edu |
| Abstract: How do informal science educators (working and teaching in museums, aquariums and the like) develop their practice? In what ways might professional development support their efforts, and how might the effectiveness of training efforts be documented, especially when there are no 'student scores' to compare? This presentation examines approaches for evaluating a professional development project aimed at supporting informal science educators. Mixed evaluation methods were used to assess changes that may have resulted from participation in the program, with greater focus on perceived institutional value and self-efficacy. In addition, suggestions for documenting long-term impacts were obtained from participants in an effort to devise more authentic assessments. Implications for alternative documentation of professional development effectiveness will be discussed. |
| Poster 90: Evaluating the Effectiveness of a Professional Development Program on Student Outcomes Based on Educator Perceptions |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Martha Klemm, University of Massachusetts, Boston, martha.klemm@umb.edu |
| Abstract: Evaluating the effectiveness of professional development programs in terms of improving student outcomes continues to be a challenge in program evaluation, requiring the development of innovative methods to capture potential change (Dede et al., 2009). This poster presentation describes how evaluators measured the impact of a professional development initiative, the Massachusetts FOCUS Academy, through operationalizing student outcomes using input from multiple stakeholders. The poster provides recommendations for implementing this technique in other evaluations. |
| Poster 91: Measuring What You Mean to Measure: Techniques for Developing and Critiquing Quantitative Data Collection Tools for Program Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Meg Knight, Pima Prevention Partnership, mknight@thepartnership.us |
| Sonia Cota-Robles, Pima Prevention Partnership, scota-robles@thepartnership.us |
| Kay Calendine, Pima Prevention Partnership, kcalendine@thepartnership.us |
| Abstract: Conclusions drawn from quantitative data are only as accurate as the data on which they rely. Good measurement tools are fundamental to the process of collecting accurate data. As evaluators we are called on to develop quantitative measurement tools, collect data using quantitative measurement tools, and conduct data analyses using data that has been collected using quantitative measurement tools. There are many challenges and pitfalls in the development of good quantitative measurement tools. This demonstration will describe how evaluators at Pima Prevention Partnership have assessed various quantitative measurement tools to promote accuracy in the data collected. In some cases, the review has led to the adaptation of tools to more accurately gather the sought information. Strategies for developing and critically assessing measurement tools will be covered. |
| Poster 92: Designing Indicators that Measure the Extent of Program Intervention into Targeted Systems |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Natalia Kosheleva, Process Consulting Company, nkochele@yandex.ru |
| Abstract: Presence of negative and positive feedback loops inside the system(s) targeted by program interventions may result in non-linear relation between program inputs and outputs and between outputs and outcomes and thus complicate making judgments about the program effectiveness. Measuring the extent of program intervention into the targeted system(s) may help better understand how the program is working. The author has developed an approach to design indicators that measure extent of program intervention: 1. Define what system(s) are targeted by the program. 2. Describe the structure of the system(s): constituent elements and relations between them. 3. Identify system elements targeted by program intervention. 4. For each identified type of element set indicators that would measure how many elements are reached by the program. The presentation will include two examples of application of this approach in two real programs one aiming to introduce community service learning to schools and another aiming to enhance the quality of social reporting at media outlets. |
| Poster 93: A Needs Assessment of Immigrant Women Starting Businesses in Saskatchewan, a Mid-Western Canadian Province |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jana Kotrcova, University of Saskatchewan, jana.kotrcova@usask.ca |
| Peter Grant, University of Saskatchewan, peter.grant@usask.ca |
| Abstract: Research on female entrepreneurs has shown that they face challenges that are not commonly encountered by their male counterparts. Further, the vast majority of studies on immigrant-owned businesses have been conducted with male business owners. Knowledge about immigrant women entrepreneurs and issues specific to their entrepreneurial experiences, therefore, remains limited. Due to the current demographic trends in Saskatchewan, immigrant business owners are essential for maintaining the entrepreneurial spirit in the province. This study examined the needs, challenges, and barriers that immigrant women face when they want to start a business, and attempted to identify the gap between these needs and the services currently available to them in the community. Information collected via in-depth interviews is described which shows that immigrant women face some similar barriers to those faced by ethnic entrepreneurs. Recommendations for the development of services and policies that would encourage immigrant women to enter entrepreneurship are then discussed. |
| Poster 94: A Method for Creating Simple, Inexpensive and Valid Comparison Groups |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Morris Lai, University of Hawaii, Manoa, lai@hawaii.edu |
| Susan York, University of Hawaii, Manoa, yorks@hawaii.edu |
| Abstract: We have developed and tested an evaluation procedure that ensures equivalent comparison groups and greatly reduces the need for parental permission. After a project has been implemented for an appropriate length of time, we find a comparison student for each project student by first locating all nonproject students with similar demographic and other background variables. To control for school differences, we also ensure that the project student and comparison counterpart attend schools with similar levels of poverty. In comparison to randomized control trials (RCTs), our procedure also better resonates with indigenous communities and is most likely less expensive than an RCT. Despite Campbell and Stanley's denigration of matching to obtain equivalent groups, we assert that the rigor of the proposed procedure is at least equal to that of RCTs. We will present this alternative design, which has been used successfully in two evaluations. |
| Poster 95: Enhancing the Graduate Student Practicum Experience: A Student Assessment Instrument |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jennifer LeBeau, Washington State University, jlebeau@wsu.edu |
| Robert Owens, Washington State University, rwowens@wsu.edu |
| Michael Trevisan, Washington State University, trevisan@wsu.edu |
| Abstract: Although the importance of student assessment as part of mentoring in graduate practicum experiences has been recognized, limited literature exists on actual assessment instruments designed for this purpose. This paper describes a student assessment tool used to enhance the practicum or internship experience of graduate students in the educational psychology program at Washington State University. The instrument was designed using Brown and Dinnel's (1992) developmental model for supervising evaluation students. To date, the tool has been used to assess the progress of three doctoral- and two master's-level students working as research assistants in the Learning and Performance Research Center in the College of Education. The paper discusses use of the instrument in formative assessment of graduate student development. Limitations and recommendations for future research are discussed. Brown, R. D., & Dinnel, D. (1992). Exploratory studies of the usefulness of a developmental approach for supervising evaluation students. Evaluation Review, 17(1), 23-39. |
| Poster 96: Evaluation Apprentice: Real Life Evaluation Experience for Students and Evaluation Capacity Building for Community Organizations |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Nicole Lewis, University of Kentucky, |
| Abstract: One of the challenges of teaching program evaluation courses is creating opportunities for students to apply what they learn in the classroom to real life evaluation situations. This proposal describes the 'Evaluation Apprentice,' a project in an advanced evaluation course. The goals of the Evaluation Apprentice are two-fold: 1) to provide evaluation services to a program that might not otherwise have them and 2) to provide students the opportunity to apply their knowledge and skills. At the beginning of the course the 'client' gives a presentation about their evaluation needs. Students are placed on teams and each team develops an appropriate evaluation plan. Students then present their plan and receive feedback from the instructor and client. In the end students typically have more confidence in their evaluation capabilities and the 'client' has increased his/her evaluation capacity and has received valuable ideas regarding program and evaluation design. |
| Poster 97: Strategies and Principles which Guide the Work of the Staff in Branco Weiss Schools for Youth at Risk: What we have learned from our successes and identifying future challenges |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chen Lifshitz, Ashkelon Academic College, chenl@brancoweiss.org.il |
| Abstract: The creation of normative educational frameworks for youth who have dropped out is particularly challenging. Branco Weiss schools for youth at risk absorb youth who have dropped out of the regular education system and are from weak socio-economic backgrounds. Our system employs ongoing evaluation; outcomes and outputs highlight a high rate of satisfaction on the part of the pupils, similar reporting by pupils and teachers, but that discipline is still an issue. However, evaluation methodologies to date have done little to examine the strategies used by the schools. During the 2009/10 school year our evaluation methodology has expanded to include case studies which examine the models being applied in 4 of the 10 schools for youth at risk in the network. Preliminary findings highlight critical issues in the methodologies being applied such as the critical role/image played /projected by the principal, and the dichotomy regarding pupil support vs. teacher support. |
| Poster 99: First-Year Evaluation for Community Connections, Community Senior Services, Claremont, California |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pi-Ju Liu, Claremont Graduate University, pi-ju.liu@cgu.edu |
| Wenjie Tang, Claremont Graduate University, haretanya@gmail.com |
| Anna Fagergren, Claremont Graduate University, annafagergren@gmail.com |
| Jonathan Snavely, Claremont Graduate University, jesnavely@gmail.com |
| Michele Milliken, Community Senior Services, cssmichele@linkline.com |
| Sandee Hayden, Community Senior Services, csshayden@linkline.com |
| Floy Biggs, Community Senior Services, cssfloy@linkline.com |
| Abstract: Research suggests increasing demands on existing transportation programs intended for older adults, nonetheless, formal evaluation reports are scarce. Responding to unmet transportation needs in the Pomona Valley area in Southern California, Community Senior Services (CSS) initiated a volunteer-driver program, 'Community Connections', in November 2010. Evaluators from Claremont Graduate University were enlisted to design and conduct a process evaluation. The evaluators, in collaboration with CSS, aim to record seniors' needs for the program, because no existing literature has examined the need in Pomona Valley. In addition, ridership profiles and program use are going to be analyzed across the eight months of program operation. And finally, the growth of stakeholders' evaluation capacity will be documented. Working with the evaluators, stakeholders will deliberate on the measurements and learn strategic evaluative thinking and practice. Final report is expected to be published in October. |
| Poster 100: Increasing the Reference-Making Capacity of Social Network Analysis Metrics: Incorporation of Logistic Regression Modeling |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chih-shen Owen Lo, University of British Columbia, vancouverowen@gmail.com |
| Arwa Alkhalaf, University of British Columbia, arwaalkhalaf@hotmail.com |
| Kristy Jang, University of British Columbia, kristyjang@gmail.com |
| Sandra Mathison, University of British Columbia, sandra.mathison@ubc.ca |
| Abstract: Applying social network analysis (SNA) in program evaluation often involves the challenges of explaining the meanings of SNA metrics (e.g., betweenness, closeness, degree) to clients and, further, finding ways for meaningful application of these results in program monitoring and enhancement. In this study, logistic regression is used to elucidate the factors underlying the existence/non-existence of a tie or relationship between actors who were part of a professional development program. A total number of 171 relationships among 19 actors (i.e., including no ties, one-way and reciprocal ties) are treated as the categorical dependent variable. The preliminary independent variables include: linguistic closeness, years of employment, existence of a mediator, power hierarchy, collegiality, and gender. The results present a new method of interpreting and using SNA metrics for program evaluation through a combination of quantitative analysis and SNA, which encourages an increased reference-making capacity and usability of SNA in future program evaluations. |
| Poster 48: Building Relationships with Policymakers: Evaluating What Works In Rural and Urban Settings |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Nikole Lobb Dougherty, Washington University, St Louis, nlobbdougherty@brownschool.wustl.edu |
| Chris Robichaux, Washington University, St Louis, crobichaux@gwbmail.wustl.edu |
| Tanya Montgomery, Washington University, St Louis, tmontgomery@gwbmail.wustl.edu |
| Amy Stringer Hessel, Missouri Foundation for Health, astringerhessel@mffh.org |
| Cheryl Kelly, St Louis University, kellycm@slu.edu |
| Jessi Erickson, St Louis University, ericksjl@slu.edu |
| Abstract: The Missouri Foundation for Health is currently funding a comprehensive evaluation of a multi-site obesity prevention initiative. Currently, 36 grantees implement physical activity and healthy eating projects in both rural and urban settings in Missouri. The presentation will examine the difference between urban and rural grantees when developing partnerships with policymakers. As part of the evaluation's core data set, information has been collected using a mixed methods approach that tracks policy activities and captures details about effective strategies for building relationships with policymakers in different environments. The authors will highlight effective strategies and the impact of policymaker relationships on outcome sustainability. |
| Poster 101: The Value of Adding Complex Problems to Support Collaborative Learning at the United States Army Command Sergeant Major's Development Program |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Linda Lynch, United States Army, linda.l.lynch@us.army.mil |
| Joe Parson, United States Army, joe.b.parson@us.army.mil |
| Abstract: The US Army Command Sergeant Majors Development Program embraces 'collaborative learning' distinguished by a focus on joint construction of knowledge by learners and instructors - called Mentor/Facilitators, with the primary aim to construct new knowledge within their exchange of ideas and experiences. Over the past year, CSMDP realized that lacking essential experiences of students and instructors resulted in reduced learning. CSMDP constructed complex problems that encompassed the missing essential learning, and then added to the curriculum to determine the value to students and impact on learning quality. Two data collection modalities were employed to demonstrate the values that were realized and positive impact on student and Mentor/Facilitator learning. |
| Poster 102: Developing a Framework of Shared Outcomes and Measures for Environmental Community Based Initiatives |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michelle Mandolia, United States Environmental Protection Agency, mandolia.michelle@epa.gov |
| Abstract: Program Evaluation presupposes the precise operational definitions of key program elements in order to yield evaluations that will help to assess program design, process, customer satisfaction, and outcomes. Community Based Initiatives in various fields are challenged by this requirement due to widely varying conceptualizations of community and variant approaches as to how each of these divergent concepts may be measured. This poster will share how members of a United States Environmental Protection Agency workgroup are approaching the challenges of measuring these programs with an eye to evaluation. The poster will outline both the process the group is using for the task and the measurement approach being developed. |
| Poster 103: "Attention Travelers!": Evaluating Health Communication with On-the-Move Audiences |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amanda McWhorter, Centers for Disease Control and Prevention, frh6@cdc.gov |
| Rebecca Myers, Centers for Disease Control and Prevention, asx1@cdc.gov |
| David Hunter, Centers for Disease Control and Prevention, exu5@cdc.gov |
| Monica Selent, Centers for Disease Control and Prevention, igv1@cdc.gov |
| Gabrielle Benenson, Centers for Disease Control and Prevention, gkb6@cdc.gov |
| Abstract: Since the 1970s, Travel Health Alert Notices (T-HANs) have been used by the Centers for Disease Control and Prevention (CDC) to provide health alerts to international travelers arriving at U.S. ports of entry. More recently, T-HANs have been distributed to travelers arriving from countries where outbreaks have occurred. T-HANs urge travelers to assess personal risk, monitor for symptoms, and seek health care if symptoms arise. The effectiveness of such communication is difficult to evaluate. Efforts to assess the effectiveness of T-HANs have ranged from formative evaluation through focus groups with international travelers and physicians who might see patients with recent international travel history to outcome research through a cross-sectional survey of travelers arriving from Haiti during the cholera outbreak. A review of T-HAN evaluation methods, challenges and successes, and lessons learned will be presented, and ideas for methods and collaboration will be sought. |
| Poster 104: What Do You Believe In? Building the Evaluation Capacity of Community- and Value-Based Interventions |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Vera Miles, Community Science, vmiles@communityscience.com |
| LaKeesha Woods, Community Science, lwoods@communityscience.com |
| Abstract: Aligning evaluation with the values of an intervention and engaging all stakeholders in evaluation present challenges and opportunities to build capacity to promote the well being of communities. To effectively serve diverse populations, it is critical to tailor interventions to the populations' strengths and needs, using evaluation to guide the ongoing development of interventions. The evaluation of a federally-funded system of care (SOC) initiative will be highlighted to illustrate these opportunities and challenges. The SOC intervention and evaluation aims to embed the system of care principles and values of family driven, youth guided, cultural and linguistically competent, and community-based into its services and supports for children with mental health needs. As evaluators, we work to elevate the voice of the community and help stakeholders build their capacity to use data for sustainability. Lessons learned for building the capacity of stakeholders to promote health will be presented. |
| Poster 105: Social Components and Initiation of Drug use Among Young People in Tehran-Iran |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jila Mirlashari, Tehran University of Medical Sciences, jmirlashari@yahoo.com |
| Apo Demirkol, University of Sydney, demirkolster@gmail.com |
| Mahvash Salsali, Tehran University of Medical Sciences, m_salsali@hotmail.com |
| Hassan Rafiey, University of Social Welfare and Rehabilitation Science, hassan441015@gmail.com |
| Jahanfar Jahanbani, Tehran Islamic Azad University, jjahanbani@yahoo.com |
| Abstract: Current estimates suggest that Iran has the highest rate of opium addiction per capita in the world. It has been suggested that multiple elements contribute to the process of drug use. The aim of this study is to explore the probable components in the society that might play a role in the initiation of drug use among young adults. The study is qualitative in nature. Twenty four in- dept interviews were conducted with drug user young men and women and their family members .Based on our data analysis developing a pro-drug attitude, conformist attitude of society towards treating youth and young individuals and having weak approach toward prevention were identified as important determinants of substance use. The results of this research suggest that dealing with a major problem such as drug use needs early intervention and comprehensive assessment of the context in which young people live and use substances. |
| Poster 106: The Role of Principals in Afterschool Program Evaluations: Expanding Values and Complexity |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Monica Montijo, Claremont Graduate University, monicamontijo20@hotmail.com |
| Tiffany Berry, Claremont Graduate University, tiffany.berry@cgu.edu |
| Abstract: A review of afterschool program literature gives limited insight into the links between the regular school day and afterschool programs. Education research suggests stronger relationships between principals and teachers lead to better academic outcomes for students, yet rarely have evaluators investigated similar links between the regular school day and the afterschool program. The purpose of this poster is to illustrate the values espoused by an underrepresented stakeholder: school principals. Using data from an on-going evaluation of After-School All-Stars, Los Angeles, a program serving disadvantaged middle and high school students, we will present results from 10 principal interviews to illustrate (1) principals' values in relation to after school programs, (2) how principals' values changed the nature of our evaluation work, and (3) the inherent complexity involved with incorporating multiple stakeholder groups. |
| Poster 107: Monitoring and Programme Evaluation: Capacity Building Experience in Western India |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Undurthy Narayana, Regional Institute of Education, undurthy@gmail.com |
| Abstract: The purpose of capacity building programme is development of awareness among educational functionaries in integrating monitoring and programme evaluation. The success of any educational programme depends on the efficacy of monitoring. Some schemes of academic monitoring are evolved to regulate school practices for quality improvement. The purpose of academic monitoring is to improve classroom practices and in turn learning achievement. In this paper six quality dimensions of academic monitoring are considered. They are Classroom Practices, Remedial Practices, Teaching Learning Material, School Performance, Teacher Performance and Evaluation Practices. Educational functionaries are made aware of Evaluation standards, Evaluation framework, Programme Logic Model, Evaluation Designs, Types of programme evaluation and Sampling procedures. This capacity building programme improved awareness levels of educational functionaries on basic terms of programme evaluation. They are able to prepare evaluation framework and programme logic model. They are further aware of utilizing monitoring data for evaluation purposes. |
| Poster 108: Informing Policy Through Monitoring and Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rachel Nelson, Centers for Disease Control and Prevention, rnelson2@cdc.gov |
| Nicholas Di Meo, Centers for Disease Control and Prevention, ndimeo@cdc.gov |
| June Liang, Emerging Infectious Diseases Program, China, june@cn.cdc.gov |
| Abstract: CDC's Global Disease Detection (GDD) program was established in 2004 to rapidly identify and contain emerging health threats. GDD currently operates eight Regional Centers that build broad-based public health capacity in host countries and within the region. In 2006, GDD began systematically collecting data on the achievements of the Regional Centers as part of its monitoring and evaluation (M&E) framework. This session will discuss the development of the GDD M&E framework and how it has improved over time. Staff from CDC in Atlanta and field staff from one Regional Center will discuss systems and best practices for data collection, quality assurance, management, and analysis. In addition, this session will highlight how monitoring and evaluation data can be used for policy purposes such as demonstrating accountability and communicating to program stakeholders, policymakers, and partners, and how to integrate monitoring and evaluation activities into a larger communications strategy. |
| Poster 109: Video Ethnography in Program Evaluation: Developing Deep Knowledge of Patient Experiences to drive Quality Improvement |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Estee Neuwirth, Kaiser Permanente Care Management Institute, estee.neuwirth@kp.org |
| Patricia Price, Kaiser Permanente Care Management Institute, patricia.m.price@kp.org |
| Ana Jackson, Kaiser Permanente Care Management Institute, ana.h.jackson@kp.org |
| Jim Bellows, Kaiser Permanente Care Management Institute, jim.bellows@kp.org |
| Abstract: Research Objective: We adapted traditional ethnographic methods to the imperatives of operational settings in health care to rapidly generate rich knowledge about how patients experience health care and to inform assessment, evaluation and quality improvement efforts. Study Design: In a practical adaptation of traditional fieldwork methods, video ethnography uses shorter timelines and smaller samples. We conduct participant observation and in-depth semi-structured interviews in inpatient, ambulatory care, and home settings, using inexpensive video equipment to record data. Principal Findings: Video ethnography methods are broadly applicable to health care quality improvement activities and quickly yield highly meaningful patient-centered perspectives that lead to innovative solutions and optimization of programs within a short time frame. Video is an effective means of communicating ethnographic insights to senior executives and front-line personnel. In addition, we found that with training, video ethnography methods can be successfully transferred beyond academically-trained social scientists. |
| Poster 110: Characteristics of Success in Residential versus Community Based Treatment: Applications of Clustering and SEM Techniques |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Frederick L Newman, Florida International University, newmanf@fiu.edu |
| Mark DeKraai, University of Nebraska, Lincoln, mdekraai@nebraska.edu |
| Meredith Hope, University of Nebraska, mhope@nebraska.edu |
| Abstract: Nebraska's youth have one of the highest rates of out-of-home and residential mental health placement as well as longer lengths of residential stays in the nation. Community services have increased to reduce need for out-of-home placement. Since 2009, residential services fell by 9% partially due to increased community-services, yet the clinical, functional, and systems outcomes have not been explored systemically. The study uses an integrated, cross-systems database, from a sample of 2,000 boys and girls, aged 5 to 18 years, containing variables contributing to current and post-treatment risk and success. The trajectory of treatment outcome is assessed by the Child and Adolescent Needs and Strengths (CANS) scale, measuring youth's risk and protective factors at three intervals. Data are collected by service providers and maintained by Magellan Health Services and the Nebraska DHHS Division of Children and Family Services. Results will inform decisions regarding evidence-based alternatives likely to promote positive outcomes. |
| Poster 111: How to Write an Effective Final Evaluation Report |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Adriane Niare, Centers for Disease Control and Prevention, aze2@cdc.gov |
| Abstract: A final evaluation report is used to present the findings, conclusions and recommendations of an evaluation and is an important product of the entire evaluation process. The final evaluation report provides a basis for results that are useful, credible, and relevant for decision-making on policies and programs particularly around learning what works in a program and why and for identifying learnings for future program improvement. This workshop will introduce new evaluators to a guidance document and tools created by the Centers for Disease Control and Prevention, Office on Smoking and Health that provides practical guidance on writing a final evaluation report. By the end of the workshop the participants will know the elements that compose an effective final evaluation report and the importance of having a well-designed dissemination strategy for the final evaluation report. |
| Poster 114: Educational Equity: Re-Conceptualizing Effectiveness in Teacher Performance Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Khawla Obeidat, University of Colorado, Denver, khawla.obeidat@ucdenver.edu |
| Abstract: Teacher effectiveness and performance evaluation increasingly become a major topic around kitchen tables as well as in research and professional meetings. Currently there are several teacher evaluation models that examine teacher competency in terms of content related practices or activities. However, these models lack an important aspect that relates to the increasingly divers students population who come from different cultural, lingual, ethnic/racial, and familial backgrounds. The needs of such diverse body of students require teachers who are well-equipped-and-trained to work with and teach such students. The current proposed study aims to address three goals. First, survey all available tools and models of teacher performance evaluation. Secondly, examine the strengths and weaknesses of these models and instruments. Finally, propose a new theoretical framework the eventually produces standardized-research-and-performance based on the premise of achieving social equity and justice by considering student's culture as the corner stone of the new teacher performance evaluation model. |
| Poster 113: Re-Conceptualizing Culturally Responsive Evaluation: Evidence-Based Stakeholder Perspective |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Khawla Obeidat, University of Colorado, khawla.obeidat@ucdenver.edu |
| Rodney Hopson, Duquesne University, hopson@duq.edu |
| Abstract: framework is that evaluators successfully capture and present the voices of stakeholders. Evaluation is tailored in a manner that meets and fits targeted clients' culture in terms of their needs, values, beliefs, experiences, and 'way of life'. Although that what 'we evaluators' believe we do, however, evidence based on the stakeholders' perspectives are hardly presented. The current study raises the question about the possibility of building a standardized scale that psychometrically sound (reliable and valid) captures the stakeholder's stand on the notion of if evaluation is culturally-based and responsive. Evaluation lacks trust of stakeholders, the desired proximal and distal effects of evaluation are unachievable. Credibility and trust are both fundamental blocks for a successful evaluation that leads to change. Without stakeholders consent that the program evaluation is culturally responsive and based then it does not matter what 'we evaluators' claim! |
| Poster 115: Evaluating the Impact of HIV Disclosure on HIV Transmission Risk |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ann A O'Connell, The Ohio State University, aoconnell@ehe.osu.edu |
| Sandra J Reed, The Ohio State University, sreed@ehe.osu.edu |
| Julianne M Serovich, The Ohio State University, serovich.1@osu.edu |
| Abstract: Interventions to assist HIV+ persons in disclosing their status to potential sexual partners can play an important role in curbing rates of HIV transmission, particularly among men who have sex with men, a disproportionately represented group in national HIV/AIDS statistics. Using baseline data from a randomized clinical trial of an HIV disclosure intervention, we present estimates of reduction in HIV transmission risk due to disclosure. At baseline, the effectiveness of disclosure at reducing risk of transmission relative to non-disclosure was 21.5%. We discuss the utility of a mathematical probability model for evaluating effectiveness of the intervention through comparison of reduction in HIV transmission risk due to disclosure relative to the disclosure-related HIV transmission risk in a standard-of-care (case-management) control condition. Implications for design and evaluation of disclosure interventions are highlighted. |
| Poster 116: The Impact of AIDS Mitigation Project on Orphans and Vulnerable Children (OVC) in Selected States in Nigeria |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Muyiwa Oladosun, MiraMonitor Consulting Limited, fso226@yahoo.com |
| Charles Toriola, MiraMonitor Consulting Limited, stinkinglyrichie@yahoo.com |
| Femi Oladosu, MiraMonitor Consulting Limited, feoladosu@yahoo.com |
| Gloria Affiku, MiraMonitor Consulting Limited, gaffiku@yahoo.co.uk |
| Abstract: The AIDS Impact Mitigation (AIM) project was implemented by Winrock International between October 2006 and October 2010 to reduce the impact of HIV on orphans in Nigeria. Education is a key part of the project targeting orphans and vulnerable children (OVC) aged 6-17. Out of school OVC were put in a nine months intensive English and mathematics class called Acada Learning Centers (ALC), before been reintegrated back to school (some reintegrated OVC did not go through ALC). The end-of-project evaluation employed mixed methods sampling of 816 Acada, and 684 reintegrated OVC, and five focussed groups discussions with OVC, and 14 key informant interviews with their teachers. Acada OVC's ability to read and write increased significantly (9% vs. 76%; and 10% vs. 73%, respectively), and for reintegrated OVC results were similar (12% vs. 72%, 12% vs.73% respectively). OVC reported significant improved access to health and psychosocial support services among others. |
| Poster 117: Action in Health: A Brazilian Experience of Monitoring and Capacity Building on Health Promotion |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Thais Oliveira, National School of Public Health, thscoutinho@yahoo.com.br |
| Elizabeth Moreira, Oswaldo Cruz Foundation, bmoreira@ensp.fiocruz.br |
| Egleubia Oliveira, National School of Public Health, biaol@ensp.fiocruz.br |
| Aline Gonalves, National School of Public Health, leal.aline@gmail.com |
| Abstract: Since 2009 the project 'Ação Saúde' aims to mobilize communities and support integrated management of projects for improvement of maternal and child health and decrease of infant mortality by the establishment of a network of local groups ('health promotion cells'). It has been developed specific educational material to support capacity building workshops taking problem-ased pedagogy and Paulo Freire's concepts of adult education. It also involves systematic monitoring and evaluation (M&E) of the projects. By analysis of management and monitoring reports, this paper aims to report this experience and analyze the process of local empowerment. Results show that diverse local contexts (urban, rural or mixed rural and urban cities) have had important influence to the projects implementation and sustainability and to increase an institutional culture of M&E. Experiences of this nature are potent to promote coordinated public heatlh policies linking public administration and society and strengthening social betterment premises. |
| Poster 118: Mentoring Evaluators: What's Happening in Canada |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lisa O'Reilly, Independent Consultant, enquiries@lisaoreilly.ca |
| Jane Whynot, Whynot & Associates, Evaluation & Research Consulting, jwhynot@magma.ca |
| Judy Lifshitz, Agriculture and Agri-Foods Canada, Judy.Lifshitz@AGR.GC.CA |
| Abstract: Mentoring offers one avenue by which individuals are able to communicate their needs in support of professional development. While mentoring initiatives are not new, only recently has there been research of mentoring specifically oriented to evaluators and the evaluation function. In Canada, a Core Mentoring Working Group initiated efforts in 2010 to develop a national program specifically for evaluators. Grounded in research, we developed a business case to pilot a mentoring program in multiple sites across the country. The Group would like to take this opportunity to present data from our research efforts on the current demand, dimensions, advantages, perceived disadvantages and risks, possible longer term options, models and related strategies for implementing such a program. Join us to share ideas and talk about how you can benefit from evaluation specific mentoring. |
| Poster 119: Redesigning Evaluation Reports: The Art and Science of Communicating Evaluation Findings |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tanya Ostrogorsky, Oregon Health & Science University, ostrogor@ohsu.edu |
| Anjanette Raber, Oregon Health & Science University, rabera@ohsu.edu |
| Abstract: The advent of the computer has provided us with the opportunity to readily create charts and graphs to illustrate key report findings; however, research demonstrates these illustrations are often created in a way that lacks tone, emphasis, functionality, and creativity. In turn, this frequently requires stakeholders to sift through evaluation reports that were created without visual intentionality. This presentation will focus on the process we engaged in to redesign a monthly evaluation report intentionally focused on the design principles of tone, emphasis, functionality, and creativity. After comparing stakeholder assessments of the pre- and post-redesigned evaluation reports we identified the key elements that supported leading stakeholders on a visual journey. This presentation will highlight the importance of visual communication within Oregon Health & Science University School of Nursing monthly evaluation reports and identify what stakeholders recognized as valuable and what muddles the lucidity of the evaluation findings for our audience. |
| Poster 120: New Directions Project: Training Graduate Students in Audience Research & Evaluation Within Informal Learning Settings |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kathryn Owen, Woodland Park Zoo, kathryn.owen@zoo.org |
| Nick Visscher, University of Washington, vissche2@uw.edu |
| Abstract: The New Directions Project is a 3 year IMLS funded grant project bringing together staff from the UW Museology Graduate Program and the Woodland Park Zoo to train graduate students in audience research and evaluation through a series of coursework, directed fieldwork, mentorship and community museum collaboration. This project sought to adopt a method of learning that involved professionalism through practice rather than strictly classroom work. The process of developing a structure of practice and community partnerships will be shared in addition to examples of student work and lessons drawn from the project's summative evaluation. |
| Poster 121: Evaluation of Practical Skills of Undergraduate Medical Students by an Innovative Mini-Objective Structured Practical Examination Design for Hospital Acquired Infections |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chitra Pai, Mahatma Gandhi Mission Institute of Health Sciences, chitrapai67@gmail.com |
| Abstract: Evaluation is the driving force promoting learning in medical students who are mostly 'exam oriented'. Practical skills are also actively learnt only when evaluated. Many developing countries face the challenge of hospital acquired infections. Doctors play a pivotal role in spreading these infections due to noncompliance with preventive measures. This could stem from lacunae in the traditional education system wherein the practical skills aren't evaluated. We designed an innovative technique for evaluating practical skills such as effective hand washing and safe hospital waste disposal. This consisted of a Mini- Objective Structured Practical Examination (OSPE) with stations where students were assessed by examiners with the help of a checklist. Details of the various steps were evaluated objectively such that examiner variation was eliminated. A statistically significant increase in acquisition and retention of practical skills was noted. This promises to be a significant step towards essential skills training of undergraduate medical students. |
| Poster 122: Searching for the Whole: Systems Modeling in a University-School-Informal Education Partnership for Educating about the Science of Health |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Loran Parker, Purdue University, carleton@purdue.edu |
| Omolola Adedokun, Purdue University, oadedok@purdue.edu |
| Soo Yeon Shin, Purdue University, shin10@purdue.edu |
| Wilella Burgess, Purdue University, wburgess@purdue.edu |
| Sandra Amass, Purdue University, amasss@purdue.edu |
| Abstract: The proposed poster uses the evaluation of a University-School-Informal Education Partnership for Educating about the Science of Health as an example of the benefits of the systems modeling process for evaluating complex projects that both align preexisting activities and develop and implement new activities that serve a common educational goal across diverse education contexts. We draw on examples and lessons learned from a University-School-Informal Education Partnership to describe the process of creating a systems model and to demonstrate a context in which it was beneficial to develop this model after a logic model had been created and the project was in the pilot implementation stages. The proposed poster will illustrate the model in detail, describe the advantages and disadvantages of creating a systems model during pilot implementation of a partnership project, and describe future directions for this evaluation. |
| Poster 124: The Centers for Ocean Sciences Education Excellence (COSEE): Evaluating Scientist Engagement |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diana Payne, University of Connecticut, diana.payne@uconn.edu |
| Pam Van Dyk, Evaluation Resources LLC, evaluationresources@msn.com |
| Abstract: The Centers for Ocean Sciences Education Excellence (COSEE) comprise a national network of 15 Centers focused on nurturing collaborations among research scientists and educators to advance ocean discovery and make known the vital role of the ocean in our lives. In 2009 and 2010, the network conducted national cross-Center evaluation of scientists engaged with COSEE. The study methodology consisted of survey research. Respondents represented a diverse group of ocean scientists from across the United States. The purpose of this poster session will be to share highlights from the studies, which focused primarily on the many ways in which scientists engage in outreach and education in their regions. |
| Poster 123: Using Common Measures: Understanding the Importance of Dosage |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pamela Payne, University of Arizona, pgargle@email.arizona.edu |
| Daniel McDonald, University of Arizona, mcdonald@ag.arizona.edu |
| Abstract: Over the past two years Cooperative Extension programs across several states have participated in a pilot study to determine the extent to which individual state Extension programs can use identical instruments to measure outcomes as part of their overall evaluation efforts. This poster will discuss the importance of measuring dosage, how those constructs are defined, and the lessons learned from the pilot study. Additionally, site evaluators were asked to assess the 'goodness of fit' of the instrument and the methods used as they applied to their specific programs. The primary focus of this poster will be results of the pilot study and how the lessons learned might be apply to Extension programs. |
| Poster 125: Utilizing an Assessment of Evaluation Competencies to Support Extension Professionals |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Cheryl Peters, Michigan State University, cpeters@anr.msu.edu |
| Bruce Haas, Michigan State University, haasb@msu.edu |
| Michelle Rodgers, Michigan State University, rodgersm@anr.msu.edu |
| Abstract: Presented is a field-tested and reliable survey to measure evaluation competencies of Extension professionals. Staff training needs and implications for assessing knowledge and skills before organizational capacity building activities are highlighted. A survey (n=142) of Extension professionals reporting their competencies was used to set priorities for trainings and to establish a baseline to assess future organizational development and change as investments are made. Specifying needed evaluation competencies has been a challenge in Extension and is of interest to researchers and professional evaluators (Boyd, 2009; Ghere, et al., 2006). A taxonomy of evaluator competencies was published and revised (King et al., 2001; Stevahn et al., 2005) in the American Journal of Evaluation. Factor analysis of survey data verified the conceptualization of evaluation competencies for Extension professionals but suggested there may be sub-groups of skills to further explore related to training needs. |
| Poster 126: Building Evaluation Capacity in Informal Science Education (ISE) Through Strategic Collboration |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tina Phillips, Cornell University, tina.phillips@cornell.edu |
| Gil Noam, Harvard University, gil_noam@harvard.edu |
| Rick Bonney, Cornell University, rickbonney@cornell.edu |
| Anahit Papazian, Harvard University, apapazian@mclean.harvard.edu |
| Jane Aibel, Harvard University, jaibel@mclean.harvard.edu |
| Abstract: A major goal of Informal Science Education (ISE) is to address and improve upon, the lack of standardized tools and strategies to effectively measure outcomes and impacts across programs. Through a dynamic collaboration between the Cornell Lab of Ornithology's DEVISE project and Harvard Medical School's Program in Education, Afterschool, and Resiliency (PEAR), we intend to test and validate existing, new, or modified instruments in afterschool settings and in projects that engage the public in scientific research (sometimes referred to as 'citizen science'). Results from the evaluations will be made available via an online toolkit and searchable database of valid and reliable assessment tools. Together, the collaboration will develop common technical standards that evaluators, practitioners, and researchers can use to build customizable evaluation designs using standardized tools and instruments. With widespread adoption of these tools, we envision future research examining cross-programmatic impacts and outcomes from ISE projects. |
| Poster 127: From "Nouns and Verbs" to "Protect and Serve:" The Use of an Academic Assessment Tool in Predicting Police Academy Success |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Laura Plybon, Drury University, lplybon@drury.edu |
| Tony Bowers, Drury University, tbowers@drury.edu |
| Abstract: This study examined whether the ASSET reading/writing skills test was effective in predicting Drury University Law Enforcement Academy students' academy success. Data on the ASSET skills test, student GPA, and the Peace Officer Standards and Training (POST) test were gathered from six cohorts across six semesters (Fall 2007 to Fall 2010). Attrition analyses showed that attritors of the academy (n = 4) had significantly lower ASSET test scores (m = 72.22) than the non-attritors (n = 155) (m = 82.57), F (4, 155) = 4.57, p < .05. The final sample was comprised of 155 academy students (91% male, 9% female, mage = 28.45 years). Results showed no significant age, gender, or cohort main effects or interaction effects on the outcomes of ASSET, GPA, and POST scores. Results from mediational regression analyses documented that higher ASSET scores significantly predicted higher GPAs, which in turn significantly predicted higher POST test scores. |
| Poster 128: Using Forms Technology to Facilitate Use of Common Indicators of Impact |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paul Pope, Texas AgriLife Extension Service, ppope@tamu.edu |
| Abstract: This poster highlights the use of two forms technologies (Teleform and Adobe Professional) and Java scripting to create a web-based system that allows Extension faculty to build customized, scannable evaluation instruments for individual events while still incorporating common indicators of impact that allow aggregation of results across many events (program level results). Two methods of instrument creation will be presented as will the fruits of this system (aggregated results). |
| Poster 129: Exploring the Net Promoter Score's Worth in Extension Program Evaluations |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paul Pope, Texas AgriLife Extension Service, ppope@aged.tamu.edu |
| Abstract: The Net Promoter Score® (NPS) is a measure of customer loyalty. NPS is calculated from responses to one simple question, measured on a 0-to-10 rating scale: 'Would you recommend us to a friend or colleague?' Based on the NPS, each customer is placed into one of three categories: promoters, passives, and detractors. Ultimately, the goal of using the NPS is to increase promoters and decrease detractors. A few years ago, Texas AgriLife Extension Service used this question on several evaluation instruments in place of the dichotomous question: 'Would you recommend us to others?' How does the NPS relate to various elements of an Extension program (content, teaching, setting, anticipated economic benefit)? What program elements and/or participant characteristics are associated with program promoters, passives, and detractors? Has use of the NPS produced insights useful for program evaluation purposes? This poster focuses on these questions. |
| Poster 130: Comparison of Blog and Newspaper Coverage of a Food Recall |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Simani Price, Westat, simaniprice@westat.com |
| Lance Potter, The Potter Group, lance.potter@thepottergroup.net |
| Craig Lefebvre, South Florida, rcraig.lefebvre@gmail.com |
| Abstract: Dissemination of health information through new media has increased exponentially in recent years. In 2009, the CDC and FDA created a blog to discuss the recall of peanut butter containing products linked to salmonella poisoning. This recall began after eight people died and more than 500 people in 43 states, half of them children, were sickened. It involved over 400 consumer products. This study compares blog discussions and newspaper articles about the peanut recall issue. Our hypothesis is that newspaper content will largely be objective reporting of the news for the general public, reliance on credible sources and blend syndicated stories with a local angle. In contrast, we hypothesize that the informal nature of the blogosphere will frame discussions on this topic in terms of personal experiences, consumer products and generally evoke more emotion and humor than newspapers. Accuracy and depth of information presented will also be examined. |
| Poster 131: Increasing Efficiency of Service Provision in a Forensic Mental Health System: An Evaluative Study of a County-Funded On-Site Forensic Behavioral Health Program |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bethany Ranes, Ranes Nonprofit Grants and Consulting, bethany@ranesgrantconsulting.com |
| Abstract: This study attempted to determine if an on-site forensic mental health treatment program staffed by graduate interns would effectively meet the needs of an incarcerated population for a low cost, and what factors would maximize program performance. The program in this study was analyzed on the basis of cost, success in meeting its contractual objectives, and number of juveniles served. Additionally, a 15-month pilot version of the program was compared to the current program in order to determine if a number of systematic policy modifications were effective in increasing performance of the current program. Results showed that while the program did not meet all its contractual objectives, program performance was significantly increased by the implementation of empirically-founded policies. If the existing process of policy analysis and monitoring is maintained, and all contractual objectives are met, the program may provide an extremely cost-effective solution for forensic mental health provision. |
| Poster 132: Measuring Client Feedback in Child Welfare Organizations |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jolanta Rasteniene, Peel Children's Aid, jrasteniene@peelcas.org |
| Brenda Moody, Peel Children's Aid, bmoody@peelcas.org |
| Abstract: Peel Children's Aid is a child protection agency in Ontario, Canada which provides services mandated by legislation. As an organization we have been focusing on continuous quality improvement for a number of years. A key component of this work is listening to clients by systematically gathering and analyzing data and using the results to inform service improvements. To obtain credible and valid information from clients, we have adopted a multi-method approach. This approach includes obtaining feedback at specific program as well as organizational levels. Examples at the program level include the use of surveys for permanency planning case conferences and kin care. At the organizational level, in addition to a mandatory complaints process, we conduct an annual survey of clients receiving service. This presentation will provide an overview of the two of these approaches, as well as how the results have been used to effect change. |
| Poster 133: Visualizing Programmatic Success Through Data: The Evolution of Data Presentation at the Southeast AIDS Training and Education Center |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shenee Reid, Southeast AIDS Training and Education Center, sreid4@emory.edu |
| Stephanie Beane, Southeast AIDS Training and Education Center, sbeane@emory.edu |
| Rebecca Culyba, Southeast AIDS Training and Education Center, rculyba@emory.edu |
| Abstract: The Southeast AIDS Training and Education Center (SEATEC) is the U.S. Public Health Service designated AIDS Education Training Center for six southeastern states. Since 1988, SEATEC has trained healthcare providers on the diagnosis and management of patients with HIV. The evolution of the process and outcome evaluation of SEATEC trainings has been driven both by requirements of funders and by the introduction of a management culture focused on transparency. SEATEC has responded by presenting evaluation findings in ways that increase overall utility to funders, management, and the overall program alike. Using visualization techniques, a complete story of programs is presented using data, which encourages interaction and participation of stakeholders. Presenters will outline the evolution of data collection to demonstrate successful data presentation techniques such as newsletters, summary reports, and maps. The aim of this demonstration is to provide the audience with data presentation tools that aid data driven decisions. |
| Poster 134: I Was an Action Researcher: Urban School Confidential An Evaluation Narrative in 16 Panels |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| William Rickards, Alverno College, william.rickards@alverno.edu |
| Abstract: The challenges of urban school districts are complex, complicated, and frequently contested. This poster examines the implementation of a two-year project evaluating an action research program for urban teachers. The evaluation was begun in the third year of the Action Research Fellows and employed a primarily descriptive, qualitative strategy -through individual and group interviews, participant observation, and document analysis—to provide an elaborated understanding of the program in operation, its key elements, and implications for further related interventions. Using the format of a graphic novel, the poster uses a composite interview as a narrative to track the experience of participants and identify key elements and observations, and then reflect on the evaluation itself—as a kind of action research. |
| Poster 135: A Quasi-Study of A National Implementation: Identifying Protective Factors to Help Youth from Military Families |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Melissa Rivera, National Center for Prevention and Research Solutions, mrivera@ncprs.org |
| Abstract: This study investigated the efficacy of Stay on Track, an evidence-based program, through a quasi-experimental design. The national evaluation methodology was replicated to examine the quality of implementation and adherence to program material in 12 Utah classrooms. Levels of risk and protective factors were examined to determine if the results for Stay on Track participants from military families differed from those in the general population. The data were further examined to determine whether the same variables differed among students dealing with a parent's deployment in comparison with students reporting no current family member on deployment. Previous studies conducted by NCPRS have shown that the program empowers youth by providing knowledge and life skills relevant to health-promoting behavior. The intervention has been associated with short-term improvements in student motivation to abstain from substance use, personal competence skills, healthy interpersonal and social skills, and selection of a healthy social environment. |
| Poster 136: Refining and Disseminating HealthyU: Health Science Learning in Underserved Schools |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Diana Austria, Stanford University, daustria@stanford.edu |
| Tien-Wen Wiedmann, Stanford University, twtw1@stanford.edu |
| Sam Comstock, Stanford University, sam@comstock.net |
| Nancy Morioka-Douglas, Stanford University, nmd@stanford.edu |
| Abstract: With increasing populations of youth impacted by chronic health conditions, interventions are needed to help reverse current obesity trends by empowering young people to develop positive, healthy behaviors that help facilitate future academic, professional, and personal success. HealthyU was developed as a comprehensive, science-based health education web module (focusing on Nutrition, Exercise, Weight, and Sleep) that aims to fill an existing gap between available materials and their utilization in school settings. In partnership with three Redwood City School District middle schools, we evaluated and disseminated an improved model of the HealthyU curriculum among 200 seventh grade students. Results suggest that students showed moderate but promising improvement in overall, short-term knowledge. We are currently investigating ways to improve the retention of knowledge over a longer period of time. With continued refinement, dissemination, and evaluation, this model has the potential to aid schools in the transformative shift towards positive health behaviors. |
| Poster 137: Assessing State and Local Health Department Response to the H1N1 Influenza Pandemic - Integrating Methods, Findings and Values in Two CDC-led Evaluations |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Dale Rose, Centers for Disease Control and Prevention, ido8@cdc.gov |
| Stephanie Rubel, ICF Macro, srubel@icfi.com |
| Gillian Cross, ICF Macro, gcross@icfi.com |
| Miriam Cho, ICF Macro, mcho@icfi.com |
| John Zimmerman, ICF Macro, jzimmerman@icfi.com |
| Christopher La Rose, ICF Macro, clarose@icfi.com |
| Aba Essuon, Centers for Disease Control and Prevention, aee9@cdc.gov |
| Erica Bushong, Centers for Disease Control and Prevention, goj8@cdc.gov |
| Abstract: This presentation presents methods and key findings from two evaluation projects of H1N1 pandemic influenza emergency response at the state and local level. One project examines the Public Health Emergency Response program, administered by the Centers for Disease Control and Prevention (CDC), to fund and assist 62 state, local and territorial health departments to plan for and implement vaccination and additional response efforts. The project presents findings from analyses of funding investments, activities, and key outputs at the state and local level. The second project focuses on an assessment of H1N1 after-action reports submitted to CDC by all 62 health departments by examining aspects of AAR completeness and quality. Both projects identify and reflect upon key values informing the evaluations, including notions of "accountability" and "quality" - and suggest ways to improve future response-related evaluations to inform the emergency preparedness continuous improvement cycle. |
| Poster 138: Evaluating the Impact of Employment on the Well-Being of Displaced Workers |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ajanta Roy, Centerstone Research Institute, ajanta.roy@centerstone.org |
| Tricia Amonette, Centerstone Research Institute, tricia.amonette@centerstone.org |
| Ellen Pogue, Centerstone Research Institute, ellen.pogue@centerstone.org |
| Abstract: Due to the automotive restructuring and economic downturn in the U.S, many residents in southern counties of Tennessee have faced employment challenges. The economic downturn has left many people who had manufacturing jobs with limited options and in need of training to ensure future employment. To help these unemployed and displaced workers an intervention of education and job placement services in the Allied Health Field was implemented. This poster will present the evaluation findings of individual factors associated with re-employment as well as the level to which participant's well-being and mental health was impacted while participating in a job-retraining program incorporating an educational intervention, job placement assistance and support maintaining employment. |
| Poster 139: Evaluating Mining Safety and Health Training, Technology Transfer, and Communications |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Laurie Ruberg, Wheeling Jesuit University, lruberg@cet.edu |
| Debra C Piecka, Wheeling Jesuit University, dpiecka@cet.edu |
| Abstract: Six months after receiving a grant to improve mining health and safety, a tragic coal mining accident occurred that resulted in the death of 29 miners, working 1,000 feet underground, 45 miles from the West Virginia state capital. The mining tragedy helped to underscore and reinforce the need for greater health and safety regulations, training, and technology support that this project was designed to address. This evaluation adapts a logic model first used by the National Academies in their assessment of the National Institute for Occupational Safety and Health and Mining Programs (2005, 2007) to guide the project team towards strategies that maximize outputs and outcomes for training, technology transfer, and communications. This paper describes the evaluation framework, method, and data collection processes across five diverse project tasks, each led by technical experts. The findings highlight the role of social media tools to support data collection and team coordination processes. |
| Poster 140: Evaluation of Changes in the Quality of Child Care: Central Indiana Child Care Ministry Improvement Project |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Karen Ruprecht, Ruprecht and Hoke Consulting LLC, karenruprecht@me.com |
| Robert Hoke, Independent Consultant, robert@roberthoke.com |
| Abstract: The Central Indiana Child Care Ministry Improvement Project (CICCMIP) began in January 2010 as an eighteen month partnership between the United Way of Central Indiana, the Indiana Bureau of Child Care, and Child Care Answers. The project used a combination of ARRA and local foundation funding to provide mentoring, professional development, curriculum and facility improvement grants to increase the quality through participation in voluntary certification process. In Indiana, registered child care ministries may receive CCDF funds but are exempt from regulations other than health and safety. Through the engagement of two external evaluators, the quality of child care was evaluated through pre/post classroom observations using the Infant Toddler Environment Rating Scale-Revised (ITERS-R) or the Early Childhood Environment Rating Scale-Revised (ECERS-R) and the Arnett Caregiver Interaction Scale. A newly developed 'State of Change' scale that measures providers' readiness to change and adapt new practice was also completed. Post-assessment classroom observations using those same tools are underway at the time of submittal of this poster abstract. The final evaluation report will be completed by July 2011 and it is anticipated that this poster will include comparisons of the pre/post assessments, changes required in project implementation due to time restraints and unforeseen challenges, and suggested application of the methods to future child care evaluations. |
| Poster 142: Connecting the Dots: Using Internal Evaluation to Shape Decision Making, Demonstrate Impact, and Enhance Communication |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bernadette Sangalang, San Francisco AIDS Foundation, bsangalang@sfaf.org |
| Manali Nekkanti, San Francisco AIDS Foundation, mnekkanti@sfaf.org |
| Bob Rybicki, San Francisco AIDS Foundation, brybicki@sfaf.org |
| Abstract: Many nonprofit organizations recognize the value of evaluation, but often struggle with insufficient time, resources, and expertise to develop and implement an effective monitoring and evaluation system. In this poster, we will describe the development and implementation of an evaluation framework at the San Francisco AIDS Foundation, which aims to radically reduce new HIV infections in San Francisco. Developed internally through a collaborative process, the evaluation system offers practical approaches to measure and track program- and organization-level outcomes that contribute to the Foundation's ambitious goals. With an emphasis on utilization-focused evaluation, evaluation activities are integrated into program planning, design, and implementation in order to provide immediate and practical value to program staff in assessing program effectiveness. Evaluation tools were designed to demonstrate the important pathways linking individual program outcomes with agency goals, and highlight contributions to the overall goals of local funders and the National HIV/AIDS Strategy. |
| Poster 143: The Effect of Economic Conditions on Life Domain Outcomes for Consumers in Maryland's Public Mental Health System |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Timothy Santoni, University of Maryland, tsantoni@psych.umaryland.edu |
| Abstract: In an attempt to understand the changes in employment status among consumers participating in a mental health Outcomes Measurement System (OMS), analyses examining the relationships between statewide unemployment rates and employment rates among individuals in Maryland's Public Mental Health System (PMHS) were undertaken. Then, an examination of changes observed in annual outcome changes over time was undertaken. Given the effect of larger economic forces were demonstrated to have on mental health consumer employment outcomes, analyses were broadened to examine the effects of the State economy at large, represented by monthly unemployment and consumer price index changes, on various other outcome domains as measured in the Maryland PMHS Outcomes Measurement System. |
| Poster 144: Evaluating Local Policy Adoption Campaigns: A Case Study of Tobacco Retail License Adoption in California |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Travis Satterlund, UC Davis Center for Evaluation and Research, tdsatter@ucdavis.edu |
| Abstract: The California Tobacco Control Program (CTCP) has advocated for local tobacco control projects under their purview to target their local communities to adopt tobacco retail license (TRL) fees, as the adoption and implementation of TRLs is seen as a primary strategy to combat the problem of illegal sales of tobacco products to minors. Despite the research that overwhelming demonstrates the relationship between enacting and enforcing local tobacco youth access policies—such as TRLs—and their effect on the reduction of youth tobacco use (Chen & Forster, 2006), local communities still face challenges in adopting these policies. Thus, the objective of this paper is to explore and evaluate the processes used by 55 local projects in their campaigns to adopt TRL policies in communities across the State of California. Particular attention will be paid to how the process evaluations used by the local projects affected adoption of TRL policy. |
| Poster 145: An Individual-level Application of Empowerment Evaluation for Substance Abuse Treatment |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jonathan Scaccia, University of South Carolina, jonathan.p.scaccia@gmail.com |
| Bonnie Wright, NorthEast Treatment Centers, bwright@net-centers.org |
| Karen Mauro, NorthEast Treatment Centers, kmauro@net-centers.org |
| C Joseph Schultz, NorthEast Treatment Centers, jschultz@net-centers.org |
| Abstract: A Recovery-Oriented System of Care (ROSC) for substance abuse treatment is client-driven, focused on multiple holistic goals, research-based, outcome-driven, and requires ongoing monitoring for systems improvement (McLellan, 2010; Laudet and White, 2010). In this demonstration, we will describe a novel method by which joint process and outcome tracking systems were used to increase the capacity of the consumer receiving services, monitor the progress of consumer recovery, and inform continuous quality improvement (CQI) on several intervention levels: consumer, counselor, and organization. We will describe the selection criteria for recovery indicators and how alternate indicators might be chosen. These methods represent a step forward in individual level evaluation. We draw upon the values and tools of an Empowerment Evaluation framework (Fetterman and Wandersman, 2005) and the Getting to Outcomes (tm) model (Chinman, Imm, and Wandersman, 2004) to conceptualize client-driven treatment in a way that incorporates the evidence-base for quality outcomes. |
| Poster 146: Educating New Evaluators: Building Evaluation Capacity Through Local American Evaluation Association Affiliates |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Linda Schrader, Florida State University, lschrader@fsu.edu |
| Gwen Willems, Willems and Associates, wille002@umn.edu |
| Allison Titcomb, ALTA Consulting LLC, altaconsulting@cox.net |
| Abstract: Many professionals in the fields of education, health services, and public administration enter evaluation 'through the back door.' While highly skilled and knowledgeable in their own content area, these individuals are often called on to undertake or supervise evaluations of their own programs with little information about formal program evaluation. To meet this growing demand for evaluation expertise, local affiliates have responded by providing a range of professional development seminars and resources targeted to this audience. This session will present an overview of the outreach activities and services implemented by AEA local affiliates to support quality program evaluation at the local level. In addition, products developed by the Local Affiliate Collaborative (LAC) - an organizational mechanism to coordinate activities among affiliates and AEA - will be presented. A series of informational materials (program tip sheets, conference planning guides, strategic planning techniques) developed by the LAC and local affiliates will be available for distribution. |
| Poster 168: Rapid Response Team: Improving Outcomes Through Developmental Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Vicki Schug, St Catherine University, vlschug@stkate.edu |
| Susan Forneris, St Catherine University, sgforneris@stkate.edu |
| Suellen Campbell, St Catherine University, secampbell@stkate.edu |
| Abstract: The purposes of this session are to: 1) present a framework for developmental evaluation (Patton, 2010; Mintzberg, 2007) and 2) engage critical conversations surrounding curriculum innovation and the need for corresponding evaluation mechanisms that adapt to rapid and responsive change to emerging phenomena in health care delivery. |
| Poster 147: Youth Program Climate Survey: Quick & Easy, Valid & Reliable |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Walter Schumm, Kansas State University, wrschumm@ksu.edu |
| Benjamin Silliman, North Carolina State University, ben_silliman@ncsu.edu |
| Abstract: This poster describes the Youth Program Climate (YPC) survey on program quality, including discussion of validity and reliability and recommendations on survey administration and use of results. Developed from research on youth program quality, the survey serves as a formative measure of program qualities (e.g., safety, support, skill-building, self-efficacy promoting, social inclusion, social norms of responsibility and service, structure appropriate to the age and setting, synergy with family and community) and as a summative indicator of best practices, as perceived by program participants. Reliability with diverse pre-teen and teen audiences is typically high (Cronbach alphas from .85-.95). The poster will also describe procedures for administration of the survey in diverse modes, diverse forms, for demographically diverse audiences and diverse types of community-based programs, conferences, and camps. Use of the YPC for program improvement, accountability, and capacity-building and implications of this approach for other Extension programs will be discussed. |
| Poster 148: Evaluation of Item and Test Development Practices at a State Education Department: A Case Study of How Interpersonal Transactions Affect Item and Score Validation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Charles Secolsky, County College of Morris, csecolsky@ccm.edu |
| Abstract: One key component of accepted test development practice is its effect on test score validation or the interpretation of test results. If the items are not of high technical quality then inferences based on scores may not be as meaningful. One source of test development that has been given little attention are the relationships between those validating a test and the subject-matter experts (SMEs) who write and rate items. What effect, if any, do these relationships and the transactions between and among validators and SMEs have on item and test development practices? One state education department was selected as the case for the investigation. Transactions were documented. It is believed that different validators may make different demands on the same set of SMEs resulting in differences in which items are used from the item pool for the test. |
| Poster 149: Developing Comprehensive Community Initiatives through Participatory Action Research: Five Facilitators of Reflective Practice |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Daniela Seskar-Hencic, Region of Waterloo Public Health, dshencic@hotmail.com |
| Rich Janzen, Centre for Community Based Research, rich@communitybasedresearch.ca |
| Abstract: Evaluations of Comprehensive Community Initiatives provide a good opportunity not only to learn about the progress and outcomes of desired change, but also to gain insight into how research and evaluation contribute to, and drive the development of the intervention. A case example of one Comprehensive Community initiative, Waterloo Region Immigrant Employment Network, will be used to illustrate how participatory action research provided an opportunity to understand key developmental issues and contribute to the improvements of the intervention. The learning nature of the participatory action research will be described through five facilitators of reflective practice: a) be location-based, b) value experiential and practical knowledge, c) provide ongoing feedback, d) facilitate democratic dialogue, and e) focus on a vision for the common good. Each of the facilitators will be discussed in the context of specific situations from the case example, while drawing on the theory and practice of the participatory action research. |
| Poster 150: SMILER: How an Improved Approach to Project Monitoring can Support Evaluation, Learning and Action |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Guy Sharrock, Catholic Relief Services, gsharroc@crs.org |
| Hap Carr, Catholic Relief Services, hcarr@crs.org |
| Abstract: The poster session will illustrate how the process of 'Getting Organized' in setting up a M&E Operating Manual, developing a table of contents, a purpose statement and a stakeholder analysis; 'Setting Up' to review available documents that have information on the specific objectives, indicators and activities of the project; and 'Designing Forms and Reports' to develop data flow maps that document the types of forms and report formats required, and the flow and use of reports, including prompt sheets for 'learning to action discussions,' along with information on data management. The SMILER approach enables different actors to participate in discussions to document progress and determine future direction. Thus, project staff can, at last, understand their contribution to the project's monitoring operations upon which implementation decisions are taken and impact evaluations are based. CD-ROMs of the SMILER manual will be available for visitors to the stand. |
| Poster 151: A Cluster Evaluation of the Process and Impacts of the Joint Fire Science Program |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Loretta Singletary, University of Nevada Reno, singletaryl@unce.unr.edu |
| Lorie Sicafuse, University of Nevada, Reno, lsicafuse@unr.edu |
| William Evans, University of Nevada, Reno, wevans@unr.edu |
| Abstract: This presentation focuses on the development and implementation of a national fire science education cluster program evaluation. Grounded in the Logic model, this research employs a mixed-methods approach to measure the impacts of eight regional fire science network and delivery consortia. Broadly, this evaluation will assess consortia progress toward their shared goals of improving the dissemination and application of fire science research, increasing communication and collaboration between fire science researchers and practitioners, and coordinating fire science delivery efforts. Components of the current evaluation include a quantitative e-survey disseminated to fire scientists, practitioners, and the general public, collection of user data from consortia websites, qualitative interviews with consortia investigators, and the development and distribution of an evaluation guide designed to assist consortia investigators in maximizing the success of their outreach efforts. Preliminary results will be presented, along with a discussion of their implications for future cluster evaluation research and practice. |
| Poster 152: The Colorado Veteran Trauma Court: Challenges in Evaluating a Jail Diversion and Trauma Recovery, Priority to Veterans Grant |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michelle Slattery, University of Colorado, Colorado Springs, peakresearch@usa.net |
| Mallory Dugger, University of Colorado, Colorado Springs, mdugger@uccs.edu |
| Laura Williams, Colorado Department of Human Services, laura.williams@state.co.us |
| Abstract: This session explores the unique evaluation challenges that have occurred in the creation of a Veteran Trauma Court in Colorado Springs, ground zero for soldiers returning from OIF/OEF conflicts with unprecedented levels of PTSD and TBI. The impact on the community has been reflected in the headlines as more and more soldiers and veterans have entered the Criminal Justice System. Over the past 2˝ years, the community has rallied to create a functional Veteran Trauma Court, funded by the Substance Abuse and Mental Health Services Administration (SAMHSA), Jail Diversion and Trauma Recovery - Priority to Veterans grant. Process evaluation challenges are explored from community, criminal justice, and individual levels. Working with sick soldiers within traumatized systems is one of the biggest challenges we have faced. Preliminary outcomes and lessons learned from 3 years on the front lines are discussed. |
| Poster 153: Disaster Accountability Project's Report on Transparency of Haiti Relief: Evaluating Disaster Relief When Information is Hard to Get or Not Available |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ben Smilowitz, Disaster Accountability Project, ben@disasteraccountability.org |
| Abstract: This report analyzes surveys sent to 196 organizations soliciting for Haiti relief. While 38 completed the survey by the deadline, this figure is nearly double the six-month report where nearly 80% did not respond. Collectively, responding organizations reported receiving $1.4 billion and spending $730 million (52%). $1.8 million in interest was raised by five respondents. Ten reported they did not know how much interest they raised. The rest did not respond, indicating this figure is higher. Eleven reported that interest raised is spent directly on Haiti relief, while four stated interest is used for general operating. Organizations were asked questions about the transparency of their activities and three reported they did not provide "publicly available situation/activity reports detailing their specific activities on the ground" and thirty-five indicated they did. Disaster Accountability Project's assessment of groups' websites found otherwise. Only 8 of 195 reviewed were rated 'somewhat' or 'more transparent.' |
| Poster 155: Evaluating an Online Financial Education Program: A Description of the Love Your Money Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tiffany Smith, University of Tennessee, Knoxville, tsmith92@utk.edu |
| Jennifer Morrow, University of Tennessee, Knoxville, jamorrow@utk.edu |
| Gary Skolits, University of Tennessee, Knoxville, gskolits@utk.edu |
| Abstract: The proposed poster investigates the process evaluating an online financial education program that included participants from five different universities. The purpose of this evaluation was to examine whether there are any long-term changes among those who participated in the program. The poster will explore the challenges presented when working with universities through long distance communication, including the limited ability to communicate and work with stakeholders and difficulty in promoting evaluation use through internet collaboration. Along with these challenges, there are many benefits to working with an online education program, including ease of access to those who have participated in the program and the wealth of information that can be received from participants. Even though working on online evaluations can prove to be challenging there are also many benefits that need exploration. |
| Poster 154: An Exploration of Fundamental Valuing: Ethics as a Pivotal and Ineludible Criterion in Evaluation Practice |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tiffany Smith, University of Tennessee, tsmith92@utk.edu |
| Gary Skolits, University of Tennessee, Knoxville, gskolits@utk.edu |
| Abstract: The proposed paper/poster addresses the basis for ethical obligations within evaluation. According to Sheinfeld & Lord (1981), evaluators are most concerned about political conflicts of interest and personal conflicts. However, there are many ethical considerations that can be addressed, ranging from Aristotle's virtue ethics to specific analytic philosophical debates on emergent ethical issues. The various conceptualizations of ethics are often debated in the field of philosophy from a technical perspective that is not readily accessible to evaluation practitioners. An ethical framework is provided to evaluators in the form of the American Evaluation Association's principles of professional practice. While these general principles are situated in the world of practice, they are necessarily vague and limited to fairly specific situations. In essence, this paper explores two questions: When evaluators are encouraged to adhere to ethical obligations, what should they refer to? And, what is our basis for ethical values and valuing? |
| Poster 156: Evaluation Study on Attitude Towards Empowerment of Women with Respect to Different Types of Educational Institution Among Indian Student Teachers |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shameem Subhan Sahib, Teacher Education Research and Training, shameemprof@gmail.com |
| Abstract: Attitude is enduring response dispositions with affective, behavioral and cognitive components. In India, there has been shift in policy approaches on attitude towards empowerment of women from the concept of 'welfare' in the seventies to 'development' in the eighties to 'empowerment' in the nineties and now 'women are equal partners like men'. Empowerment has multiple, interrelated and interdependent dimensions economic, social, cultural and political. This evaluation study is designed to assess and analyse the attitude towards empowerment of women among 1758 student teachers in India based on their type of educational institution, which was evaluated under the following five dimensions. Women Education Economic freedom of women Gender Role Perception Contribution of Women in the Household and Contribution of Women in the Community. It is recommended that for optimal utilization of human resources, gender equality and equity, women should be adequately represented, is the prime motivator of the country's economy. |
| Poster 157: How Does The Student's Active Participation in Development and Research Projects Influence Their Job Selection After Graduation? |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jirina Svitakova, Czech University, jirina.svitakova@gmail.com |
| Magdalena Vyborna, Czech University, vyborna@its.czu.cz |
| Milan Slavik, Czech University, slavikm@ivp.czu.cz |
| Abstract: The research question is whether there is and how extensive is the degree of influence of student's active involvement into practice during studies on following job specialization of graduates in the same labor market's specialization as was their previous study program. Between years 2000 - 2010 Institute of Tropics and Subtropics (ITS) a faculty of Czech University of Life Sciences in Prague (CULS) carried out 25 development projects and other research activities mainly according to the Official Development Assistance Concept of the Czech Republic. The target group of this research is 231 students graduated between the academic years 2004/2005 to 2009/2010. Research of this paper answers the question whether it would be beneficial to involve all students into realization of development and research projects obligatorily with the main goal to establish more graduates as managerial-engineering experts on development problematic and reduce the rate of exit from their study field. |
| Poster 158: Guidelines for Independent Consultants/New Evaluators: How to Work With a Potential Client to Write an Evaluation Plan |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Mary Anne Sydlik, Western Michigan University, mary.sydlik@wmich.edu |
| Abstract: The proposed presentation will provide guidelines for an independent consultant who needs to write an evaluation plan to be included in a potential client's grant proposal. First, the presenter will go over a check list of questions to ask the client about their project's goals and objectives, the project's scope, expected outcomes, time line, and budgetary considerations. Next, examples of typical evaluation questions will be presented, along with hints for how to choose measures/benchmarks of accomplishment and data collection techniques for each. The demonstration will end with suggestions for how to develop generic evaluation plan templates that can be tailored to each new project's specifications. The demonstration will end with examples of the difficulties that can arise in the process of developing and getting a high quality evaluation plan to the project PI in a timely fashion. |
| Poster 159: Telling Our Story with Data: The Value of Using Data in Native American Communities |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Nadine Tafoya, Nadine Tafoya and Associates, nayanet2426@gmail.com |
| Ann Del Vecchio, Alpha Assessment Associates LLC, ann.delvecchio@gmail.com |
| Abstract: A group of service providers and Tribal leaders came together to identify a process for assisting Tribes with using data. Part of the process addressed the issue of ethical concerns related to hundreds of years of research and evaluation conducted with the New Mexico's Native Americans, sometimes without their knowledge or consent. Controlling data, respect for Tribal protocols related to data and using data were topics of concern among the leadership of the 22 Tribes in New Mexico. The Tribal Data Work Group met monthly for two years and planned and wrote two separate guidebooks: Telling Our Story with Data and Whose Data is It?. The first guide was reviewed by Tribal leadership and service providers and feedback indicated the guide needed to be split up. The second guide was developed for government agencies and non-Natives to help them understand data needs and protocols for accessing and using Tribal data. |
| Poster 160: Evaluation Capacity Building at the National Institute of Allergy and Infectious Diseases (NIAID): Continuing Efforts to Build a Culture of Evaluation and Planning |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brandie Taylor, National Institutes of Health, taylorbr@mail.nih.gov |
| Guy Arcuri, National Institutes of Health, garcuri@mail.nih.gov |
| Cheryl Silver, National Institutes of Health, csilver@mail.nih.gov |
| Kevin Wright, National Institutes of Health, wrightk@mail.nih.gov |
| Jane Lockmuller, National Institutes of Health, lockj@mail.nih.gov |
| Abstract: NIAID's Strategic Planning and Evaluation Branch (SPEB) is working towards building Institutional evaluation capacity through numerous ongoing educational and participatory activities. SPEB staff are pursuing a multipronged approach to engage and inform Institute staff on evaluation methodologies and practices. Various forums are being used for outreach and education including: a special interest group; Planning and Evaluation (P&E) meetings; working groups and training sessions. Furthermore, SPEB has instituted a 'beat system' in which NIAID Divisions work with SPEB evaluation and planning specialists synergistically to implement and foster evaluation activities at a more individualized level. These evaluation capacity building activities are part of NIAID's effort to build and maintain a culture of accountability and transparency within NIAID, with the ultimate goal of having evaluation established as a critical tool in decision making and strategic planning within the Institute. SPEB's evaluation capacity efforts, along with their successes, challenges, and next steps will all be illustrated. |
| Poster 161: Performance Management as a Tool for Political Prioritization in Local Government |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Mads Teisen, City of Copenhagen Technical and Environmental Administration, mads.teisen@gmail.com |
| Abstract: Performance Management, although the schemes varies, mostly tend to lead to discussions related to their own effect, their ability the create knowledge and legitimacy and their ability to document effect. So far, there has been little or no discussion about how performance management influences political life in local government. However especially in local government, there are dilemmas. Given that a well implemented performance management system creates clear performance indicators at a regular basis, it should serve as the ideal basis for political prioritization. But at the same time it in reality takes away power from politicians, by making politics technocratic and evidence based, and leaving little room for personal judgement. And the management fear, that politicians could be interfering with the daily running of the operation. The dilemma is pointing out the need for a discussion on how involved the political level should become with performance management? |
| Poster 162: Strengths and weaknesses of developing a logic model via web conferencing |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Mao Thao, University of Minnesota, thao0181@umn.edu |
| Eric Moore, Rainbow Research, moor0210@umn.edu |
| Frances Lawrenz, University of Minnesota, lawrenz@umn.edu |
| Abstract: This paper summarizes our experience with using Adobe Connect, a web conferencing software, to develop a logic model in a multi-site evaluation. The process included four meetings over a period of one month, with each meeting lasting approximately two hours. There were different interaction styles with some people meeting together in one room and connecting on to the web conference via a large computer screen and centered microphone, and others logging on individually at their own computer. A shared screen with documents could be seen by all and the webcam video option was not used. Discussion took place primarily through the audio and written chat options; however, at one meeting the use of telephone conferencing was used to complement the web conference due to issues with the audio. Strengths and weaknesses of this process are highlighted, as well as lessons learned to provide tips for logic via web conferencing. |
| Poster 163: Using Evaluation to Scale-Up Practitioner-Driven Programming: The Story of HERstory |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amanda C Thompkins, The Leadership Program, athompkins@tlpnyc.com |
| Lisa M Chauveron, The Leadership Program, lisa@tlpnyc.com |
| Abstract: The Leadership Program's HERstory is a year-long afterschool program for adolescent girls that uses teambuilding activities and ethnographic theater to promote healthy identity exploration and positive youth development. A practice-to-research initiative currently in its sixth year of implementation, HERstory emerged out of one practitioner's experience at a middle school in New York City where she 'saw the deep need for the predominantly female participants to not just learn to perform but to perform their own stories.' The Leadership Program is currently going through the process of preparing to scale-up HERstory for replication in other cities. As part of that process, the internal evaluation team is working with the program developer to finalize the HERstory logic model and curriculum and move from qualitative to comprehensive mixed methods evaluation. In addition to highlighting key evaluation findings, this poster presentation will discuss the challenges and opportunities of transitioning toward comprehensive evaluation. |
| Poster 164: Stakeholder Utility: Perspectives on School-wide Data for Measurement, Feedback, and Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Gita Upreti, University of Texas at El Paso, gitaupreti@gmail.com |
| Holly Lewandowski, Evaluation for Change Inc, hlewandowski@yahoo.com |
| Carl Liaupsin, University of Arizona, liaupsin@email.arizona.edu |
| Dan Koonce, Chicago School of Professional Psychology, dkoonce@thechicagoschool.edu |
| Abstract: More than 14,300 schools in the United States have adopted the multi-tiered model of behavioral and academic support known as school-wide positive behavior interventions and supports (PBIS; www.pbis.org). Schools and districts adopting, implementing, and sustaining PBIS are charged with collecting and disseminating data generated by and related to students, parents, teachers, and administrators. Additionally, evaluators and technical assistance providers collect school- and district-level measures to measure outcomes related to PBIS implementation. The interests and needs of this broad range of stakeholders impact the usefulness of each piece of data that is collected for each stakeholder group and highlight the uniqueness of the PBIS model. This poster will demonstrate how the concept of stakeholder utility, driven by stakeholder role and purpose, may help stakeholders design and appraise measures to be used for assessment, evaluation, and research. |
| Poster 112: Women-Focused Rapid Response Project to Fight Avian Influenza |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Silvia Valentini, CARE International, svalentini@hotmail.com |
| Hadia Nusrat, CARE International, hnusrat.z@gmail.com |
| Tahira Siddiqi, CARE International, tsiddi@gmail.com |
| Muhammad Arshad, CARE International, |
| Zahid Mahmood, CARE International Pakistan, zmahmood@carepk.org |
| Abstract: More than 70% of Pakistan's poultry is produced in Mansehra and Abbottabad Districts; a significant proportion comes from backyard poultry farms. The avian influenza (AI) surveillance and response system formed by the Government of Pakistan have several gaps with regard to responding to pandemic threats. Existing surveillance covers commercial poultry farms but ignores backyard poultry farms. With community vulnerability in mind, this CDC-supported project targeted 45 villages in 4 Union Councils of Mansehra District. Over the course of the two-year project, a community-based system was established to enhance existing surveillance and response mechanisms. A holistic, district-level model that engaged communities, was found to be necessary for addressing AI risk while mobilizing, organizing and empowering stakeholders to respond to pandemic threats. Men and women community volunteers were trained on signs, symptoms, transmission and prevention of AI. Communities were linked with government actors at district and provincial levels. |
| Poster 165: Evaluation of the Eastside Aboriginal Space for Youth (EASY): Valuing Relationships |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kim van der Woerd, Reciprocal Consulting, kvanderwoerd@gmail.com |
| Abstract: Aboriginal youth continue to face special challenges during the transition to adulthood, including geographic isolation, poor economic conditions, living in single parent families, and abuse. Furthermore, Aboriginal and other minority youth have been found to be more likely to engage in risky behaviours, such as substance use and delinquency. The EASY program was developed to address urban Aboriginal youth gangs in Vancouver through a reduction in a number of risk factors. The EASY consists of a Late Night Resource programming including outreach, adult and peer mentoring along with Aboriginal cultural workshops. This presentation reviews the evaluation of the EASY gang prevention program. Preliminary findings suggest the importance of investing in prevention and health promotion for youth at risk of engaging in gang behaviours or lifestyles. Furthermore, it was found that relationships and positive mentoring to be paramount for both youth, and program staff in building capacity and reducing risky behaviours. |
| Poster 166: Culturally Responsive Evaluation: Collecting and Analyzing Data in English and Spanish |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Hever Velazquez, Oregon Museum of Science and Industry, hvelazquez@omsi.edu |
| Abstract: The Evaluation division at the Oregon Museum of Science and Industry implemented culturally responsive evaluation practices (Hopson, 2003, among others) that were critical in understanding and engaging the Spanish-preferring and bilingual Hispanic families in the Portland metro area. These practices, along with OMSI guidelines on collecting and analyzing data in two languages, have been useful in determining project evaluation staff members as well as developing appropriate research protocols. This poster shows how culturally responsive considerations were applied for both the National Science Foundation-funded "Sustainability" and National Institutes of Health Science Education Partnership Award-funded "Zoo in You" projects to address Hispanic research audiences. References Hopson, R. (2003). Overview of multicultural and culturally competent program evaluation: Issues, challenges and opportunities. California Endowment, Diversity in Health Evaluation Project. Woodland Hills, CA: California Endowment. |
| Poster 167: Developing and implementing a global Programme Information Management System for a large International Non-government Organization |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bernard Vicary, World Vision International, bernard_vicary@wvi.org |
| Abstract: World Vision is implementing a global programme management information system, designed around the organization's design, monitoring and evaluation framework, for Learning through Evaluation with Accountability and Planning (LEAP). The system is being developed through four releases for managing documents, managing data, impact reporting and informing strategy. The system manages data across the global enterprise and also provides a set of project and programme management tools. User-centric design processes focus on business process maturity and promote pro-active data governance and accountability for data quality. The poster will describe the system, the development process, development of training and transformational resources and the system's roll out across the globe. Lessons learned will be discussed in relation to the history of the programme, moving through a summary of current implementation to a presentation of the system's future roadmap. PMIS is making a significant contribution to World Vision's approaches to programme design, monitoring and evaluation. |
| Poster 169: Reflecting on Students' Self-Assessments of Knowledge Gained through a Program Evaluation Graduate Seminar |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Wendy Viola, Portland State University, violaw@pdx.edu |
| Lindsey Patterson, Portland State University, lbp@pdx.edu |
| Mary Gray, Portland State University, grayme@pdx.edu |
| Ashley Boal, Portland State University, aboal@pdx.edu |
| Katherine McDonald, Portland State University, kmcdona@pdx.edu |
| Abstract: This poster presentation will address the process of learning program evaluation through a formalized course, from the perspectives of four students studying evaluation within an Applied Psychology doctoral program. Results of students' Program Evaluation Self-Assessment Instruments (Stufflebeam, 2001) from before and after their participation in the 10-week course will be presented to demonstrate how their knowledge of, and confidence in, their skills as evaluators changed after taking the course. These findings are intended to act as a springboard for eliciting feedback from professional evaluators regarding the skills and proficiencies that they consider the most crucial for students of program evaluation to learn in a formal classroom context. |
| Poster 170: Development of Mechanism of Competitive Distribution of Budgetary Funds Through Introduction of Evaluation Mechanism |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Liudmila Vladyko, Government of the Krasnoyarsk region, milavladi@mail.ru |
| Abstract: 1.About government program of grants of Krasnoyarsk region 2.The basic evaluation processes of the Program 3.Basis criterias of evaluation of government program of grants of Krasnoyarsk region 4. The basic resources for introduction of all cycle of the evaluation government program of grants of Krasnoyarsk region 5.Structure of the report of impact evaluation of government program of grants of Krasnoyarsk region 6.The basic results and conclusions of the carried out evaluation |
| Poster 171: Approaches to Evaluating Community College English as a Second Language (ESL) Programs |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Dawn Wallace-Pascoe, Ohio State University, wallace-pascoe.103@osu.edu |
| Abstract: Community college English as a Second Language (ESL) programs serve a large and growing number of adult immigrant students in the U.S. These students, with their varied educational and linguistic backgrounds, seek goals that range anywhere from learning enough English to function day-to-day, to improving their employment prospects, or to earning a college degree. In response, community colleges offer different types of ESL programs, including for-credit, non-credit, and vocational ESL. Considering the significance of ESL programs to the immigrant community and to community colleges, the need exists to evaluate these programs to ensure that they are effective in achieving their goals. However, few evaluations of community college ESL programs have been published to date. This poster will highlight the type of evaluation work that is currently being done on community college ESL programs, with a particular emphasis on non-credit ESL. Some possible evaluation design ideas will also be proposed. |
| Poster 172: Navigating Diverse Values in Re-Entry-Based Mental Health Program Evaluation |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Maryann Waugh, Aurora Research Institute, maryannwaugh@aumhc.org |
| Richard Swanson, Aurora Research Institute, richardswanson@aumhc.org |
| Abstract: Policy makers are increasingly focusing on the 'what works' literature to ensure that only effective programming is funded. Mentally ill offenders are particularly costly to the criminal justice system and evaluation is crucial to determining program efficacy for this sub-population. Evaluating a program for efficacy may seem straightforward. However, in the applied world of criminal justice and re-entry programming, a variety of conflicting values affects each evaluation decision. This poster describes how the Aurora Research Institute is navigating evaluation decisions within a context of diverse values, held by local agencies collaboratively providing re-entry services to mentally ill offenders. Evaluation issues, while being raised in the context of re-entry, are common to evaluators across a variety of diverse situations. It is the hope of the authors that the poster will spark conversation regarding methodology choices that exist within such complex situations, and contribute to an increasing standard of applied evaluation quality. |
| Poster 173: Evaluating the Impact of a Nonformal 'Mommy and Me' Marine Science Program |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jim Wharton, University of South Florida, jimwharton@mote.org |
| Miranda Wrobel, Mote Marine Laboratory, miranda@mote.org |
| Kerri Wilkinson, Mote Marine Laboratory, |
| Abstract: Evaluation in nonformal science education programs has historically taken a back seat to development and implementation. This is changing. This year Mote Marine Laboratory has initiated a collaborative, internal evaluation of 'Mommy and Me at Mote'—a nonformal, family education program for 2-5 year-olds and their favorite caregiver. This formative evaluation pilot study is being conducted to determine if selected program outcomes are being achieved, and to identify opportunities for improvement. The evaluation team chose three areas of focus: 1) the impact of Mommy and Me at Mote on child and parent attitudes towards the ocean, 2) its impact on school-readiness, and 3) the program's financial model. The study takes a multi-method approach to address program impact, engaging parent surveys, focus groups, and parent/child interviews. In this poster, I will present the study's impact measures and reflect on lessons learned in collecting reliable data from, and about, a pre-school audience. |
| Poster 174: Caseworker Values as a Context for Understanding Family Outcomes in Child Welfare Populations Randomized to Investigation or Differential Response |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Cailin Wheeler, Human Services Research Institute, cwheeler@hsri.org |
| Abstract: Using a randomized control trial, HSRI is in the process of evaluating Differential Response (DR) in six Ohio counties as an alternative approach to the traditional investigation for low risk families with a report of abuse or neglect. Under DR, caseworkers conduct an assessment of family needs and provide appropriate services and supports without any formal allegation or determination of maltreatment. Nonetheless, variations in individual caseworker values also need to be taken into consideration when examining differences outcomes. The 'Work Practice and Values Scales for Child Protection' (Dalgleish, 2000), was completed by caseworkers in order to understand the cultural climate of the caseworker/family dyad at study start. Using path analyses an ecological perspective was employed to explicitly take these values into account while assessing self-perceived differences within and between intervention and control group families in family satisfaction, caseworker/family relationships and family experience. Results of analyses are presented. |
| Poster 175: Human Resource Policies Inform LGBT Client-Based Services |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Wendy Wolfersteig, Arizona State University, wendy.wolfersteig@asu.edu |
| Aimee Sitzler, Arizona State University, aimee.sitzler@asu.edu |
| Travis Johnson, Arizona State University, travis.a.johnson@asu.edu |
| DesiRae Deschine, Arizona State University, desirae.deschine@asu.edu |
| Abstract: Current research shows Lesbian, Gay, Bisexual and Transgender (LGBT) individuals experience health disparities and barriers to health services associated with gender identity and sexual orientation not experienced by their heterosexual and/or gender normative counterparts. Research Question: Do agencies with Human Resource (HR) policies/practices that support LGBT employees also document and identify LGBT individuals who seek and/or receive their services? Results: The evaluation findings showed that 55% of agencies with written LGBT-inclusive HR policies and procedures, and 50% of agencies who support the visibility of LGBT employees do not identify and document LGBT individuals seeking or receiving their services. Implications for providers: Behavioral health organization need to have in place policies and procedures sensitive to their employees and clients needs and diversity if they wish to offer LGBT culturally appropriate services and a welcoming workplace climate. |
| Poster 176: Formulating Questions as a Reporting Strategy to Enhance Evaluation Utilization |
| Poster Presentation 144 to be held in California A on Wednesday, Nov 2, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Judy Woods, Queen's University, judy.woods@queensu.ca |
| Abstract: Transforming evaluation findings from statements into questions generates reflection, discussion and further action by program decision-makers. Presenting evaluation findings in the form of questions enhances the utilization of an evaluation. This strategy was employed in an evaluation of an integrated clinical clerkship program. The primary purpose of this evaluation was to inform improvement in the delivery and experience of medical education in a community setting. Drawing on a developmental evaluation framework, this work focused on creating a program description, program logic model, and stakeholder map. For this purpose interviews were conducted with stakeholders representing program developers, administrators and implementers. Responses to seven semi-structured questions described the program in action. Evaluation findings were reported in the form of questions. Program personnel responses to these questions determine the next steps in program decision-making. Reporting in the form of questions is a strategy with real potential for enhancing evaluation product utilization. |