| Does Community Participation Increase Agency Understanding of Community Change? |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Asma Ali, University of Illinois Chicago, asmamali@yahoo.com |
| Abstract: Many agencies struggle with understanding the effects of community change on their programs and clientele (Stoeker 1997, Vidal 2002). In addition, social service organizations within neighborhoods experiencing comprehensive community change may be perplexed by the changes in institutions, community population, or the utilization of their programs (Staeger 2001). Current thinking on the evaluation of comprehensive community change encourages social service organizations to include community members in the development of program and evaluation plans-- leading to more responsive organization practices. However, there is limited literature on whether including community members in such efforts increase the agency’s ability to navigate community changes (Aspen Institute 2001). This poster proposal outlines a project with a Center for Independent Living, a disability- related social service agency, near Chicago. The minority outreach efforts of the agency were primarily planned, implemented, and guided by participatory working groups of community members, staff, and clients in order to better understand the African American and Latino consumers in agency’s catchment area. The poster will discuss development of an outreach and outreach evaluation plan through this participatory process, staff’s developing understanding of the minority community through their engagement in this process, and the outcomes of each effort in the process. |
| Use of Two Focus Group Interview Formats to Evaluate Public Health Grand Rounds: Methods and Results |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bayram Yilmaz, The Ohio State University, yilmaz.12@osu.edu |
| James Altschuld, The Ohio State University, altschuld.1@osu.edu |
| Abstract: While face-to-face focus group interviews (FGI) have been widely used for program evaluation, online FGIs have recently become popular among evaluators. In an evaluation of Public Health Grand Rounds (PHGR), a series of seminars on current public health issues, we used both techniques and conducted three FGI sessions (one face-to-face, two cyber). They included individuals from stakeholding groups such as public health students, local public health practitioners, and Ohio Center for Public Health Preparedness (O-CPHP) staff. The focus was on how well the program worked, what should be changed, what key public health topics could be emphasized in subsequent grand rounds, reasons for attending PHGR presentations, barriers to attending, reaching public health audiences other than students, and similar concerns. We compared the findings of the two data collection procedures, and made suggestions regarding the effective use of them and for changes in PHGR. Although the cyber FGI produced about same amount of data, took less time, needed less arrangements and a smaller budget than that of face-to-face, it has its own difficulties in that it is not easy to probe for responses, slow typing skills may prevent some from fully joining the process, and the possibility of response bias. |
| Students as Evaluators: Encouraging Them to Use Their (Photo) Voice |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anna Robic, University of South Florida, robic@tempest.coedu.usf.edu |
| Patricia Alvarez McHatton, University of South Florida, mchatton@tempest.coedu.usf.edu |
| Stephanie Martinez, Florida Mental Health Institute, smartinez@fmhi.usf.edu |
| Abstract: A trend in evaluation has been to utilize students as evaluators of programs. What happens when the students who have participated offer little feedback? How can you access what they might not be telling you? This session presents information on this dilemma as well as suggestions on how to overcome the problem. |
| The Unexpected Benefits of a Consumer Survey Team: Factors Relating to Motivation for Employment |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Derek Anderson, University of Colorado Denver, derek.anderson@cudenver.edu |
| Kathryn DeRoche, Mental Health Center of Denver, kathryn.deroche@mhcd.org |
| Abstract: At the Mental Health Center of Denver, we have embraced the use of a “consumer survey team” to administer surveys to other consumers (peer-to-peer data collection), and have noticed an unexpected benefit that members commonly find employment after involvement with the team. The primary goal of the evaluation was to examine attitudes towards employment of consumer survey team members, and to determine if involvement with the team is a model of employment training for individuals struggling with mental health problems. Using a mixed-methods approach, we examined the experiences of team members through semi-structured interviews, in addition to quantitative survey data regarding employment and mental health recovery. The evaluation findings will be discussed focusing on the empowering benefit of involvement, the process of motivation for employment, and characteristics which influence employment motivation. |
| Building Evaluation Capacity in Regional Centers Using Formal Evaluation Courses |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Janet Clinton, University of Auckland, j.clinton@auckland.ac.nz |
| Sarah Appleton, University of Auckland, sk.appleton@auckland.ac.nz |
| Abstract: This paper describes the implementation of a suite of formal public health education programs within a large scale long term evaluation with the aim of building evaluation capacity and organizational learning in regional centre within New Zealand. Evaluation capacity and willingness to engage in evaluation were measured and the educational programs were tailored to meet the communities need. The immediate impact of the first of the formal evaluation courses, the technical assistance and the short course are explored. This paper contents that combining evaluation of programs with building capacity through formal training is a useful procedure for rural or distant communities where access to services is limited. A number of issues were identified throughout this process. In addition, embedding evaluation knowledge into the practical environment the ethical considerations of the evaluator teaching formal qualifications as a part of a long term evaluation contract are discussed. |
| Practical Considerations for Implementing Strong Evaluation Designs in School-Based Prevention Programs |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Darcy Richardson, LeCroy and Milligan Associates Inc, darcy@lecroymilligan.com |
| Joanne Basta, LeCroy and Milligan Associates Inc, jbasta@lecroymilligan.com |
| DeeAnn Arroyo, Pima Prevention Partnership, darroyo@thepartnership.us |
| Geoff Wood, LeCroy and Milligan Associates Inc, geoff@lecroymilligan.com |
| Abstract: School-based pregnancy prevention programs face numerous challenges with implementation and sustainability. A rigorous evaluation can be advantageous for addressing these challenges because it can provide periodic information on implementation processes and progress. It can also provide reliable and valid outcome data on which program improvement and best-practices can be based. However, an effective evaluation is difficult to accomplish in a school-setting. The strongest evaluation design may be impractical based on the complexities of the program being offered, financial considerations, and/or the responsiveness of the community. The challenge with these programs is to design a rigorous evaluation that is practical in a school-setting. This session will include discussion of the design and implementation of two school-based pregnancy prevention programs, including challenges and lessons learned. The importance of the partnership between program and evaluation will be emphasized throughout this interactive session. |
| Telling the Story of Indigenous Evaluation and Mapping the Practice on Study in Progress |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kas Aruskevich, University of Hawaii Manoa, kasa@hawaii.edu |
| Abstract: This poster session, Telling the Story of Indigenous Evaluation and Mapping the Practice is the title of my dissertation study in progress to explore the theories of evaluation for Indigenous-based projects. By the time of the AEA Conference in November, I will have conducted in-depth interviews with Key Informant evaluators from the Canadian, New Zealand; and Australian national evaluation associations and will describe initial emerging themes. Phase II of the study, mapping practice, will occur after all interviews have taken place and utilize an instrument developed through Key-Informants’ contribution of ten statements that guide their evaluation practice. The survey will be administered online to all Key-Informants and evaluation practioners of national evaluation associations that have conducted at least one Indigenous-based project. The purpose is to see how emerging theory maps and if it is influencing practice. The poster session will also place a call for evaluators of at least five Indigenous-based projects to act as Key-Informants for the United States. |
| Determining the Longitudinal Impacts of Summer Research Experiences on Underrepresented Minority Undergraduates Intent to Pursue Graduate Degrees |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Karyl Askew, University of North Carolina Chapel Hill, karyls@email.unc.edu |
| Roy Charles, University of North Carolina Chapel Hill, rac@email.unc.edu |
| Abstract: The poster presents a method for evaluating the longitudinal impacts of undergraduate summer research programs (USRP). For this evaluation the USRP host institution was one of three members of the North Carolina Alliance to Create Opportunity Through Education (NC OPT-ED), a National Science Foundation’s funded Alliance for Graduate Education and the Professoriate (AGEP) partnership. This is part of a multi-year evaluation to determine the extent to which programming contributes to the AGEP mission of increasing the number of underrepresented minority students receiving science, technology, engineering and mathematics (STEM) doctorates and entering the professoriate. To examine longitudinal student impacts, evaluators used previously submitted student letters of recommendations and employed purposive sampling to identify 11 faculty advisors from 7 institutions. The original sample included 122 program participant recommendation letters from three summer cohorts. Evaluators conducted semi-structured interviews with faculty advisors. Presenter will share successes and lessons learned from implementing this technique. |
| Government Program Evaluation in Central Asian Countries |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jamila Assanova, Civil Society Development Association (ARGO), jamila_asanova@yahoo.com |
| Abstract: This poster is presenting situation with monitoring and evaluation of government programs in Central Asia countries. Central Asia has great experience in conducting program evaluation of international programs. However, here government program evaluation existing on a formal level mostly. This caused by 1) social – economic and political changes in Central Asian countries that are in transition; 2) recent development and administration of government programs oriented to the results (for last 5-7 years);3) lack of practical skills and knowledge of government programs implementers in monitoring and evaluation even though the programs its selves include that process. Nevertheless, Central Asian countries realizes necessity in evaluation by forming national strategies, economical, social, national and local result oriented programs, state budget formations. In some of these countries there are first steps made in development of evaluation policy and legal documentation with norms that formalizes evaluation policy for government programs and where evaluation is conducted as a pilot projects. This paper includes the data that presented first time outside of Central Asia. Discussion of that data would expand opportunities for studies of international experience, receiving feedback and implementing established standards in formation of evaluation policy for government programs. |
| The Good, The Bad, and The Ugly: Outsourcing Experiences |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristin Pankey, Southern Illinois University at Carbondale, knpankey@siu.edu |
| Joel Nadler, Southern Illinois University at Carbondale, jnadler@siu.edu |
| Nicole Cundiff, Southern Illinois University at Carbondale, karim@siu.edu |
| Nicholas G Hoffman, Southern Illinois University at Carbondale, nghoff@siu.edu |
| Gargi Bhattacharya, Southern Illinois University at Carbondale, gargi@siu.edu |
| Alen Avdic, Southern Illinois University at Carbondale, alen@siu.edu |
| Abstract: The Applied Psychology program at the Southern Illinois University at Carbondale (SIUC) has been providing training in program evaluation since 1982. Perhaps the most unique part of this training is the opportunity to conduct professional evaluations as a part of Applied Research Consultants (ARC), a student-run consulting firm. Over the past few years, ARC has been hired for an increasing amount of outsourcing services and has had a variety of experiences with each of these projects and evaluations. Three specific situations stand out and can be described as “The Good, The Bad, and The Ugly.” A good experience is defined as a project that allows for the development of skills and abilities. “Bad” and “Ugly” experiences are profitable but with low educational value. This presentation will discuss these three situations, ARC’s experiences and relationships with these clients, and tips for dealing with outsourcing. |
| Evaluation of Program Intended for Low-Income Students: Needs vs. Wants |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fatma Ayyad, Western Michigan University, fattmah@hotmail.com |
| Marie Cannell, Western Michigan University, marie.e.cannell@wmich.edu |
| Abstract: This evaluation examined ways to improve a program intended to motivate children of low income families to help them graduate from high school and to pursuit higher education. The present evaluation focuses on the assessment of the program’s component which engages the students in university campuses visits. This component of the program known as “Creating a Want to Continue Education”, targets students in low-income communities. Qualitative and quantitative data were collected in the forms of observations and surveys involving students, teachers and tour guides. Although teachers and tour guides seem to be satisfied about the program, the activities conducted present a lack of interactive elements for the students. It is assumed that results from this formative evaluation will help the program’s leaders to improve up coming activities. |
| Early Childhood Programs and Children’s Behavior and Social Skills: A Longitudinal Investigation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sheretta Barnes, High Scope Educational Research Foundation, sbutler-barnes@highscope.org |
| Marijata Daniel-Echols, High Scope Educational Research Foundation, mdaniel-echols@highscope.org |
| Mary Delcamp, High Scope Educational Research Foundation, mdelcamp@highscope.org |
| Beth Scholz, High Scope Educational Research Foundation, bscholz@highscope.org |
| Olivia McKinney, High Scope Educational Research Foundation, oliviam@highscope.org |
| Rebecca Blisset, High Scope Educational Research Foundation, bblissett@highscope.org |
| Abstract: According to the U.S. Census Bureau (2006-2007) 54% of children between the ages of 3 and 4 are enrolled in school. Furthermore, the positive effects preschool attendance has on young children’s cognitive performance and school readiness is supported (Barnett, Lamy, & Jung, 2005). However, there is a dearth of research examining the different types of programs young children attend and the effects on socioemotional development. Therefore, it is important to explore how the type of early childhood program influences behavioral adjustment and social skills. The present study utilized a longitudinal growth model to examine demographic factors and program type (i.e. federally funded, state funded, or center based program) on children’s behavioral adjustment and social skills across a four year period. Results indicate that gender, parent’s education, food stamp assistance, and program type significantly accounted for the differences in young children’s behavioral adjustment and social skills. |
| Practical Considerations for Implementing Strong Evaluation Designs in School-Based Prevention Programs |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Darcy Richardson, LeCroy and Milligan Associates Inc, darcy@lecroymilligan.com |
| Joanne Basta, LeCroy and Milligan Associates Inc, jbasta@lecroymilligan.com |
| DeeAnn Arroyo, Pima Prevention Partnership, darroyo@thepartnership.us |
| Geoff Wood, LeCroy and Milligan Associates Inc, geoff@lecroymilligan.com |
| Abstract: School-based pregnancy prevention programs face numerous challenges with implementation and sustainability. A rigorous evaluation can be advantageous for addressing these challenges because it can provide periodic information on implementation processes and progress. It can also provide reliable and valid outcome data on which program improvement and best-practices can be based. However, an effective evaluation is difficult to accomplish in a school-setting. The strongest evaluation design may be impractical based on the complexities of the program being offered, financial considerations, and/or the responsiveness of the community. The challenge with these programs is to design a rigorous evaluation that is practical in a school-setting. This session will include discussion of the design and implementation of two school-based pregnancy prevention programs, including challenges and lessons learned. The importance of the partnership between program and evaluation will be emphasized throughout this interactive session. |
| Real World Challenges to Conducting an Implementation Evaluation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Herbert Baum, Macro International Inc, herbert.m.baum@macrointernational.com |
| William Scarbrough, Macro International Inc, william.h.scarbrough.iii@macrointernational.com |
| Felicia Solomon, National Institutes of Health, solomonf@mail.nih.gov |
| Arnold Love, Independent Consultant, arnold.love@gmail.com |
| Abstract: The National Cancer Institute has been working with community partners to disseminate an evidence-based program, Body & Soul, to increase the consumption of fruits and vegetables among Christian African Americans. Macro International was hired to conduct an implementation evaluation, determining what worked well with the dissemination effort and what did not. All evaluations depend on the timely and accurate submission of data, and working with Community-based organizations presents challenges with respect to data collection efforts. This paper reports on some of those challenges and how some were overcome. |
| Theory to Practice to Theory: Completing the Evaluation Cycle through Project-Based Learning |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deborah Bennett, Purdue University, bennett@purdue.edu |
| Susan Geier, Purdue University, geiers@purdue.edu |
| Abstract: Professionals in Higher Education are frequently unprepared to address the evaluation requirements of their positions. This is particularly concerning given the heightened attention to program accountability. These professionals also have little experience linking the theoretical models of their discipline to the programs they develop and serve. This poster describes a project-based approach to teaching program development and evaluation at the graduate level. Two student projects will be shared illustrating how theory can be linked to practice and how well designed and executed evaluation can help inform, refine, and expand theory. |
| Local Outcomes from the Rhode Island State Incentive Grant |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| John Stevenson, University of Rhode Island, jsteve@uri.edu |
| Nathan Stein, University of Rhode Island, nstein@mail.uri.edu |
| Israel Beren, University of Rhode Island, chezsruli@mail.uri.edu |
| Paul Florin, University of Rhode Island, pflorin@uri.edu |
| Abstract: This paper looks at outcomes from science-based substance abuse prevention initiatives funded in seven school districts in RI. Despite the development of a state-level policy linking local and state evaluations in a collaborative approach, evidence for local program effects was limited. Changes over time from 6th grade pre-tests to 7th grade post-tests on risk and protective factor measures (MANOVA and Chi-square analyses) were negligible. However, when “high implementation” schools were compared to “low implementation” schools as well as to the statewide picture, there were notably greater reductions in self-reported substance use for 8th graders in high implementation schools. Comparable fidelity of delivery was achieved but youth participation rates were starkly different. Conclusions are drawn about the importance of the delivery system for prevention (curricular insertion vs. after-school) as compared to the particular evidence-based curriculum (All Stars, Project Northland, Life Skills Training) for producing effects with practical significance for a community. |
| Evaluating the Impact of a Three-Hour Neuroscience of Addiction Curriculum on College Students’ Knowledge and Attitudes: Results From the National Institute on Drug Abuse Enters College Project |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Joyce Hartje, University of Nevada Reno, jhartje@casat.org |
| Michelle Berry, University of Nevada Reno, mberry@casat.org |
| Nancy Roget, University of Nevada Reno, nroget@casat.org |
| Wendy Woods, University of Nevada Reno, wwoods@casat.org |
| Angela Broadus, University of Nevada Reno, abroadus@casat.org |
| Abstract: The purpose of this project was to evaluate the impact of a 3-hour science of addiction research-based curriculum infusion on university undergraduate students’ knowledge and attitudes about substance use disorders. Two sections of existing introductory courses in three pre-professional disciplines (criminal justice, nursing, and social work) were selected. One section served as the curriculum infusion (implementation) group (N = 122); the other as the control (non-implementation) group (N = 178). Pre- and post-test measures were developed based on curriculum objectives to assess the effect on knowledge and attitudes. Results showed a significant increase in knowledge about addiction and a corresponding decrease in stigmatizing attitudes in 1) the implementation vs. non-implementation groups; and 2) the implementation group pretest vs. post-test measures. These findings lend support to the potential effectiveness of infusing a brief research-based addiction curriculum into existing undergraduate courses and could have long-term implications for preparing helping professionals. |
| Evaluating the 2nd Year Implementation and Effectiveness of the Kansas Afterschool Leadership Development Initiative to Enhance Afterschool Program Settings |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diane Betzen, Wichita State University, dmbetzen@wichita.edu |
| Tara Gregory, Wichita State University, tara.gregory@wichita.edu |
| Sarah Jolley, Wichita State University, sarah.jolley@wichita.edu |
| Emily Grant, Wichita State University, eagrant@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Debra Renollet, Wichita State University, dlrennolet@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Afterschool programs have long been seen as an appropriate avenue for creating positive youth development. However, there are few models that provide guidance to programs in creating the setting characteristics necessary to positive youth development. For this reason, the Kansas Afterschool Leadership Development Initiative (KALDI) youth leadership capacity-building intervention was implemented as a way to increase organizational effectiveness and promote efforts to achieve positive youth development outcomes through a focus on positive youth-adult relationships, youth-centered norms, and youth participation in decision-making. Results to be presented include the outcomes from measures of KALDI’s effectiveness in its second year, such as surveys of youth and adults in the programs, and observation and interview measures. Insights regarding the successes and challenges of efforts to impact settings through youth leadership capacity-building will be presented. |
| Community Needs Assessment for a Northern Sudanese Community in an Eastern Iowa Town |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Douglas Grane, University of Iowa, douglas.grane@gmail.com |
| Kirsten Beyer, University of Iowa, kirsten-beyer@uiowa.edu |
| Abstract: Immigrant and refugee communities face various struggles in the United States ranging from acculturation, access to social and health services, reinventing a community identity in the U.S., and employment and self-sufficiency challenges. Immigrants and refugees from northern Sudan intentionally founded a community in Iowa City, Iowa, after many in the community lived in other parts of the U.S. and other countries outside Sudan. We worked with the community to assess community needs through participant observation, Participatory Geographic Information Systems (PGIS) mapping, focus groups, and several in depth interviews. The needs assessment found that community leadership successfully mobilized the community in meeting most basic needs; the community expressed that their most significant challenge centered around highly educated community members working in occupations far below their educational status. |
| The Good, The Bad, and The Ugly: Outsourcing Experiences |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristin Pankey, Southern Illinois University at Carbondale, knpankey@siu.edu |
| Joel Nadler, Southern Illinois University at Carbondale, jnadler@siu.edu |
| Nicole Cundiff, Southern Illinois University at Carbondale, karim@siu.edu |
| Nicholas G Hoffman, Southern Illinois University at Carbondale, nghoff@siu.edu |
| Gargi Bhattacharya, Southern Illinois University at Carbondale, gargi@siu.edu |
| Alen Avdic, Southern Illinois University at Carbondale, alen@siu.edu |
| Abstract: The Applied Psychology program at the Southern Illinois University at Carbondale (SIUC) has been providing training in program evaluation since 1982. Perhaps the most unique part of this training is the opportunity to conduct professional evaluations as a part of Applied Research Consultants (ARC), a student-run consulting firm. Over the past few years, ARC has been hired for an increasing amount of outsourcing services and has had a variety of experiences with each of these projects and evaluations. Three specific situations stand out and can be described as “The Good, The Bad, and The Ugly.” A good experience is defined as a project that allows for the development of skills and abilities. “Bad” and “Ugly” experiences are profitable but with low educational value. This presentation will discuss these three situations, ARC’s experiences and relationships with these clients, and tips for dealing with outsourcing. |
| An Example of Technical Training Utilization |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sandhya Rao, Pricewaterhousecoopers LLP, sandhya.rao@us.pwc.com |
| James Lehman, Pricewaterhousecoopers LLP, james.lehman@us.pwc.com |
| Daniel Birch, Pricewaterhousecoopers LLP, daniel.birch@us.pwc.com |
| Abstract: An important measure of the impact of training is change in job-related behaviors. Behavior change is Level 3 of the Kirkpatrick training evaluation model (1959); in it one examines changes in job behavior as a result of training. Level 3 needs to be taken together with Reaction, Learning and Business Results for a holistic understanding of training. The study was designed to understand job behavior changes as a result of sending 90 new hires to a six-week long residential technical training program. Follow-up information was gathered from the learners -- employees of PricewaterhouseCoopers, LLP-- 5 months after training. Learners were asked when they applied learning to the job, frequency of accessing training materials, opportunities for applying training and additional training needed. Learners were also asked to identify peers and supervisors who had worked with them post-training, who were in turn surveyed to understand learners' behaviors. Preliminary analyses indicated that in certain technical areas, over 75% of learners indicated that they had not had a chance to utilize any knowledge and skills from training. The results are valuable as they will be used to improve training content and usability for future offerings. |
| A Logic Model Framework for Evaluation of a National Workforce Development Endeavor |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pamela Bishop, Oak Ridge Institute for Science and Education, pbaird@utk.edu |
| Samuel Held, Oak Ridge Institute for Science and Education, sam.held@orau.org |
| Abstract: In response to the Academic Competitiveness Council initiative, existing federally-funded education programs are developing or increasing their current evaluation capacities. Because many government programs are national in scope, ad hoc evaluators are faced with the difficult task of aligning evaluation plans across multiple levels and sites. This poster presentation provides a unique approach to the development of a nationally aligned evaluation of a multisite, multilevel, multiprogam workforce development enterprise using a logic model framework. The scope of the evaluation includes both the evaluation of five federally-funded, multisite science education programs, and a metaevaluation of common enterprise outcomes and impacts of all programs. The proposed evaluation framework aligns resources and outcomes at the local and national levels, so that the evaluation at all levels remains focused on the same fundamental goals for participants, programs, and the enterprise as a whole. |
| Early Childhood Programs and Children’s Behavior and Social Skills: A Longitudinal Investigation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sheretta Barnes, High Scope Educational Research Foundation, sbutler-barnes@highscope.org |
| Marijata Daniel-Echols, High Scope Educational Research Foundation, mdaniel-echols@highscope.org |
| Mary Delcamp, High Scope Educational Research Foundation, mdelcamp@highscope.org |
| Beth Scholz, High Scope Educational Research Foundation, bscholz@highscope.org |
| Olivia McKinney, High Scope Educational Research Foundation, oliviam@highscope.org |
| Rebecca Blisset, High Scope Educational Research Foundation, bblissett@highscope.org |
| Abstract: According to the U.S. Census Bureau (2006-2007) 54% of children between the ages of 3 and 4 are enrolled in school. Furthermore, the positive effects preschool attendance has on young children’s cognitive performance and school readiness is supported (Barnett, Lamy, & Jung, 2005). However, there is a dearth of research examining the different types of programs young children attend and the effects on socioemotional development. Therefore, it is important to explore how the type of early childhood program influences behavioral adjustment and social skills. The present study utilized a longitudinal growth model to examine demographic factors and program type (i.e. federally funded, state funded, or center based program) on children’s behavioral adjustment and social skills across a four year period. Results indicate that gender, parent’s education, food stamp assistance, and program type significantly accounted for the differences in young children’s behavioral adjustment and social skills. |
| Evaluating Smaller Learning Communities in United States High Schools |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Linda Bol, Old Dominion University, lbol@odu.edu |
| Shana Pribesh, Old Dominion University, spribesh@odu.edu |
| Christopher Fischer, Old Dominion University, cfischer@nps.k12.va.us |
| John Nunnery, Old Dominion University, jnunnery@odu.edu |
| Abstract: Creating smaller schools/learning communities (SLCs) have been advocated as a specific reform for improving high school student engagement and graduation rates (NRC, 2002). The linkages of smaller learning communities to student achievement have been found to be promising (Felner, Ginter, and Primavera, 1982; NRC, 2002). However, the research connecting the SLC structure with student performance is tenuous – mostly due to methodological issues. We propose to discuss strategies to evaluate the effect of smaller learning communities in U.S. high schools on student achievement and other outcome measures. This poster will use case studies to propose innovative, rigorous designs to yield more valid evaluation findings. In addition, we would identify other constructs (e.g., school climate, student self-concept and motivation) theoretically linked to achievement that could be employed as additional outcome measures. This poster will be useful for practitioners faced with evaluating smaller schools within schools in public school districts. |
| Building Capacity in Science: Evaluation of a Science Inquiry Center |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bridget Cotner, University of South Florida, bcotner@cas.usf.edu |
| Chrystal Smith, University of South Florida, casmith5@cas.usf.edu |
| Kathryn Borman, University of South Florida, kborman@cas.usf.edu |
| Abstract: This evaluation used an objective oriented approach to determine if the Center's projects are accomplishing the four goals set forth when founded. These four goals are 1) provide research opportunities in science education and learning, 2) rebuild and diversify the human resource base for science, technology, engineering and mathematic (STEM) education, 3) increase the number of K-12 STEM educators with the knowledge to implement standards-based science instruction, and 4) create synergy between research and practice through research based practice and translate results into practical suggestions for science educators. The evaluation provided formative and summative information to the Center to assist them in their quest to build capacity in science. |
| An Alternative Method of Comparing Samples: The Analysis of Means |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| George Bratton, Independent Consultant, gbratton@conwaycorp.net |
| Abstract: The analysis of means (ANOM) is a graphical procedure for comparing a collection of means, rates, or proportions to see if any of them are significantly different from the overall, mean, rate, or proportion. When interested in the comparison of means ANOM is an alternative to ANOVA. Even though it is not a broadly applicable as ANOVA, ANOM has the advantages of being much more intuitive and providing an easily understood graphical result. Its results clearly indicate any means that are different from the overall mean and all for easy assessment of practical as well as statistical significance. The graphical presentation of results is easy for nonstatisticians to understand. |
| Evaluation of an Obesity Prevention Intervention for Food Pantry Clients: The WELL program (Walk Eat Learn Live) |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Laura Runnels, Transtria LLC, lrunnels@transtria.com |
| Julie Claus, Transtria LLC, julie@transtria.com |
| Laura Brennan Ramirez, Transtria LLC, laura@transtria.com |
| Abstract: Obesity in the United States is an economic issue related to limited social and economic resources and linked to disparities in access to healthy foods. The W.E.L.L. program (Walk. Eat. Learn. Live.) aims to improve health and reduce rates of obesity among food pantry clients through a variety of strategies including cooking classes, taste tests, walking groups, and community education. A variety of measures have been utilized to evaluate the program, including surveys, interviews, focus groups, feedback forms, and attendance sheets, which assess improvements in knowledge, beliefs, and behaviors of participants. Pantry audits measure improvements in healthy food donations. Preliminary findings indicate that while participants’ knowledge is high, they face numerous barriers to making healthy food choices. Pantry audits have shown an increase in the number of healthy foods donated by the broader community. |
| A Comprehensive Evaluation of the Active Living by Design Initiative: Innovative Approaches to Increase Physical Activity |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Melissa Hall, Transtria LLC, melissa@transtria.com |
| Laura Brennan Ramirez, Transtria LLC, laura@transtria.com |
| Cheryl Kelly, Saint Louis University, kellycm@slu.edu |
| Abstract: The Active Living by Design (ALbD) initiative establishes innovative approaches to increase physical activity through community design, public policies and communications strategies. Evaluation efforts focus on the ALbD Community Action Model components: preparation, promotions, programs, physical projects, and policies. This evaluation is intended to capture the range of grantee impacts and the strengths and challenges of implementing the ALbD interventions at the local level while focusing on three primary aims: 1) to assess the environmental impacts of physical projects and associated policy changes, 2) to document the range of interventions and accomplishments, and 3) to identify strengths and challenges in the process of planning, developing, and implementing the interventions. Visits to each of the 25 grantee sites will be completed by November 2008. This poster will highlight the methods used in the evaluation as well as findings and recommendations for building community design initiatives to increase physical activity. |
| Critical Success Factors in Implementing a Program Evaluation for a Scientific Research Center of Excellence Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rajni Samavedam, Booz Allen Hamilton, samavedam_rajni@bah.com |
| Elaine Brenner, Booz Allen Hamilton, brenner_elaine@bah.com |
| Abstract: In conducting a process and outcome evaluation study for a Federally-funded scientific research program, the evaluation team identified four critical success factors that contribute to the success in conducting the evaluation: creating a blended evaluation team; using an expert panel; striving for creative solutions for defining the right comparison group; and having flexibility in approach as information is learned. |
| Evaluating the Impact of a Three-Hour Neuroscience of Addiction Curriculum on College Students’ Knowledge and Attitudes: Results From the National Institute on Drug Abuse Enters College Project |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Joyce Hartje, University of Nevada Reno, jhartje@casat.org |
| Michelle Berry, University of Nevada Reno, mberry@casat.org |
| Nancy Roget, University of Nevada Reno, nroget@casat.org |
| Wendy Woods, University of Nevada Reno, wwoods@casat.org |
| Angela Broadus, University of Nevada Reno, abroadus@casat.org |
| Abstract: The purpose of this project was to evaluate the impact of a 3-hour science of addiction research-based curriculum infusion on university undergraduate students’ knowledge and attitudes about substance use disorders. Two sections of existing introductory courses in three pre-professional disciplines (criminal justice, nursing, and social work) were selected. One section served as the curriculum infusion (implementation) group (N = 122); the other as the control (non-implementation) group (N = 178). Pre- and post-test measures were developed based on curriculum objectives to assess the effect on knowledge and attitudes. Results showed a significant increase in knowledge about addiction and a corresponding decrease in stigmatizing attitudes in 1) the implementation vs. non-implementation groups; and 2) the implementation group pretest vs. post-test measures. These findings lend support to the potential effectiveness of infusing a brief research-based addiction curriculum into existing undergraduate courses and could have long-term implications for preparing helping professionals. |
| Results of a Baseline Survey of Barriers and Facilitators of Clinical Translational Research |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fredric Wolf, University of Washington, wolf@u.washington.edu |
| Douglas Brock, University of Washington, dmbrock@u.washington.edu |
| Lynne Robins, University of Washington, lynner@u.washington.edu |
| Pamela Nagasawa, University of Washington, pnaga22@u.washington.edu |
| Abstract: Barriers to the conduct of clinical and translational research were identified as part of a mixed-methods evaluation of a newly created Institute of Translational Health Sciences (ITHS). The ITHS was created as part of the National Institute of Health’s Roadmap Clinical and Translational Sciences Award program. Ten barriers identified in an earlier open-ended survey were rated in terms of 1) current level of satisfaction and 2) degree to which each represented a barrier to research and were then 3) “prioritized” in terms of urgency for addressing each (Not a priority, > 10 years, 5-10 years, 1-5 years, Year 1, Immediate). Modal responses of 215 researchers indicated an “Immediate” need to address two areas: Institutional Review Board (IRB) operations and need for more local grant support to gather preliminary data to prepare larger grant applications. All other areas received modal responses suggesting they be addressed within five years. |
| Achieving a Successful Semester-Long Program Evaluation Experience: Tips from a One-on-One Course |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carrie Brown, Saint Louis University, cbrown53@slu.edu |
| Angela Walmsley, Saint Louis University, walmsley@slu.edu |
| Abstract: Graduate students can feel discouraged to conduct a program evaluation by thinking it will take too long to complete. The graduate student and professor involved with this presentation recently completed a program evaluation together, from start to finish, within a semester time-frame. The evaluation was conducted to fulfill the student’s requirement for a one-on-one program evaluation course, and investigated an economics educational program for 3rd and 4th graders at a children’s museum. Both qualitative and quantitative methods were implemented. Methods triangulation was employed through paper-and-pencil surveys, telephone interviews, e-mail correspondence, and site visits. Participant triangulation was also utilized. Upon conclusion of the semester, a final written report was given to the museum. Graduate students interested in conducting a program evaluation, and professors interested in giving a program evaluation course, may find this experience helpful in thinking of ways to develop a semester-long program evaluation that is both complete and satisfying. |
| Using Evaluation as a Reflective Tool to Sustain Changes When Schools Reform |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sharon Brown, Cleveland State University, s.a.brown54@csuohio.edu |
| James Salzman, Cleveland State University, j.salzman@csuohio.edu |
| Abstract: Even in effective school change efforts, the results are often short lived (Tyack, 1995). Continuance of school change beyond the timeline of the funded grant is challenging and a process that is often left to chance (Fullan, 2001). In Ohio, school personnel were trained in the use of a tool to help them maintain fidelity to three program standards for Reading First by identifying relevant artifacts that represented their implementation, and independent consultants used this tool three times/year to monitor fidelity. By training stakeholders in document selection using principles similar to auditing (relevance, reliability, sufficiency, representativness and timeliness) and guiding participants in programmatic conversations, the process of document review proved to be a powerful, cost-effective tool for collecting implementation data and building organizational capacity for program sustainability. This paper is a case study of one district’s use of monitoring program fidelity to sustain their school reform efforts. |
| Interactive Evaluation as an Essential Component in An Innovative Annual Workshop |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Lynds, University of Colorado, susan.lynds@colorado.edu |
| Tamara Ledley, TERC, tamara_ledley@terc.edu |
| Susan Buhr, University of Colorado, susan.buhr@colorado.edu |
| Abstract: Since 2004, the annual AccessData and DLESE (Digital Library for Earth Systems Education) Data Services workshops have gathered scientists, data managers, technology specialists, teachers, and curriculum developers to work together creating classroom-ready scientific data modules. Teams of five (one participant from each of the five professions) develop topic-specific online educational units for online dissemination Extensive evaluation has been included in the design of each workshop. The evaluation results have been used each year to improve the agenda, topics, and procedures of each subsequent event. Evaluation data has also informed modification of the evaluation protocol each year. Some of the information collected each year reveals attendees’ experiences using scientific data for educational purposes—analysis of multi-year data sets reveals how these experiences have change over time. This poster reviews the evaluation design that has been essential in enhancing this unique and effective annual workshop. |
| Developing and Teaching Evaluation Courses for Both Brick and Mortar and Also Online Institutions Of Higher Education |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Cheryl Bullock, Walden University, cdbulloc@uiuc.edu |
| Abstract: The presenter has developed and taught graduate level project based evaluation courses in traditional brick and mortar, hybrid, and completely online venues in higher education. Two of these venues (traditional face to face, and hybrid) were taught at a Research One Brick and Mortar Institute. The third type was taught at an accredited online institution of higher education. Information about the seminaries and differences of developing and teaching evaluation in these three different venues will be presented. Additionally, information about how students were able to complete their projects and make final presentations, both in class and through the use of technology for the online students will be discussed. Finally, the presenter will be available le to answer any questions about the process and/or the product of designing and teaching a graduate level evaluation course (both online and face to face). |
| Time Gap Model for the Bombings in Region XII, Mindanao, Philippines |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Epimaco Jr Cabanlit, Mindanao State University, maco_727@yahoo.com |
| Robert Kiunisala, South Cotabato Police Provincial Office, robert_bing@yahoo.com |
| Romeo Jr Galgo, Regional Mobile Group, academy158@yahoo.com |
| Steiljes M Cabanlit, Mindanao State University, steilt_727@yahoo.com |
| Abstract: From the year 2000 to 2006, a total of 96 bombing incidents occurred, 101 bombs exploded, 86 were killed and 674 were injured in Region XII, Mindanao, Philippines. This paper presents a time gap model for these bombings. The mixture of exponential and uniform distribution shows a good fit. |
| Time Gap Model for the Bombings in Region XII, Mindanao, Philippines |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Epimaco Jr Cabanlit, Mindanao State University, maco_727@yahoo.com |
| Robert Kiunisala, South Cotabato Police Provincial Office, robert_bing@yahoo.com |
| Romeo Jr Galgo, Regional Mobile Group, academy158@yahoo.com |
| Steiljes M Cabanlit, Mindanao State University, steilt_727@yahoo.com |
| Abstract: From the year 2000 to 2006, a total of 96 bombing incidents occurred, 101 bombs exploded, 86 were killed and 674 were injured in Region XII, Mindanao, Philippines. This paper presents a time gap model for these bombings. The mixture of exponential and uniform distribution shows a good fit. |
| Providing Intervention to Struggling Readers in Bureau of Indian Education Reading First Schools: Does the Data Show Its Making a Difference? |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Catherine Callow-Heusser, EndVision Research and Evaluation, cheusser@endvision.net |
| Heather Chapman, EndVision Research and Evaluation, hjchapman@cc.usu.edu |
| Abstract: The Reading First program was designed without opportunities for determining program effectiveness using the “gold standard” of randomized experimental designs, where students, teachers or schools are randomly assigned to treatment and control groups. However, Reading First provides an ideal context for implementing a quasi-experimental regression discontinuity design (RDD), where students below a pre-specified cutoff receive intervention that is different that those above the cutoff. We used a RDD with Bureau of Indian Education Reading First data to answer the question, “Do students who begin the school year two or more grade levels behind as measured by the DIBELS benchmark assessments receive intervention that is 'different' than students who read nearer grade level, and what is the magnitude of that difference?” Analyses included DIBELS and SAT-10 outcome measures. Findings indicate that effect sizes were as large as 1.5 based on DIBELS outcomes and 0.7 based on SAT-10 outcomes. |
| Evaluation of Program Intended for Low-Income Students: Needs vs. Wants |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fatma Ayyad, Western Michigan University, fattmah@hotmail.com |
| Marie Cannell, Western Michigan University, marie.e.cannell@wmich.edu |
| Abstract: This evaluation examined ways to improve a program intended to motivate children of low income families to help them graduate from high school and to pursuit higher education. The present evaluation focuses on the assessment of the program’s component which engages the students in university campuses visits. This component of the program known as “Creating a Want to Continue Education”, targets students in low-income communities. Qualitative and quantitative data were collected in the forms of observations and surveys involving students, teachers and tour guides. Although teachers and tour guides seem to be satisfied about the program, the activities conducted present a lack of interactive elements for the students. It is assumed that results from this formative evaluation will help the program’s leaders to improve up coming activities. |
| Nonprofit Evaluation: How the Costs and Benefits of Membership Influence Evaluation Criteria |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tosha Cantrell-Bruce, University of Illinois Springfield, tcantrel@uis.edu |
| Abstract: This research examines the criteria stakeholders use to assess the effectiveness of their nonprofit organization and the relationship between these criteria and the perceived costs and benefits of joining the nonprofit. The multiple constituency approach to organizational effectiveness provides the theoretical basis used to assess performance in member-benefit nonprofits, and specifically in self-help groups. Social exchange and political economy theory provides the theoretical basis used to assess the costs and benefits of membership. Data collected from several chapters of the National Alliance on Mental Illness of Illinois (NAMI) are used to test the hypothesis that effectiveness criteria are positively related to costs and benefits. This research informs evaluators of specific dimensions of effectiveness criteria used by self-help nonprofit stakeholders as well as the influence of the costs and benefits of membership. Knowledge of this relationship can inform specific evaluation practice. |
| An Assessment of the Process and Instrumental Use in an Application of the Most Significant Change Technique |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Nancy Carrillo, Albuquerque Public Schools, carrillo_n@aps.edu |
| Abstract: This paper describes how I assessed the process and instrumental use of an evaluation using the Most Significant Change (MSC) technique. A school district received a grant to fund a substance abuse prevention counselor for three years in each of twelve high schools. MSC was one technique used to evaluate the initiative; however, the application of MSC was suboptimal as the funder did not provide feedback. Given that participatory evaluation techniques such as MSC are expected to increase instrumental and process use of evaluation, I wondered if this application of MSC had any impact. I have three research questions. First, to what extent did process and instrumental use occur as a result of the MSC technique? Second, what types of process use, if any, did stakeholders engage in? Third, did counselors and district personnel perceive a diminishing rate of return for MSC? I collected data using questionnaires, observations and interviews. |
| Examining Time-to-Degree within a Multidisciplinary, Multi-Institutional Graduate Training Program--Successes and Challenges |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Dylan Flather, University of Colorado Boulder, dylan.flather@colorado.edu |
| Kristin Kusmierek, University of Michigan, kkusmierek@stanfordalumni.org |
| Tyler Silverman, University of Colorado Boulder, tsilverman1@gmail.com |
| Mary Anne Carroll, University of Michigan, mcarroll@umich.edu |
| David Karowe, Western Michigan University, karowe@wmich.edu |
| Abstract: Time-to-degree is particularly important in interdisciplinary programs, often thought to add time compared to traditional programs. The Biosphere-Atmosphere Research and Training (BART) program, a project of the National Science Foundation’s Integrative Graduate Education and Research Traineeship (IGERT) initiative, is an interdisciplinary program with participants from over a dozen disciplines and universities. This analysis compares time-to-degree of participants in BART’s two-year supplemental program against the time for their non-BART peers from corresponding home institutions. The preliminary results show that former BART students graduate in about the same time as non-BART students. |
| The Effect of On-Campus, Academic Year, Laboratory Based Research Experiences on STEM PhD Outcomes for Underrepresented Minorities |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Frances Carter, University of Maryland Baltimore County, frances2@umbc.edu |
| Marv Mandell, University of Maryland Baltimore County, mandell@umbc.edu |
| Abstract: The shortage of underrepresented minorities in science, technology, engineering and mathematics (STEM) research careers is attracting increased attention from evaluators and policymakers. Numerous programs intended to increase the participation of underrepresented minorities in STEM research careers have been implemented to address this problem. A key component in many of these programs is providing undergraduate research opportunities to program participants. However, limited evaluation has been done regarding the impact of participation in undergraduate research on pursuit and attainment of a STEM Ph.D. This paper analyzes the impacts of on-campus, academic year, laboratory based research in STEM fields on STEM PhD outcomes. The analysis uses data from fourteen cohorts of participants in the Meyerhoff Scholars Program at the University of Maryland, Baltimore County. A variety of estimation strategies, including OLS, instrumental variables estimation, and propensity matching are used to estimate the effects and address possible issues of endogeneity. |
| A Long-Distance Evaluation of an In-Home Family Services Model: Planning Processes, Successes and Barriers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stephanie Ingram, Boys Town, ingrams@boystown.org |
| Tanya Shaw, Boys Town, shawt@boystown.org |
| Kristin Duppong Hurley, University of Nebraska Lincoln, kdupponghurley2@unl.edu |
| Katy Casey, University of Nebraska Lincoln, katyjcasey@yahoo.com |
| Abstract: Family service interventions are under increasing pressure from funding agencies and clients to demonstrate their effectiveness. Specifically, agencies want to gain a better understanding of what works, for which clients, under what conditions. This difficult task becomes even more complex when conducting a large scale evaluation of at-risk populations in a collaborative research effort, as there are many challenges and processes to plan, including: how to select appropriate measures, how to train staff in complex data collection procedures, how to conduct program fidelity, how to maintain client confidentiality between the service provider and an university research center, and how to monitor data collection, all from 1500 miles away from the intervention site. This paper will discuss the planning processes, challenges and ways to overcome barriers of a long-distance, complex evaluation to help other researchers planning large scale evaluations across the miles. |
| Supporting Causal Inferences and Generalization When Evaluating Large-scale, Multi-state, Multi-level Implementations of an Integrated Instructional Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brad Chaney, Westat, bradchaney@westat.com |
| Jane Delgado, College Board, jdelgado@collegeboard.org |
| Doreen Finkelstein, College Board, dfinkelstein@collegeboard.org |
| Joan Michie, Westat, michiej1@westat.com |
| Abstract: In a large-scale, longitudinal, summative evaluation effort, issues of internal validity, external validity, and reliability and were addressed in supporting causal inferences about the impact and efficacy of an integrated instructional program. The program included curriculum, assessment and professional development for both English language arts and mathematics and implementation patterns were very different by subject area, school, district, and state. An interim dataset of more than 1.5 million observations of individual student achievement across multiple years was secured from participating and comparison schools in nine districts and six different states, both before and after program exposure. A repeated measures multi-level modeling approach was utilized in the SAS analyses. Complexities addressed include: using scale scores from different statewide standardized tests as the measure of achievement, determining an effect size when state scale scores differed in type and range, dealing with patterns of missing data, and accounting for varying degrees of exposure at the level of the student and the school. |
| Providing Intervention to Struggling Readers in Bureau of Indian Education Reading First Schools: Does the Data Show Its Making a Difference? |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Catherine Callow-Heusser, EndVision Research and Evaluation, cheusser@endvision.net |
| Heather Chapman, EndVision Research and Evaluation, hjchapman@cc.usu.edu |
| Abstract: The Reading First program was designed without opportunities for determining program effectiveness using the “gold standard” of randomized experimental designs, where students, teachers or schools are randomly assigned to treatment and control groups. However, Reading First provides an ideal context for implementing a quasi-experimental regression discontinuity design (RDD), where students below a pre-specified cutoff receive intervention that is different that those above the cutoff. We used a RDD with Bureau of Indian Education Reading First data to answer the question, “Do students who begin the school year two or more grade levels behind as measured by the DIBELS benchmark assessments receive intervention that is 'different' than students who read nearer grade level, and what is the magnitude of that difference?” Analyses included DIBELS and SAT-10 outcome measures. Findings indicate that effect sizes were as large as 1.5 based on DIBELS outcomes and 0.7 based on SAT-10 outcomes. |
| Exploring Logic Model Applications to Course Design and Outcome Goals: Assisting Teaching and Evaluation Efforts |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Roy Charles, University of North Carolina Chapel Hill, rac@unc.edu |
| Abstract: Logic modeling offers a systematic approach to determining, understanding, and planning the components of a variety of activities. One critical activity in which not only the quality of delivery as well as the effectiveness of the evaluation of the activity may be improved is that of teaching. In recent years, there has been significant discussion around the topic of accountability in higher education, which is putting greater emphasis on the need for effective evaluation of various factors including teaching and learning. This paper explores concepts of instructional design that can be married with the process of logic model development in order to improve potentially teacher preparation and delivery, as well as student topical conceptual understanding and overall learning. |
| Determining the Longitudinal Impacts of Summer Research Experiences on Underrepresented Minority Undergraduates Intent to Pursue Graduate Degrees |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Karyl Askew, University of North Carolina Chapel Hill, karyls@email.unc.edu |
| Roy Charles, University of North Carolina Chapel Hill, rac@email.unc.edu |
| Abstract: The poster presents a method for evaluating the longitudinal impacts of undergraduate summer research programs (USRP). For this evaluation the USRP host institution was one of three members of the North Carolina Alliance to Create Opportunity Through Education (NC OPT-ED), a National Science Foundation’s funded Alliance for Graduate Education and the Professoriate (AGEP) partnership. This is part of a multi-year evaluation to determine the extent to which programming contributes to the AGEP mission of increasing the number of underrepresented minority students receiving science, technology, engineering and mathematics (STEM) doctorates and entering the professoriate. To examine longitudinal student impacts, evaluators used previously submitted student letters of recommendations and employed purposive sampling to identify 11 faculty advisors from 7 institutions. The original sample included 122 program participant recommendation letters from three summer cohorts. Evaluators conducted semi-structured interviews with faculty advisors. Presenter will share successes and lessons learned from implementing this technique. |
| Minimization as an Alternative to Unrestricted Randomization in Educational Research |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Hui-Fang Chen, University of Denver, hchen25@du.edu |
| Kathy Green, University of Colorado Denver, kgreen@du.edu |
| Abstract: This paper describes an alternative to unrestricted randomization in experimental design, minimization, which can be used to achieve better balance of critical factors in small to medium-sized experimental studies. With minimization, several critical factors are controlled which are known to influence outcomes but which are not the foci of the study. Based on the critical factor values of previously entered participants, a new participant will be allocated into an experimental group to minimize imbalance across groups in critical factors. Minimization has been primarily employed in medical interventions and clinical trials. This article proposes examples of the potential use of minimization in education, its advantages, and limitations. |
| The Domino Effect: Informing Program Policy Through Evaluative Practices- A Day Treatment Program Case Study |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jacquelyn Christensen, Los Angeles Child Guidance Clinic, jacquelynsheac@hotmail.com |
| Barbara Stroud, Los Angeles Child Guidance Clinic, bstroud@lacgc.org |
| Tony Wu, Los Angeles Child Guidance Clinic, tonyw@lacgc.org |
| Abstract: This paper highlights the importance of evaluating early intervention programs, and provides a case study that describes how utilization-focused evaluation not only resulted in increased program accountability, but also elicited program improvement at both the program level as well as overall organization policy. In evaluating a nonprofit early childhood day-treatment program, it was demonstrated that parental stress was significantly reduced after children’s participation in the program. Data from this preliminary study suggested the benefits of an early intervention program in addition to how the outcomes were utilized to guide and inform administrative policy in terms of program implementation and service delivery. Implications from this study revealed the necessity of evaluation and speak to the reciprocal interaction between evaluation and program improvement. |
| Assessing the Quality of School Improvement Program in Three Special Education Centers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ariana Leo Ramírez, University of Autonoma Yucatan, chinari_17@hotmail.com |
| Edith Cisneros-Cohernour, University of Autonoma Yucatan, cchacon@uady.mx |
| Abstract: This paper presents the findings of a qualitative study assessing the quality of a school improvement program in Mexico. The study examines how the program, developed as part of a standardized approach for school improvement, was responsive to the needs of the main stakeholders at the school level. |
| Evaluation of an Obesity Prevention Intervention for Food Pantry Clients: The WELL program (Walk Eat Learn Live) |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Laura Runnels, Transtria LLC, lrunnels@transtria.com |
| Julie Claus, Transtria LLC, julie@transtria.com |
| Laura Brennan Ramirez, Transtria LLC, laura@transtria.com |
| Abstract: Obesity in the United States is an economic issue related to limited social and economic resources and linked to disparities in access to healthy foods. The W.E.L.L. program (Walk. Eat. Learn. Live.) aims to improve health and reduce rates of obesity among food pantry clients through a variety of strategies including cooking classes, taste tests, walking groups, and community education. A variety of measures have been utilized to evaluate the program, including surveys, interviews, focus groups, feedback forms, and attendance sheets, which assess improvements in knowledge, beliefs, and behaviors of participants. Pantry audits measure improvements in healthy food donations. Preliminary findings indicate that while participants’ knowledge is high, they face numerous barriers to making healthy food choices. Pantry audits have shown an increase in the number of healthy foods donated by the broader community. |
| Building Evaluation Capacity in Regional Centers Using Formal Evaluation Courses |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Janet Clinton, University of Auckland, j.clinton@auckland.ac.nz |
| Sarah Appleton, University of Auckland, sk.appleton@auckland.ac.nz |
| Abstract: This paper describes the implementation of a suite of formal public health education programs within a large scale long term evaluation with the aim of building evaluation capacity and organizational learning in regional centre within New Zealand. Evaluation capacity and willingness to engage in evaluation were measured and the educational programs were tailored to meet the communities need. The immediate impact of the first of the formal evaluation courses, the technical assistance and the short course are explored. This paper contents that combining evaluation of programs with building capacity through formal training is a useful procedure for rural or distant communities where access to services is limited. A number of issues were identified throughout this process. In addition, embedding evaluation knowledge into the practical environment the ethical considerations of the evaluator teaching formal qualifications as a part of a long term evaluation contract are discussed. |
| Popular Opinion Leader (POL) On the Internet: Adapting an Evidence Based Intervention for MSM Who Seek Sex Online |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sara Gillen, Harlem United Community AIDS Center Inc, sgillen@harlemunited.org |
| George Gates, Harlem United Community AIDS Center Inc, ggates@harlemunited.org |
| Rashi Rohatgi, Harlem United Community AIDS Center Inc, rrohatgi@harlemunited.org |
| Michael Coley, Harlem United Community AIDS Center Inc, mcoley@harlemunited.org |
| Abstract: To address the increasing HIV rates among YMCSM and the need for more prolific prevention efforts, the Popular Opinion Leader (POL) program has been adapted and will be implemented and evaluated by Harlem United Community AIDS Center in New York City. POL is a community-level HIV prevention intervention designed to enlist and train the most influential peers to encourage safer sex norms and behaviors through informal risk-reduction conversations. Objectives of the formative evaluation include identifying service needs and HIV-related risk factors among community members, and innovative ways to properly adapt the POL intervention for the internet. Preliminary, formative findings, identified a community ostracized for engaging in unprotected anal sex, despite simultaneously practicing safer sex; a lack of online sex positive and harm reduction prevention messaging; effective methods for engaging men online (e.g. most popular websites), powerful alternative prevention messaging, and behavioral norms (e.g. lack of discussion concerning unsafe sex practices) |
| Building Capacity in Science: Evaluation of a Science Inquiry Center |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bridget Cotner, University of South Florida, bcotner@cas.usf.edu |
| Chrystal Smith, University of South Florida, casmith5@cas.usf.edu |
| Kathryn Borman, University of South Florida, kborman@cas.usf.edu |
| Abstract: This evaluation used an objective oriented approach to determine if the Center's projects are accomplishing the four goals set forth when founded. These four goals are 1) provide research opportunities in science education and learning, 2) rebuild and diversify the human resource base for science, technology, engineering and mathematic (STEM) education, 3) increase the number of K-12 STEM educators with the knowledge to implement standards-based science instruction, and 4) create synergy between research and practice through research based practice and translate results into practical suggestions for science educators. The evaluation provided formative and summative information to the Center to assist them in their quest to build capacity in science. |
| Data Acquisition: Considered the Future, but Stuck to the Past |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| John Enger, Nova Southeastern University, enger@bellsouth.net |
| Candace Lacey, Nova Southeastern University, chlacey@bellsouth.net |
| Nydia Cummings, Nova Southeastern University, cnydia@nova.edu |
| Abstract: While moving to online data acquisition to obtain student self-report measures, this presentation shows the problems encountered that resorted in data collection using paper and pencil. Further, in evaluating a comprehensive middle school character education program, two new surveys were created for the project. Their attributes will also be addressed. |
| The Good, The Bad, and The Ugly: Outsourcing Experiences |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristin Pankey, Southern Illinois University at Carbondale, knpankey@siu.edu |
| Joel Nadler, Southern Illinois University at Carbondale, jnadler@siu.edu |
| Nicole Cundiff, Southern Illinois University at Carbondale, karim@siu.edu |
| Nicholas G Hoffman, Southern Illinois University at Carbondale, nghoff@siu.edu |
| Gargi Bhattacharya, Southern Illinois University at Carbondale, gargi@siu.edu |
| Alen Avdic, Southern Illinois University at Carbondale, alen@siu.edu |
| Abstract: The Applied Psychology program at the Southern Illinois University at Carbondale (SIUC) has been providing training in program evaluation since 1982. Perhaps the most unique part of this training is the opportunity to conduct professional evaluations as a part of Applied Research Consultants (ARC), a student-run consulting firm. Over the past few years, ARC has been hired for an increasing amount of outsourcing services and has had a variety of experiences with each of these projects and evaluations. Three specific situations stand out and can be described as “The Good, The Bad, and The Ugly.” A good experience is defined as a project that allows for the development of skills and abilities. “Bad” and “Ugly” experiences are profitable but with low educational value. This presentation will discuss these three situations, ARC’s experiences and relationships with these clients, and tips for dealing with outsourcing. |
| Using Theory-Driven Evaluation to Assess Perceptions of Barriers And Supports to Education And Career Attainment Among Income-Eligible Youth Participants in a Summer Jobs Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Timothy Cunningham, Harvard University, tcunning@hsph.harvard.edu |
| Abstract: According to Weiss (1997), theory-driven evaluation is defined as examining the mechanisms that mediate the process and outcomes of a program. Counseling psychology research indicates that youth of different race/ethnicity often share similar aspirations, interests, and career decision-making proficiency, however, both barriers and supports often mediate the relationship between individual factors and educational and career attainment. This evaluation sought to examine perceptions of barriers and supports to educational and career attainment among youth that participated in a summer jobs program for income-eligible youth. The findings of this evaluation reinforce previous evidence identifying contextual barriers and supports as perceived by youth. The evaluation results suggest that educational and career interventions should address multiple factors that influence educational and career attainment, including youth perceptions, family involvement, and peer influence. |
| Early Childhood Programs and Children’s Behavior and Social Skills: A Longitudinal Investigation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sheretta Barnes, High Scope Educational Research Foundation, sbutler-barnes@highscope.org |
| Marijata Daniel-Echols, High Scope Educational Research Foundation, mdaniel-echols@highscope.org |
| Mary Delcamp, High Scope Educational Research Foundation, mdelcamp@highscope.org |
| Beth Scholz, High Scope Educational Research Foundation, bscholz@highscope.org |
| Olivia McKinney, High Scope Educational Research Foundation, oliviam@highscope.org |
| Rebecca Blisset, High Scope Educational Research Foundation, bblissett@highscope.org |
| Abstract: According to the U.S. Census Bureau (2006-2007) 54% of children between the ages of 3 and 4 are enrolled in school. Furthermore, the positive effects preschool attendance has on young children’s cognitive performance and school readiness is supported (Barnett, Lamy, & Jung, 2005). However, there is a dearth of research examining the different types of programs young children attend and the effects on socioemotional development. Therefore, it is important to explore how the type of early childhood program influences behavioral adjustment and social skills. The present study utilized a longitudinal growth model to examine demographic factors and program type (i.e. federally funded, state funded, or center based program) on children’s behavioral adjustment and social skills across a four year period. Results indicate that gender, parent’s education, food stamp assistance, and program type significantly accounted for the differences in young children’s behavioral adjustment and social skills. |
| Retaining Female Faculty in Higher Education |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| John Dantzler, University of Alabama Birmingham, dantzler@uab.edu |
| Heather Pleasants, University of Alabama, hpleasan@bamaed.ua.edu |
| Abstract: An outcomes evaluation of a National Science Foundation sponsored program to increase the recruitment and retention of female faculty in higher education will be presented. The evaluation utilized qualitative methods within a critical gender framework. A discussion of the unique challenges associated with the evaluation, as well as themes developed from the analysis will be displayed. |
| Early Childhood Programs and Children’s Behavior and Social Skills: A Longitudinal Investigation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sheretta Barnes, High Scope Educational Research Foundation, sbutler-barnes@highscope.org |
| Marijata Daniel-Echols, High Scope Educational Research Foundation, mdaniel-echols@highscope.org |
| Mary Delcamp, High Scope Educational Research Foundation, mdelcamp@highscope.org |
| Beth Scholz, High Scope Educational Research Foundation, bscholz@highscope.org |
| Olivia McKinney, High Scope Educational Research Foundation, oliviam@highscope.org |
| Rebecca Blisset, High Scope Educational Research Foundation, bblissett@highscope.org |
| Abstract: According to the U.S. Census Bureau (2006-2007) 54% of children between the ages of 3 and 4 are enrolled in school. Furthermore, the positive effects preschool attendance has on young children’s cognitive performance and school readiness is supported (Barnett, Lamy, & Jung, 2005). However, there is a dearth of research examining the different types of programs young children attend and the effects on socioemotional development. Therefore, it is important to explore how the type of early childhood program influences behavioral adjustment and social skills. The present study utilized a longitudinal growth model to examine demographic factors and program type (i.e. federally funded, state funded, or center based program) on children’s behavioral adjustment and social skills across a four year period. Results indicate that gender, parent’s education, food stamp assistance, and program type significantly accounted for the differences in young children’s behavioral adjustment and social skills. |
| Supporting Causal Inferences and Generalization When Evaluating Large-scale, Multi-state, Multi-level Implementations of an Integrated Instructional Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brad Chaney, Westat, bradchaney@westat.com |
| Jane Delgado, College Board, jdelgado@collegeboard.org |
| Doreen Finkelstein, College Board, dfinkelstein@collegeboard.org |
| Joan Michie, Westat, michiej1@westat.com |
| Abstract: In a large-scale, longitudinal, summative evaluation effort, issues of internal validity, external validity, and reliability and were addressed in supporting causal inferences about the impact and efficacy of an integrated instructional program. The program included curriculum, assessment and professional development for both English language arts and mathematics and implementation patterns were very different by subject area, school, district, and state. An interim dataset of more than 1.5 million observations of individual student achievement across multiple years was secured from participating and comparison schools in nine districts and six different states, both before and after program exposure. A repeated measures multi-level modeling approach was utilized in the SAS analyses. Complexities addressed include: using scale scores from different statewide standardized tests as the measure of achievement, determining an effect size when state scale scores differed in type and range, dealing with patterns of missing data, and accounting for varying degrees of exposure at the level of the student and the school. |
| The Unexpected Benefits of a Consumer Survey Team: Factors Relating to Motivation for Employment |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Derek Anderson, University of Colorado Denver, derek.anderson@cudenver.edu |
| Kathryn DeRoche, Mental Health Center of Denver, kathryn.deroche@mhcd.org |
| Abstract: At the Mental Health Center of Denver, we have embraced the use of a “consumer survey team” to administer surveys to other consumers (peer-to-peer data collection), and have noticed an unexpected benefit that members commonly find employment after involvement with the team. The primary goal of the evaluation was to examine attitudes towards employment of consumer survey team members, and to determine if involvement with the team is a model of employment training for individuals struggling with mental health problems. Using a mixed-methods approach, we examined the experiences of team members through semi-structured interviews, in addition to quantitative survey data regarding employment and mental health recovery. The evaluation findings will be discussed focusing on the empowering benefit of involvement, the process of motivation for employment, and characteristics which influence employment motivation. |
| Evaluation Capacity Building: Lessons Learned |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anthony Lee, University of South Carolina, abslee@gwm.sc.edu |
| Ruth Saunders, University of South Carolina, rsaunders@sc.edu |
| Abdoulaye Diedhiou, University of South Carolina, diedhiou@mailbox.sc.edu |
| Abstract: The finding in a needs assessment carried out concluded that the overall evaluation needs of a Chronic Disease Prevention Division in a State Health Department needed capacity building. Over a six-month period, planning ensued involving buy in from various organizational leaders, including funding agencies, recruitment of staff and interested partners among, a host of other planning activities. As a result of the planning, chronic disease prevention staff and partner organizations of a state health department immersed themselves in two evaluation capacity training seminars, over another six-month period. Over the longer term, the goal is to inject and reify the role of evaluation among implementing staff by prescribing evaluation into their daily work life. The findings from two seminar series suggest, in the short term, enhanced confidence and capacity relating to evaluation skills. Larger scale, prospective studies with comparison groups are needed to further investigate the efficacy of this intervention. |
| Process Evaluation of a Homeless Alcohol Harm Reduction Program: A Voice of Participants |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lisa Dirks, Northern Arizona University, ldirks@pire.org |
| Jennifer Norland, Pacific Institute for Research and Evaluation, jnorland@pire.org |
| Knowlton Johnson, Pacific Institute for Research and Evaluation, kjohnson@pire.org |
| Abstract: Evaluations related to harm reduction (HR) models are limited. The overarching goal of HR is to reduce harm associated with addictive behavior (Marlatt and Tapert, 1993). A main tenet of the HR model is the responsibility of participants for their own goal setting. Since a HR approach is fairly individualized, a participant level perspective is appropriate to consider in evaluating such a program. This presentation of participant experience provides context for what it means to be in a program of this type. The results of interviews conducted as part of a process evaluation of a homeless alcohol harm reduction program are discussed. Utilizing the idea that communal narratives and personal stories are resources for empowerment (Rappaport 1995), the interviews gave voice to participants. This is useful in understanding the goal setting process as well as what it means to be in such a program from the perspective of those being served. The perceptions of the actual participants of the program glean an understanding of what may lead to successful participation. |
| A Primer for Extension Program Planning and Evaluation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Joseph L Donaldson, University of Tennessee, jldonaldson@tennessee.edu |
| Abstract: The purpose of the Tennessee Extension Program Planning and Evaluation Model is to provide a straight-forward, yet comprehensive approach to program planning and evaluation that fits the program complexity and organizational structure of Extension. The model’s four dimensions are: issue, plan, deliver, and evaluate. Each dimension is defined by a set of action steps, and the model includes 19 total action steps. This presentation will expand on those action steps and present the research and practice most applicable to planning, implementing and evaluating a high-quality Extension program. This model has been used successfully with University of Tennessee and Tennessee State University Extension agents and specialists for five years. The success of the model is demonstrated by feedback from Extension personnel as well as the marked improvement in USDA-CSREES review summaries of both Annual Accomplishment Reports and Plans of Work over the past five years. |
| A Long-Distance Evaluation of an In-Home Family Services Model: Planning Processes, Successes and Barriers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stephanie Ingram, Boys Town, ingrams@boystown.org |
| Tanya Shaw, Boys Town, shawt@boystown.org |
| Kristin Duppong Hurley, University of Nebraska Lincoln, kdupponghurley2@unl.edu |
| Katy Casey, University of Nebraska Lincoln, katyjcasey@yahoo.com |
| Abstract: Family service interventions are under increasing pressure from funding agencies and clients to demonstrate their effectiveness. Specifically, agencies want to gain a better understanding of what works, for which clients, under what conditions. This difficult task becomes even more complex when conducting a large scale evaluation of at-risk populations in a collaborative research effort, as there are many challenges and processes to plan, including: how to select appropriate measures, how to train staff in complex data collection procedures, how to conduct program fidelity, how to maintain client confidentiality between the service provider and an university research center, and how to monitor data collection, all from 1500 miles away from the intervention site. This paper will discuss the planning processes, challenges and ways to overcome barriers of a long-distance, complex evaluation to help other researchers planning large scale evaluations across the miles. |
| Evaluating the 2nd Year Implementation and Effectiveness of the Kansas Afterschool Leadership Development Initiative to Enhance Afterschool Program Settings |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diane Betzen, Wichita State University, dmbetzen@wichita.edu |
| Tara Gregory, Wichita State University, tara.gregory@wichita.edu |
| Sarah Jolley, Wichita State University, sarah.jolley@wichita.edu |
| Emily Grant, Wichita State University, eagrant@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Debra Renollet, Wichita State University, dlrennolet@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Afterschool programs have long been seen as an appropriate avenue for creating positive youth development. However, there are few models that provide guidance to programs in creating the setting characteristics necessary to positive youth development. For this reason, the Kansas Afterschool Leadership Development Initiative (KALDI) youth leadership capacity-building intervention was implemented as a way to increase organizational effectiveness and promote efforts to achieve positive youth development outcomes through a focus on positive youth-adult relationships, youth-centered norms, and youth participation in decision-making. Results to be presented include the outcomes from measures of KALDI’s effectiveness in its second year, such as surveys of youth and adults in the programs, and observation and interview measures. Insights regarding the successes and challenges of efforts to impact settings through youth leadership capacity-building will be presented. |
| Understanding Service Needs of Older Adults |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Understanding and responding to the needs of older adults is a growing interest due to changing demographics, economic turns, and new ways to interact socially. At the same time it can be difficult to understand older adults’ needs for social interaction and community integration other than at a “surface level”. To overcome this challenge, this evaluation explores the needs of older adults in a Midwest community using a mixed method approach including personal interviews with older adults and key informants, focus groups with older adults and caregivers, and archival data. Through these methods, 200 participants provided feedback on available services for seniors as well as input on what they would like to see offered to older adults in the future. Results of this mixed method approach will be presented, including how they have helped guide a local agency and its partners’ plan for future older adult services. |
| A Fresh Perspective on Evaluating a School-University Partnership |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Steven Thomas, College of Charleston, thomassl@cofc.edu |
| Krysti Webber, College of Charleston, kristina_webber@charleston.k12.sc.us |
| Stephen Jurs, Independent Consultant, sgjurs@charter.net |
| Paula Egelson, College of Charleston, egelsonp@cofc.edu |
| Abstract: This program evaluation report includes information about a partnership effort between the College of Charleston and the Charleston County School District to improve the student outcomes at Burke High School. During the 2007-2008 school year, quantitative and qualitative data were collected to investigate the development of the partnership relationship. Prior evaluation led to several conclusions indicating that stronger relationships have developed between university and school personal. The 2007-2008 report will employ social network analysis to further contextualize the relationships that have developed as a result of this partnership. |
| Student Association of Global Affairs Uses Empowerment Evaluation Concepts, Techniques, and Findings to Foster Self-Determination, Capacity Building and Program Improvement |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Roberta Eleazer, Oklahoma State University, eleazer@gci.net |
| James Eleazer, Oklahoma State University, eleazer@gci.net |
| Don Eskridge, Oklahoma State University, don.eskridge@okstate.edu |
| Kathleen D Kelsey, Oklahoma State University, kathleen.kelsey@okstate.edu |
| Abstract: This poster will examine how graduate students used empowerment evaluation techniques (Fetterman, 2001) to foster sustainable self-determination for the membership of a graduate student organization, the Student Association of Global Affairs (SAGA) at a land-grant university. SAGA is a nine-year old student organization; however, it stagnated and failed to attract member participation at activities and was unable to fill club officer positions. Using a deliberative democratic process (House, 1993), a graduate student team of evaluators coached the executive board members and interested club members in designing an evaluation that resulted in improved program participation and outcomes. The student evaluators surveyed the membership’s opinions, attitudes, beliefs, behaviors, and preferences using an on-line survey. The evaluators involved the board members at every turn to empower stakeholders. Findings, conclusions, and recommendations were developed in conjunction with the board. The student evaluators believe that the evaluation process transformed SAGA into a sustainable organization with a more engaged board and student membership. References: Fetterman, D. M. (2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage Publications. House, E. R. (1993). Professional Evaluation: Social Impact and Political Consequences. Thousand Oaks: Sage. |
| Student Association of Global Affairs Uses Empowerment Evaluation Concepts, Techniques, and Findings to Foster Self-Determination, Capacity Building and Program Improvement |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Roberta Eleazer, Oklahoma State University, eleazer@gci.net |
| James Eleazer, Oklahoma State University, eleazer@gci.net |
| Don Eskridge, Oklahoma State University, don.eskridge@okstate.edu |
| Kathleen D Kelsey, Oklahoma State University, kathleen.kelsey@okstate.edu |
| Abstract: This poster will examine how graduate students used empowerment evaluation techniques (Fetterman, 2001) to foster sustainable self-determination for the membership of a graduate student organization, the Student Association of Global Affairs (SAGA) at a land-grant university. SAGA is a nine-year old student organization; however, it stagnated and failed to attract member participation at activities and was unable to fill club officer positions. Using a deliberative democratic process (House, 1993), a graduate student team of evaluators coached the executive board members and interested club members in designing an evaluation that resulted in improved program participation and outcomes. The student evaluators surveyed the membership’s opinions, attitudes, beliefs, behaviors, and preferences using an on-line survey. The evaluators involved the board members at every turn to empower stakeholders. Findings, conclusions, and recommendations were developed in conjunction with the board. The student evaluators believe that the evaluation process transformed SAGA into a sustainable organization with a more engaged board and student membership. References: Fetterman, D. M. (2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage Publications. House, E. R. (1993). Professional Evaluation: Social Impact and Political Consequences. Thousand Oaks: Sage. |
| Data Acquisition: Considered the Future, but Stuck to the Past |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| John Enger, Nova Southeastern University, enger@bellsouth.net |
| Candace Lacey, Nova Southeastern University, chlacey@bellsouth.net |
| Nydia Cummings, Nova Southeastern University, cnydia@nova.edu |
| Abstract: While moving to online data acquisition to obtain student self-report measures, this presentation shows the problems encountered that resorted in data collection using paper and pencil. Further, in evaluating a comprehensive middle school character education program, two new surveys were created for the project. Their attributes will also be addressed. |
| Student Association of Global Affairs Uses Empowerment Evaluation Concepts, Techniques, and Findings to Foster Self-Determination, Capacity Building and Program Improvement |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Roberta Eleazer, Oklahoma State University, eleazer@gci.net |
| James Eleazer, Oklahoma State University, eleazer@gci.net |
| Don Eskridge, Oklahoma State University, don.eskridge@okstate.edu |
| Kathleen D Kelsey, Oklahoma State University, kathleen.kelsey@okstate.edu |
| Abstract: This poster will examine how graduate students used empowerment evaluation techniques (Fetterman, 2001) to foster sustainable self-determination for the membership of a graduate student organization, the Student Association of Global Affairs (SAGA) at a land-grant university. SAGA is a nine-year old student organization; however, it stagnated and failed to attract member participation at activities and was unable to fill club officer positions. Using a deliberative democratic process (House, 1993), a graduate student team of evaluators coached the executive board members and interested club members in designing an evaluation that resulted in improved program participation and outcomes. The student evaluators surveyed the membership’s opinions, attitudes, beliefs, behaviors, and preferences using an on-line survey. The evaluators involved the board members at every turn to empower stakeholders. Findings, conclusions, and recommendations were developed in conjunction with the board. The student evaluators believe that the evaluation process transformed SAGA into a sustainable organization with a more engaged board and student membership. References: Fetterman, D. M. (2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage Publications. House, E. R. (1993). Professional Evaluation: Social Impact and Political Consequences. Thousand Oaks: Sage. |
| Managing the Values of Key Stakeholders in Health Care Settings |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Awgu Ezechukwu, Western Michigan University, eawgu@yahoo.com |
| Abstract: Stakeholders constitute one element of the political environment or context in which health care operates. The values of key stakeholders consist of such things as their desires, and goals. In the past decade, the numbers of stakeholders in the health care settings have been on the rise. This has increased complexity and diversity of values. The diversity of values have decreased supportiveness and put pressure on the activities and organizational success or goals of health care settings, which consist of things such as clinical quality, profitability, and so forth. This presentation will address these questions: What are values in health care settings?, What are the values of the key stakeholders?, How can values and standards of the key stakeholders can be synthesized into merit and worth?. Additionally, author will discuss the qualitative synthesis method and the stakeholders assessment mapping. |
| Early Childhood Services and Academic Achievement: Shedding Light on the Obstacles to Early School Success |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Eileen Robertson Rehberg, Community Research Council, errehberg@researchcouncil.net |
| Ione Farrar, Community Research Council, ifarrar@researchcouncil.net |
| Abstract: Currently, the State of Tennessee invests in pre-school and Pre-K services for quality child care and education that targets children from low-income families. These services are regarded as a public investment in the future, in which an increased number of children from low-income families are expected to succeed in their early school years. This paper and presentation will discuss the need for broader descriptive and geographically specific baseline data about children and their families at the neighborhood level to better target programs. A case study analysis of neighborhoods in Tennessee’s Hamilton County school district will demonstrate how administrative data from the health department and the school district can be used to assess the variety and diversity of factors that can have an affect on early childhood development and school performance. The case study will also discuss the sustainability of early childhood program effects and the validity of using long-term outcome measures to determine the value of these services. |
| Evaluation of the Chinese Initiatives: Measuring Impact Across Grades, States, and Cultures |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jing Feng, Fordham University, fengjing26@gmail.com |
| Kelly Godfrey, College Board, kgodfrey@collegeboard.org |
| Doreen Finkelstein, College Board, dfinkelstein@collegeboard.org |
| Abstract: Chinese Guest Teacher program and Chinese Bridge Delegation program are two essential programs within the Chinese Initiatives at the College Board. Currently a total of 97 Chinese guest teachers are teaching Chinese in schools and districts across 40 states in the United States. The annual program of Chinese Bridge Delegation was launched in the summer of 2006 and had its second trip to China in June of 2007. The purpose of the evaluation was to inform the program management of the impact of the program and further needs of its participants. This poster highlights the sensitive process of measuring knowledge gain among a group of non-native English speakers as well as measuring impact and further implications for a series of programs designed to inspire considerable reform efforts in a variety of settings. |
| Evaluation of the Chinese Initiatives: Measuring Impact Across Grades, States, and Cultures |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jing Feng, Fordham University, fengjing26@gmail.com |
| Kelly Godfrey, College Board, kgodfrey@collegeboard.org |
| Doreen Finkelstein, College Board, dfinkelstein@collegeboard.org |
| Abstract: Chinese Guest Teacher program and Chinese Bridge Delegation program are two essential programs within the Chinese Initiatives at the College Board. Currently a total of 97 Chinese guest teachers are teaching Chinese in schools and districts across 40 states in the United States. The annual program of Chinese Bridge Delegation was launched in the summer of 2006 and had its second trip to China in June of 2007. The purpose of the evaluation was to inform the program management of the impact of the program and further needs of its participants. This poster highlights the sensitive process of measuring knowledge gain among a group of non-native English speakers as well as measuring impact and further implications for a series of programs designed to inspire considerable reform efforts in a variety of settings. |
| Supporting Causal Inferences and Generalization When Evaluating Large-scale, Multi-state, Multi-level Implementations of an Integrated Instructional Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brad Chaney, Westat, bradchaney@westat.com |
| Jane Delgado, College Board, jdelgado@collegeboard.org |
| Doreen Finkelstein, College Board, dfinkelstein@collegeboard.org |
| Joan Michie, Westat, michiej1@westat.com |
| Abstract: In a large-scale, longitudinal, summative evaluation effort, issues of internal validity, external validity, and reliability and were addressed in supporting causal inferences about the impact and efficacy of an integrated instructional program. The program included curriculum, assessment and professional development for both English language arts and mathematics and implementation patterns were very different by subject area, school, district, and state. An interim dataset of more than 1.5 million observations of individual student achievement across multiple years was secured from participating and comparison schools in nine districts and six different states, both before and after program exposure. A repeated measures multi-level modeling approach was utilized in the SAS analyses. Complexities addressed include: using scale scores from different statewide standardized tests as the measure of achievement, determining an effect size when state scale scores differed in type and range, dealing with patterns of missing data, and accounting for varying degrees of exposure at the level of the student and the school. |
| Evaluating Smaller Learning Communities in United States High Schools |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Linda Bol, Old Dominion University, lbol@odu.edu |
| Shana Pribesh, Old Dominion University, spribesh@odu.edu |
| Christopher Fischer, Old Dominion University, cfischer@nps.k12.va.us |
| John Nunnery, Old Dominion University, jnunnery@odu.edu |
| Abstract: Creating smaller schools/learning communities (SLCs) have been advocated as a specific reform for improving high school student engagement and graduation rates (NRC, 2002). The linkages of smaller learning communities to student achievement have been found to be promising (Felner, Ginter, and Primavera, 1982; NRC, 2002). However, the research connecting the SLC structure with student performance is tenuous – mostly due to methodological issues. We propose to discuss strategies to evaluate the effect of smaller learning communities in U.S. high schools on student achievement and other outcome measures. This poster will use case studies to propose innovative, rigorous designs to yield more valid evaluation findings. In addition, we would identify other constructs (e.g., school climate, student self-concept and motivation) theoretically linked to achievement that could be employed as additional outcome measures. This poster will be useful for practitioners faced with evaluating smaller schools within schools in public school districts. |
| What's Happening to Queer Male Youths in the Math Classroom Once they Come Out? |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| David J Fischer, University of Minnesota, fisch413@umn.edu |
| Abstract: Math is a gatekeeper course in high school. Because of this it is imperative that all students have every opportunity and encouragement to participate in classes beyond Algebra. One group of students who we know almost nothing about in regards to their mathematics experience is queer students, and when they are “out” queer students we know even less. This session will focus on the lived experiences of several “out” queer male high school students in the mathematics classroom. This session is designed to help further the discussion of what is happening to queer students as they come out, and also to encourage others to begin to look at other parts of the curriculum. |
| Examining Time-to-Degree within a Multidisciplinary, Multi-Institutional Graduate Training Program--Successes and Challenges |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Dylan Flather, University of Colorado Boulder, dylan.flather@colorado.edu |
| Kristin Kusmierek, University of Michigan, kkusmierek@stanfordalumni.org |
| Tyler Silverman, University of Colorado Boulder, tsilverman1@gmail.com |
| Mary Anne Carroll, University of Michigan, mcarroll@umich.edu |
| David Karowe, Western Michigan University, karowe@wmich.edu |
| Abstract: Time-to-degree is particularly important in interdisciplinary programs, often thought to add time compared to traditional programs. The Biosphere-Atmosphere Research and Training (BART) program, a project of the National Science Foundation’s Integrative Graduate Education and Research Traineeship (IGERT) initiative, is an interdisciplinary program with participants from over a dozen disciplines and universities. This analysis compares time-to-degree of participants in BART’s two-year supplemental program against the time for their non-BART peers from corresponding home institutions. The preliminary results show that former BART students graduate in about the same time as non-BART students. |
| Local Outcomes from the Rhode Island State Incentive Grant |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| John Stevenson, University of Rhode Island, jsteve@uri.edu |
| Nathan Stein, University of Rhode Island, nstein@mail.uri.edu |
| Israel Beren, University of Rhode Island, chezsruli@mail.uri.edu |
| Paul Florin, University of Rhode Island, pflorin@uri.edu |
| Abstract: This paper looks at outcomes from science-based substance abuse prevention initiatives funded in seven school districts in RI. Despite the development of a state-level policy linking local and state evaluations in a collaborative approach, evidence for local program effects was limited. Changes over time from 6th grade pre-tests to 7th grade post-tests on risk and protective factor measures (MANOVA and Chi-square analyses) were negligible. However, when “high implementation” schools were compared to “low implementation” schools as well as to the statewide picture, there were notably greater reductions in self-reported substance use for 8th graders in high implementation schools. Comparable fidelity of delivery was achieved but youth participation rates were starkly different. Conclusions are drawn about the importance of the delivery system for prevention (curricular insertion vs. after-school) as compared to the particular evidence-based curriculum (All Stars, Project Northland, Life Skills Training) for producing effects with practical significance for a community. |
| Steps for Improving Outcome Measurement of a Family-Centered Poverty Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deborah Frater, University of Michigan, dfrater@umich.edu |
| Sue Ann Savas, University of Michigan, ssavas@umich.edu |
| Ann Kalass, Starfish Family Services, akalass@sfish.org |
| Abstract: As a more effective alternative to public poverty programs, Starfish Family Services of Michigan developed the Family Success Program to move families out of poverty into financial stability. The pilot program identified the following expected outcomes: 200% of the Federal Poverty Level, hope for the future, and one person outside of the family supporting their move to independence. This data was collected through face-to-face interviews and phone surveys at program end and 6 months post-program end. A retrospective pre-test survey was administered to capture missing baseline data. The preliminary outcome results indicated that the program produced a number of changes that were not identified as expected outcomes. This led program stakeholders to a re-conceptualization of the program’s theory of change. With the second implementation, evaluators will utilize standardized outcome measures to assess social support, life skills, coping skills, and parenting skills as well as measures of financial stability. |
| Time Gap Model for the Bombings in Region XII, Mindanao, Philippines |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Epimaco Jr Cabanlit, Mindanao State University, maco_727@yahoo.com |
| Robert Kiunisala, South Cotabato Police Provincial Office, robert_bing@yahoo.com |
| Romeo Jr Galgo, Regional Mobile Group, academy158@yahoo.com |
| Steiljes M Cabanlit, Mindanao State University, steilt_727@yahoo.com |
| Abstract: From the year 2000 to 2006, a total of 96 bombing incidents occurred, 101 bombs exploded, 86 were killed and 674 were injured in Region XII, Mindanao, Philippines. This paper presents a time gap model for these bombings. The mixture of exponential and uniform distribution shows a good fit. |
| A Randomized Experiment Comparing Random to Cutoff-Based Assignment |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rudolfo Galindo, University of California Merced, rgalindo@ucmerced.edu |
| William Shadish, University of California Merced, wshadish@ucmerced.edu |
| Abstract: Several studies have attempted to show that nonrandomized experiments can approximate results from randomized experiments for testing the effects of policies and practices in fields such as education, medicine, public health, job training, and psychology. The methods used in past studies all compared results from randomized and nonrandomized experiments, the latter usually subject to adjustments like econometric selection bias models or propensity score analysis. Several years ago, a laboratory analogue paradigm was developed to improve this design by randomly assigning participants to be in a randomized or nonrandomized experiment in which they could choose their training, and where they were otherwise treated identically. The current design follows that paradigm by randomizing participants into a randomized experiment (RE) or a regression discontinuity design (RDD). Regression discontinuity designs are cut-off based designs which assign participants to a treatment dependent upon a single score on a pre-test. Preliminary results are presented here. |
| Popular Opinion Leader (POL) On the Internet: Adapting an Evidence Based Intervention for MSM Who Seek Sex Online |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sara Gillen, Harlem United Community AIDS Center Inc, sgillen@harlemunited.org |
| George Gates, Harlem United Community AIDS Center Inc, ggates@harlemunited.org |
| Rashi Rohatgi, Harlem United Community AIDS Center Inc, rrohatgi@harlemunited.org |
| Michael Coley, Harlem United Community AIDS Center Inc, mcoley@harlemunited.org |
| Abstract: To address the increasing HIV rates among YMCSM and the need for more prolific prevention efforts, the Popular Opinion Leader (POL) program has been adapted and will be implemented and evaluated by Harlem United Community AIDS Center in New York City. POL is a community-level HIV prevention intervention designed to enlist and train the most influential peers to encourage safer sex norms and behaviors through informal risk-reduction conversations. Objectives of the formative evaluation include identifying service needs and HIV-related risk factors among community members, and innovative ways to properly adapt the POL intervention for the internet. Preliminary, formative findings, identified a community ostracized for engaging in unprotected anal sex, despite simultaneously practicing safer sex; a lack of online sex positive and harm reduction prevention messaging; effective methods for engaging men online (e.g. most popular websites), powerful alternative prevention messaging, and behavioral norms (e.g. lack of discussion concerning unsafe sex practices) |
| Urbanicity, Race/Ethnicity, and School Level Analyses from the Communities In Schools (CIS) National Evaluation: Informing Policy and Practice |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Julie Gdula, ICF International, jgdula@icfi.com |
| Susan Siegel, Communities In Schools Inc, ssiegel@cisnet.org |
| Abstract: Communities In Schools, Inc. (CIS) is a nationwide initiative to connect schools with community resources to help at-risk students successfully learn, stay in school, and prepare for life. As part of a five-year comprehensive national evaluation of CIS, various studies were conducted to determine the impact of the CIS model on school-level outcomes and the extent of program implementation. In addition to the main findings, subgroup analyses were conducted to assess the performance of different types of CIS schools. These subgroups were divided by urbanicity (urban, suburban, rural), race/ethnicity (African American, Hispanic/Latino, white, and diverse), and school level (elementary, middle, high). To better understand the context of the findings, profiles of each subgroup were created based on survey responses regarding such domains as planning, needs assessment, service delivery, and monitoring. Results are presented as they pertain to the different challenges of implementing the CIS model to serve different student subgroups. |
| Theory to Practice to Theory: Completing the Evaluation Cycle through Project-Based Learning |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deborah Bennett, Purdue University, bennett@purdue.edu |
| Susan Geier, Purdue University, geiers@purdue.edu |
| Abstract: Professionals in Higher Education are frequently unprepared to address the evaluation requirements of their positions. This is particularly concerning given the heightened attention to program accountability. These professionals also have little experience linking the theoretical models of their discipline to the programs they develop and serve. This poster describes a project-based approach to teaching program development and evaluation at the graduate level. Two student projects will be shared illustrating how theory can be linked to practice and how well designed and executed evaluation can help inform, refine, and expand theory. |
| Violence Against Women on the College Campus: The Impact of Evaluation on Service Delivery and Prevention Programming |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Roberta Gibbons, University of Minnesota, gibbo005@umn.edu |
| Abstract: In response to the high incidence rate of violence against women on college campuses, The U.S. Department of Justice has awarded over $100 million to more than one-hundred universities to develop violence intervention and prevention programs. Although funded programs are required to participate in the government-sponsored “measuring effectiveness” initiative, there remain questions about the outcomes and quality of the individual programs. This poster presentation will focus on the impact of local program evaluations on the violence prevention and intervention programs funded under the Grants to Reduce Violence Against Women on Campus. The complexity of assessing the success of violence prevention and intervention effort is well-documented (Burt & Zweig, Edleson, Sullivan, Ward) as is the complexity of understanding use/influence (Alkin & Taut, Mark & Henry, Patton, Weiss). This poster brings these literatures together to help frame the question: How does evaluation impact campus-based violence prevention and intervention programs? |
| Tag Team Evaluation as Community Outreach: Lessons Learned From the Field |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Laura Pejsa, University of Minnesota, pejs0001@umn.edu |
| Roberta Gibbons, University of Minnesota, gibbo005@sossgw.stu.umn.edu |
| Abstract: The proposed poster will detail the process of evaluating a community-based vocational training program for pregnant and parenting teens. There are several dynamics that are unique to this project: 1) The evaluation was conducted by two graduate students; 2) The evaluation was a “tag-team” effort, with one student planning the evaluation the other carrying it out; and 3) Although the primary intended user of the evaluation was the director of the program, the evaluation was sponsored by a University of Minnesota community outreach center. This presentation will explore barriers to evaluation implementation and lessons learned throughout the process from three perspectives: 1) the authors; 2) the University outreach program; and 3) the primary intended user. In particular, we will discuss how evaluation use prevailed despite numerous structural challenges of the internship and evaluation plan. We will also show how several evaluation concepts learned from coursework were applicable in the case. |
| Popular Opinion Leader (POL) On the Internet: Adapting an Evidence Based Intervention for MSM Who Seek Sex Online |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sara Gillen, Harlem United Community AIDS Center Inc, sgillen@harlemunited.org |
| George Gates, Harlem United Community AIDS Center Inc, ggates@harlemunited.org |
| Rashi Rohatgi, Harlem United Community AIDS Center Inc, rrohatgi@harlemunited.org |
| Michael Coley, Harlem United Community AIDS Center Inc, mcoley@harlemunited.org |
| Abstract: To address the increasing HIV rates among YMCSM and the need for more prolific prevention efforts, the Popular Opinion Leader (POL) program has been adapted and will be implemented and evaluated by Harlem United Community AIDS Center in New York City. POL is a community-level HIV prevention intervention designed to enlist and train the most influential peers to encourage safer sex norms and behaviors through informal risk-reduction conversations. Objectives of the formative evaluation include identifying service needs and HIV-related risk factors among community members, and innovative ways to properly adapt the POL intervention for the internet. Preliminary, formative findings, identified a community ostracized for engaging in unprotected anal sex, despite simultaneously practicing safer sex; a lack of online sex positive and harm reduction prevention messaging; effective methods for engaging men online (e.g. most popular websites), powerful alternative prevention messaging, and behavioral norms (e.g. lack of discussion concerning unsafe sex practices) |
| Planning and Policy in the Interpretation and Presentation of Outcomes from Randomized Controlled Trials of Community-Advocacy Programs |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fred Glantz, Kokopelli Associates LLC, fred@kokopelliassociates.com |
| Kim Sprague, Brown University, kim_sprague@brown.edu |
| Pamela Raya-Carlton, RMC Research Corporation, prayacarlton@rmccorp.com |
| Diane Schilder, Evaluation Analysis Solutions, dschilder@eval-inc.com |
| Abstract: These studies are considered critical by the department to contribute to the field by addressing the primary research question “what interventions are effective?” In addition, these studies are also some of the first to address what methods should be employed in such RCT trials, what is required in terms of policy for implementation and collaboration to conduct such efforts, and what is required to appropriately assess their implementation. Larger-scale RCTs are newer to the field of education and the purposes and methods of such studies do not appear widely understood. The goal of this paper is to present more information regarding the benefits of such studies in the context of their complications. |
| Evaluation of the Chinese Initiatives: Measuring Impact Across Grades, States, and Cultures |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jing Feng, Fordham University, fengjing26@gmail.com |
| Kelly Godfrey, College Board, kgodfrey@collegeboard.org |
| Doreen Finkelstein, College Board, dfinkelstein@collegeboard.org |
| Abstract: Chinese Guest Teacher program and Chinese Bridge Delegation program are two essential programs within the Chinese Initiatives at the College Board. Currently a total of 97 Chinese guest teachers are teaching Chinese in schools and districts across 40 states in the United States. The annual program of Chinese Bridge Delegation was launched in the summer of 2006 and had its second trip to China in June of 2007. The purpose of the evaluation was to inform the program management of the impact of the program and further needs of its participants. This poster highlights the sensitive process of measuring knowledge gain among a group of non-native English speakers as well as measuring impact and further implications for a series of programs designed to inspire considerable reform efforts in a variety of settings. |
| Community Needs Assessment for a Northern Sudanese Community in an Eastern Iowa Town |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Douglas Grane, University of Iowa, douglas.grane@gmail.com |
| Kirsten Beyer, University of Iowa, kirsten-beyer@uiowa.edu |
| Abstract: Immigrant and refugee communities face various struggles in the United States ranging from acculturation, access to social and health services, reinventing a community identity in the U.S., and employment and self-sufficiency challenges. Immigrants and refugees from northern Sudan intentionally founded a community in Iowa City, Iowa, after many in the community lived in other parts of the U.S. and other countries outside Sudan. We worked with the community to assess community needs through participant observation, Participatory Geographic Information Systems (PGIS) mapping, focus groups, and several in depth interviews. The needs assessment found that community leadership successfully mobilized the community in meeting most basic needs; the community expressed that their most significant challenge centered around highly educated community members working in occupations far below their educational status. |
| Evaluating the 2nd Year Implementation and Effectiveness of the Kansas Afterschool Leadership Development Initiative to Enhance Afterschool Program Settings |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diane Betzen, Wichita State University, dmbetzen@wichita.edu |
| Tara Gregory, Wichita State University, tara.gregory@wichita.edu |
| Sarah Jolley, Wichita State University, sarah.jolley@wichita.edu |
| Emily Grant, Wichita State University, eagrant@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Debra Renollet, Wichita State University, dlrennolet@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Afterschool programs have long been seen as an appropriate avenue for creating positive youth development. However, there are few models that provide guidance to programs in creating the setting characteristics necessary to positive youth development. For this reason, the Kansas Afterschool Leadership Development Initiative (KALDI) youth leadership capacity-building intervention was implemented as a way to increase organizational effectiveness and promote efforts to achieve positive youth development outcomes through a focus on positive youth-adult relationships, youth-centered norms, and youth participation in decision-making. Results to be presented include the outcomes from measures of KALDI’s effectiveness in its second year, such as surveys of youth and adults in the programs, and observation and interview measures. Insights regarding the successes and challenges of efforts to impact settings through youth leadership capacity-building will be presented. |
| Minimization as an Alternative to Unrestricted Randomization in Educational Research |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Hui-Fang Chen, University of Denver, hchen25@du.edu |
| Kathy Green, University of Colorado Denver, kgreen@du.edu |
| Abstract: This paper describes an alternative to unrestricted randomization in experimental design, minimization, which can be used to achieve better balance of critical factors in small to medium-sized experimental studies. With minimization, several critical factors are controlled which are known to influence outcomes but which are not the foci of the study. Based on the critical factor values of previously entered participants, a new participant will be allocated into an experimental group to minimize imbalance across groups in critical factors. Minimization has been primarily employed in medical interventions and clinical trials. This article proposes examples of the potential use of minimization in education, its advantages, and limitations. |
| Broadening Participation of Minority Professionals in Program Evaluation: A Concept Mapping Workshop as Strategy |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ruth Greene, Johnson C Smith University, rgreene@jcsu.edu |
| Frank Parker, Johnson C Smith University, fparker@jcsu.edu |
| Abstract: This poster focuses on the use of concept mapping as a collaborative planning and evaluation process to engage faculty and administrators in effective collaborative efforts . The poster details the framework that was used to guide the process, outlines the major methodological issues and problems and recommends ways in which the tool can be used for training, increasing diversity in the evaluation pipeline and enhancing evaluation and planning in higher education. |
| Improving Individual and Collective Performance through Self-Assessment in a Multi-site Research Network |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sarah Greene, Group Health Center for Health Studies, greene.sm@ghc.org |
| Abstract: Collaborative research networks are used increasingly to study health and improve healthcare. Optimizing the collaborative process helps ensure not only technical quality of the research, but satisfaction of individual collaborators and durability of the partnership. Internal evaluation is vital for research networks seeking to identify strengths and deficient aspects of the collaboration. Our interdisciplinary partnership, the Cancer Research Network, conducts annual surveys of its participants to determine effectiveness of leadership, communication, and operations. The Cancer Research Network is modeled on features of healthcare alliances described by Kaluzny and Warnecke. The Partnership Self-Assessment Tool, coupled with internally-developed questions to measure the impact of our research, form the foundation of our evaluative strategy. Each year, we make changes Network-wide derived from evaluation findings. This evaluation helps uncover both individual- and group-wide opportunities for improvement. We will present high-level observations and implications for other research partnerships who desire to systematically assess their performance. |
| Evaluating the 2nd Year Implementation and Effectiveness of the Kansas Afterschool Leadership Development Initiative to Enhance Afterschool Program Settings |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diane Betzen, Wichita State University, dmbetzen@wichita.edu |
| Tara Gregory, Wichita State University, tara.gregory@wichita.edu |
| Sarah Jolley, Wichita State University, sarah.jolley@wichita.edu |
| Emily Grant, Wichita State University, eagrant@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Debra Renollet, Wichita State University, dlrennolet@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Afterschool programs have long been seen as an appropriate avenue for creating positive youth development. However, there are few models that provide guidance to programs in creating the setting characteristics necessary to positive youth development. For this reason, the Kansas Afterschool Leadership Development Initiative (KALDI) youth leadership capacity-building intervention was implemented as a way to increase organizational effectiveness and promote efforts to achieve positive youth development outcomes through a focus on positive youth-adult relationships, youth-centered norms, and youth participation in decision-making. Results to be presented include the outcomes from measures of KALDI’s effectiveness in its second year, such as surveys of youth and adults in the programs, and observation and interview measures. Insights regarding the successes and challenges of efforts to impact settings through youth leadership capacity-building will be presented. |
| Context, Culture and Change: Examining Racial and Ethnic Health Disparities Through Qualitative Evaluation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pattie Tucker, Centers for Disease Control and Prevention, ptucker1@cdc.gov |
| C Shannon Griffin Blake, Centers for Disease Control and Prevention, sgriffinblake@cdc.gov |
| Amanda Navarro, Centers for Disease Control and Prevention, anavarro2@cdc.gov |
| Shawna Howell, Centers for Disease Control and Prevention, showell@cdc.gov |
| Abstract: In 1999, the Centers for Disease Control and Prevention initiated Racial and Ethnic Approaches to Community Health (REACH) 2010 as the cornerstone of its efforts to eliminate racial and ethnic health disparities. REACH aimed to examine and address individual and environmental contexts which included differences across race, ethnicity, education, income, and geographic location. The uniqueness of REACH was its diversity in approaches and strategies and its emphasis on community participation in all aspects of program development, implementation and evaluation. Qualitative methods were utilized to elucidate the three themes presented in this poster: (1) understanding the broader environmental contexts impacting disparities, (2) recognition of culture as an important factor of influence on coalitions, and (3) documenting change across strategies to eliminate disparities. This presentation of four different projects will provide a qualitative framework for understanding the problem and identifying key elements that ensure program sustainability and achieve greater public health impact. |
| Building Monitoring and Evaluation Capacity in Developing Countries: Case of Armenia |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristine Grigoryan, Social Protection System Strengthening Project, kgrig@spss.am |
| Abstract: Governments often know where they want to go but don’t know how to get there. The Armenian government has recognized its need to build a monitoring and evaluation capacity as a step forward in implementing economic and social change. Our project, the Social Protection System Strengthening Project, has had an interesting experience of building monitoring and evaluation capacity within the Ministry of Labour and Social Issues of the Republic of Armenia. We introduced two main tools for monitoring and evaluation capacity building: developing a monitoring plan based on the logical model framework for sixty projects and introduction of the best international experiences in monitoring and evaluation theory and practices. In parallel, we used several pilot projects as excellent demonstration exercises for designing monitoring and evaluation implementation plans. Consequently, the first step toward success has been introduced. Now there is a need for follow up experiences to sustain the existing capacity. |
| One Step at a Time: Building Mutual Respect through Participatory Evaluation of an Arts-Based Program for Youth |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jean Scandlyn, University of Colorado Denver, jean.scandlyn@cudenver.edu |
| Heidi Grove, Urban Peak, heidi@soxplace.org |
| Abstract: This poster critically analyzes the process of creating a participatory program evaluation of a drop-in, arts-based program for inner city youth ages 14-24. Founded by young people on the principles of hip hop culture to serve as a gang neutral space, the organization does not require young people to be members, sign in, or provide personal information to participate in its activities. While recognizing the need for evaluation, the program’s director was wary that evaluation activities might be viewed as disrespectful or invasive by young people who are disempowered in the larger society. A participatory approach was essential to building trust in the evaluation process and to respect the independence, security, and self-worth of the participants and staff. By moving slowly over a three-year period, the evaluator, staff, and participants built mutual respect, increased capacity and competence in evaluation within the agency, and created a basis for on-going evaluation. |
| Results of a Baseline Survey of Barriers and Facilitators of Clinical Translational Research |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fredric Wolf, University of Washington, wolf@u.washington.edu |
| Douglas Brock, University of Washington, dmbrock@u.washington.edu |
| Lynne Robins, University of Washington, lynner@u.washington.edu |
| Pamela Nagasawa, University of Washington, pnaga22@u.washington.edu |
| Abstract: Barriers to the conduct of clinical and translational research were identified as part of a mixed-methods evaluation of a newly created Institute of Translational Health Sciences (ITHS). The ITHS was created as part of the National Institute of Health’s Roadmap Clinical and Translational Sciences Award program. Ten barriers identified in an earlier open-ended survey were rated in terms of 1) current level of satisfaction and 2) degree to which each represented a barrier to research and were then 3) “prioritized” in terms of urgency for addressing each (Not a priority, > 10 years, 5-10 years, 1-5 years, Year 1, Immediate). Modal responses of 215 researchers indicated an “Immediate” need to address two areas: Institutional Review Board (IRB) operations and need for more local grant support to gather preliminary data to prepare larger grant applications. All other areas received modal responses suggesting they be addressed within five years. |
| Using Evaluation as a Reflective Tool to Sustain Changes When Schools Reform |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sharon Brown, Cleveland State University, s.a.brown54@csuohio.edu |
| James Salzman, Cleveland State University, j.salzman@csuohio.edu |
| Abstract: Even in effective school change efforts, the results are often short lived (Tyack, 1995). Continuance of school change beyond the timeline of the funded grant is challenging and a process that is often left to chance (Fullan, 2001). In Ohio, school personnel were trained in the use of a tool to help them maintain fidelity to three program standards for Reading First by identifying relevant artifacts that represented their implementation, and independent consultants used this tool three times/year to monitor fidelity. By training stakeholders in document selection using principles similar to auditing (relevance, reliability, sufficiency, representativness and timeliness) and guiding participants in programmatic conversations, the process of document review proved to be a powerful, cost-effective tool for collecting implementation data and building organizational capacity for program sustainability. This paper is a case study of one district’s use of monitoring program fidelity to sustain their school reform efforts. |
| Critical Success Factors in Implementing a Program Evaluation for a Scientific Research Center of Excellence Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rajni Samavedam, Booz Allen Hamilton, samavedam_rajni@bah.com |
| Elaine Brenner, Booz Allen Hamilton, brenner_elaine@bah.com |
| Abstract: In conducting a process and outcome evaluation study for a Federally-funded scientific research program, the evaluation team identified four critical success factors that contribute to the success in conducting the evaluation: creating a blended evaluation team; using an expert panel; striving for creative solutions for defining the right comparison group; and having flexibility in approach as information is learned. |
| Evaluation Capacity Building: Lessons Learned |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anthony Lee, University of South Carolina, abslee@gwm.sc.edu |
| Ruth Saunders, University of South Carolina, rsaunders@sc.edu |
| Abdoulaye Diedhiou, University of South Carolina, diedhiou@mailbox.sc.edu |
| Abstract: The finding in a needs assessment carried out concluded that the overall evaluation needs of a Chronic Disease Prevention Division in a State Health Department needed capacity building. Over a six-month period, planning ensued involving buy in from various organizational leaders, including funding agencies, recruitment of staff and interested partners among, a host of other planning activities. As a result of the planning, chronic disease prevention staff and partner organizations of a state health department immersed themselves in two evaluation capacity training seminars, over another six-month period. Over the longer term, the goal is to inject and reify the role of evaluation among implementing staff by prescribing evaluation into their daily work life. The findings from two seminar series suggest, in the short term, enhanced confidence and capacity relating to evaluation skills. Larger scale, prospective studies with comparison groups are needed to further investigate the efficacy of this intervention. |
| Steps for Improving Outcome Measurement of a Family-Centered Poverty Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deborah Frater, University of Michigan, dfrater@umich.edu |
| Sue Ann Savas, University of Michigan, ssavas@umich.edu |
| Ann Kalass, Starfish Family Services, akalass@sfish.org |
| Abstract: As a more effective alternative to public poverty programs, Starfish Family Services of Michigan developed the Family Success Program to move families out of poverty into financial stability. The pilot program identified the following expected outcomes: 200% of the Federal Poverty Level, hope for the future, and one person outside of the family supporting their move to independence. This data was collected through face-to-face interviews and phone surveys at program end and 6 months post-program end. A retrospective pre-test survey was administered to capture missing baseline data. The preliminary outcome results indicated that the program produced a number of changes that were not identified as expected outcomes. This led program stakeholders to a re-conceptualization of the program’s theory of change. With the second implementation, evaluators will utilize standardized outcome measures to assess social support, life skills, coping skills, and parenting skills as well as measures of financial stability. |
| One Step at a Time: Building Mutual Respect through Participatory Evaluation of an Arts-Based Program for Youth |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jean Scandlyn, University of Colorado Denver, jean.scandlyn@cudenver.edu |
| Heidi Grove, Urban Peak, heidi@soxplace.org |
| Abstract: This poster critically analyzes the process of creating a participatory program evaluation of a drop-in, arts-based program for inner city youth ages 14-24. Founded by young people on the principles of hip hop culture to serve as a gang neutral space, the organization does not require young people to be members, sign in, or provide personal information to participate in its activities. While recognizing the need for evaluation, the program’s director was wary that evaluation activities might be viewed as disrespectful or invasive by young people who are disempowered in the larger society. A participatory approach was essential to building trust in the evaluation process and to respect the independence, security, and self-worth of the participants and staff. By moving slowly over a three-year period, the evaluator, staff, and participants built mutual respect, increased capacity and competence in evaluation within the agency, and created a basis for on-going evaluation. |
| Strengthening the Evaluation Capacity of the National Institutes of Health |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brandie Taylor, National Institutes of Health, taylorbr@mail.nih.gov |
| Rosanna Ng, National Institutes of Health, ngr@mail.nih.gov |
| Meryl Sufian, National Institutes of Health, sufianm@mail.nih.gov |
| Madeleine Wallace, National Institutes of Health, wallacem2@mail.nih.gov |
| Yuanting Zhang, National Institutes of Health, zhangyuant@mail.nih.gov |
| Abstract: A comprehensive approach has been developed to strengthen the evaluation capacity of the National Institutes of Health (NIH). Led by the Evaluation Branch (EB), a part of the NIH Office of Portfolio Analysis and Strategic Initiatives, efforts are being implemented to build and sustain support for evaluation and evaluation capacity. This poster will showcase these activities (e.g., strategies to maximize the effective use of dedicated funding for evaluations, education and outreach activities, technical support and guidance, collaborative efforts to foster evaluation skills and knowledge) and highlight some of the challenges and lessons learned. |
| Sense of Belongingness and Student Retention in a New Residential STEM School: The Case of the Tennessee Governor’s Academy for Mathematics and Science |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amy Sullins, Tennessee Governor's Academy for Mathematics and Science, acsullins@gmail.com |
| Abstract: In its second year of operation, the Tennessee Governor’s Academy for Mathematics and Science (TGA) is a residential program for talented eleventh and twelfth graders, enrolling 50 students. Significant resources have been amassed to provide a rigorous academic curriculum for the students, encouraging them to major in STEM (science, technology, engineering, or mathematics) fields and become part of the STEM professional community in Tennessee. As with any pioneering educational program, student retention is key to program success and continuance, as students’ outcome and impact are being evaluated. In TGA’s first academic year, five students (of 24) left the program, and during selection for the 2008-2009 junior class, a personality trait instrument was administered to learn if connections exist between Big Five Personality Traits and other demographic variables and sense of belongingness and engagement in school. In-process results of this mixed methods, longitudinal hierarchical multiple regression study will be reported. |
| A Utilization-Focused Evaluation of a Training Program for Early Intervention Service Providers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Barbara Stroud, Los Angeles Child Guidance Clinic, bstroud@lacgc.org |
| Rupanwita Gupta, Claremont Graduate University, rupanwita@gmail.com |
| Tony Wu, Los Angeles Child Guidance Clinic, tonyw@lacgc.org |
| Abstract: This paper discusses the evaluation of a training program for professionals in early intervention (0-5 mental health services) conducted by a child guidance clinic. Trainings and concurrent supervisions were intended to equip participants to serve clients (0-5) and their families effectively. Surveys elicited quantitative and qualitative feedback to learn about participants’ perceptions of the workshops and utilization of presented topics. Follow-up telephone interviews with eight participants were completed to understand the influence of the training program on changes in clinical practice. The trainings and trainers were rated highly on several aspects. Additionally, phone interview participants mentioned that training topics had informed them of factors to pay attention to when assisting clients, while supervision sessions had provided additional support. Noteworthy are this evaluation's implications, given the dearth of ongoing professional development for early intervention professionals. |
| Evaluation of a Reading Intervention Program for Struggling Middle and High School Readers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Debbie Hahs-Vaughn, University of Central Florida, dhahs@mail.ucf.edu |
| Mary Little, University of Central Florida, mlittle@mail.ucf.edu |
| Abstract: Multisite evaluation theory is often understood in the context of federally funded programs (Lawrenz & Huffman, 2003). There is very little published, however, on how multisite evaluation theory can be applied to intervention research in schools when the school district acts as the federal agency and the individual schools implement the program. This paper will add to the knowledge base of multisite evaluation in the context of school settings. The purpose of this session is to share the following: a) the evaluation approach for this large-scale multi-site reading intervention implemented in a large school district in south Florida; b) "lessons learned" for evaluating implementation of interventions in schools; and c) some of the results of the evaluation of the reading intervention program. |
| A Comprehensive Evaluation of the Active Living by Design Initiative: Innovative Approaches to Increase Physical Activity |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Melissa Hall, Transtria LLC, melissa@transtria.com |
| Laura Brennan Ramirez, Transtria LLC, laura@transtria.com |
| Cheryl Kelly, Saint Louis University, kellycm@slu.edu |
| Abstract: The Active Living by Design (ALbD) initiative establishes innovative approaches to increase physical activity through community design, public policies and communications strategies. Evaluation efforts focus on the ALbD Community Action Model components: preparation, promotions, programs, physical projects, and policies. This evaluation is intended to capture the range of grantee impacts and the strengths and challenges of implementing the ALbD interventions at the local level while focusing on three primary aims: 1) to assess the environmental impacts of physical projects and associated policy changes, 2) to document the range of interventions and accomplishments, and 3) to identify strengths and challenges in the process of planning, developing, and implementing the interventions. Visits to each of the 25 grantee sites will be completed by November 2008. This poster will highlight the methods used in the evaluation as well as findings and recommendations for building community design initiatives to increase physical activity. |
| The Evaluation Experiences and Needs of County 4-H Educators |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristi Lekies, The Ohio State University, lekies.1@osu.edu |
| Amanda Hamiel Bennett, The Ohio State University, hamiel.3@osu.edu |
| Abstract: Extension 4-H educators are expected to document impacts of their programs on the youth they serve. A web-based survey was conducted with county 4-H educators in one state, Ohio (n=62), to examine their experiences, attitudes, knowledge, and skills regarding evaluation. Also assessed were barriers to conducting evaluation and beneficial supports. Results indicated an emphasis on outcome evaluations, limited variation in designs and methods, overall positive attitudes, and a moderate degree of confidence in conducting evaluations. Time to conduct evaluations and complete Institutional Review Board (IRB) processes were the greatest perceived barriers. Assistance from colleagues, state specialists, inservice trainings, and small group meetings were seen as the most beneficial in improving skills. Some variation by length of employment and job classification was noted for attitudes. Implications include broader involvement of youth in the evaluation process, increased use of logic models, and assistance with statistical analysis, web-based surveys, and publishing. |
| Evaluating the Impact of a Three-Hour Neuroscience of Addiction Curriculum on College Students’ Knowledge and Attitudes: Results From the National Institute on Drug Abuse Enters College Project |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Joyce Hartje, University of Nevada Reno, jhartje@casat.org |
| Michelle Berry, University of Nevada Reno, mberry@casat.org |
| Nancy Roget, University of Nevada Reno, nroget@casat.org |
| Wendy Woods, University of Nevada Reno, wwoods@casat.org |
| Angela Broadus, University of Nevada Reno, abroadus@casat.org |
| Abstract: The purpose of this project was to evaluate the impact of a 3-hour science of addiction research-based curriculum infusion on university undergraduate students’ knowledge and attitudes about substance use disorders. Two sections of existing introductory courses in three pre-professional disciplines (criminal justice, nursing, and social work) were selected. One section served as the curriculum infusion (implementation) group (N = 122); the other as the control (non-implementation) group (N = 178). Pre- and post-test measures were developed based on curriculum objectives to assess the effect on knowledge and attitudes. Results showed a significant increase in knowledge about addiction and a corresponding decrease in stigmatizing attitudes in 1) the implementation vs. non-implementation groups; and 2) the implementation group pretest vs. post-test measures. These findings lend support to the potential effectiveness of infusing a brief research-based addiction curriculum into existing undergraduate courses and could have long-term implications for preparing helping professionals. |
| Evaluation of the Guide to Clinical Preventive Services: Applying a Multi-Method Approach to the Assessment of a Preventive Medicine Product |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jacqueline King, Johnson, Bassin and Shaw International Inc, jking@jbsinternational.com |
| Julie Nicholson, Johnson, Bassin and Shaw International Inc, jnicholson@jbsinternational.com |
| Lisa Kretz, Johnson, Bassin and Shaw International Inc, lkretz@jbsinternational.com |
| Susan Hayashi, Johnson, Bassin and Shaw International Inc, shayashi@jbsinternational.com |
| Abstract: This evaluation assessed the content, user-friendliness, and usefulness of The Guide to Clinical Preventive Services, an evidence-based product developed and disseminated by the Agency for Healthcare Research and Quality (AHRQ). Three versions of the Guide were evaluated in this project: the printed guide as a handbook, the Preventive Services website, and an electronic version available for download to PDA. A multi-method approach employed both email surveys and focus groups to assess the Guide’s’ use and usefulness among primary care providers (e.g., physicians, physician assistants, and nurse practitioners). Participants were recruited from those individuals who ordered the Guide online or by telephone and who agreed to participate in the study by completing emailed surveys. Focus groups were conducted at selected medical sites and regional conferences in order to gather feedback from primary care providers. Multivariate analyses were used to analyze 1,260 email survey responses, and content analysis was used to assess focus group data. |
| Evaluation Stew: Multiple Roles of the Evaluator in Education |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chris Haynes, Oklahoma State University, chris.haynes@okstate.edu |
| Tim Kock, Oklahoma State University, tim.kock@okstate.edu |
| Kathleen D Kelsey, Oklahoma State University, kathleen.kelsey@okstate.edu |
| Abstract: This poster will serve as a vehicle to outline an evaluation of AGED 4713 International Programs in Agricultural Education and Extension at Oklahoma State University. Traditionally this course has been taught by tenured faculty members at this institution, and until recently held an enrollment of more than 100 students. Multiple changes in the department have necessitated that this course be assigned to a graduate student working on a Ph.D. in the area of Extension Education and International Development. The focus of this evaluation is to determine if the quality and rigor of the course is being upheld. The evaluation will focus on the effectiveness of the instructors’ teaching methods and how conducive it is to student learning, if the course objectives are being met, and what the actual student perceptions are of the course. Data will be determined through historical documentation, peer assessment, and through student focus group discussion. |
| A Logic Model Framework for Evaluation of a National Workforce Development Endeavor |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pamela Bishop, Oak Ridge Institute for Science and Education, pbaird@utk.edu |
| Samuel Held, Oak Ridge Institute for Science and Education, sam.held@orau.org |
| Abstract: In response to the Academic Competitiveness Council initiative, existing federally-funded education programs are developing or increasing their current evaluation capacities. Because many government programs are national in scope, ad hoc evaluators are faced with the difficult task of aligning evaluation plans across multiple levels and sites. This poster presentation provides a unique approach to the development of a nationally aligned evaluation of a multisite, multilevel, multiprogam workforce development enterprise using a logic model framework. The scope of the evaluation includes both the evaluation of five federally-funded, multisite science education programs, and a metaevaluation of common enterprise outcomes and impacts of all programs. The proposed evaluation framework aligns resources and outcomes at the local and national levels, so that the evaluation at all levels remains focused on the same fundamental goals for participants, programs, and the enterprise as a whole. |
| Culturally Responsive Early Education: Hispanic Early Childhood Education and Pacific Island PreK Boost Projects |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| JoAnn Yuen, University of Hawaii Manoa, joyuen@hawaii.edu |
| Arthur Hernandez, University of Texas San Antonio, art.hernandez@utsa.edu |
| Abstract: Shaped by federal and state funding, early education programs target the most at-risk: underserved families and those unable to afford or access early education. These families are more likely to be heterogeneous and early education programs and evaluations will need to respond to diversity. Evaluation research is poised to inform, clarify the vision and shape early childhood education policy. This panel will contrast two different early childhood programs and research evaluations: 1. Early education and health curriculum; and, 2. PreK 20-day learning behavior and literacy transition. The activities associated with the 'Healthy and Ready to Learn' and 'PreK-ACE Boost' projects were designed in a manner intended to ensure cultural appropriateness (Hispanic and Pacific Islanders) and used an "evaluation" rubric to guide program development and assess fidelity. Both programs are culturally responsive and effective in preparing young children and families for school. |
| Evaluating the Impact of a Constructivist Design Conference for Education Professionals |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Hibbard, University of South Florida, hibbard@coedu.usf.edu |
| Abstract: This paper discusses experiences with evaluating the impact of a Constructivist Design Conference for education professionals. A collaborative evaluation approach was used to actively engage conference participants and stakeholders throughout the evaluation process. The Model for Collaborative Evaluators (MCE) was used to guide the evaluation (Rodríguez-Campos, 2005). Conference observations, interviews, focus groups, and electronic surveys were used to gather data. Participants included teachers, school administrators, and teacher educators from diverse schools across the United States and worldwide. Practices for evaluating alternative programs, and collaborating across multi-sites and states will be discussed. Benefits of using a collaborative approach with education professionals will also be addressed. Lessons learned will be shared for future policy and practice. Rodríguez-Campos, L. (2005). Collaborative evaluations: A step-by-step model for the evaluator. Tamarac, FL: Llumina Press. |
| The Good, The Bad, and The Ugly: Outsourcing Experiences |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristin Pankey, Southern Illinois University at Carbondale, knpankey@siu.edu |
| Joel Nadler, Southern Illinois University at Carbondale, jnadler@siu.edu |
| Nicole Cundiff, Southern Illinois University at Carbondale, karim@siu.edu |
| Nicholas G Hoffman, Southern Illinois University at Carbondale, nghoff@siu.edu |
| Gargi Bhattacharya, Southern Illinois University at Carbondale, gargi@siu.edu |
| Alen Avdic, Southern Illinois University at Carbondale, alen@siu.edu |
| Abstract: The Applied Psychology program at the Southern Illinois University at Carbondale (SIUC) has been providing training in program evaluation since 1982. Perhaps the most unique part of this training is the opportunity to conduct professional evaluations as a part of Applied Research Consultants (ARC), a student-run consulting firm. Over the past few years, ARC has been hired for an increasing amount of outsourcing services and has had a variety of experiences with each of these projects and evaluations. Three specific situations stand out and can be described as “The Good, The Bad, and The Ugly.” A good experience is defined as a project that allows for the development of skills and abilities. “Bad” and “Ugly” experiences are profitable but with low educational value. This presentation will discuss these three situations, ARC’s experiences and relationships with these clients, and tips for dealing with outsourcing. |
| Context, Culture and Change: Examining Racial and Ethnic Health Disparities Through Qualitative Evaluation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pattie Tucker, Centers for Disease Control and Prevention, ptucker1@cdc.gov |
| C Shannon Griffin Blake, Centers for Disease Control and Prevention, sgriffinblake@cdc.gov |
| Amanda Navarro, Centers for Disease Control and Prevention, anavarro2@cdc.gov |
| Shawna Howell, Centers for Disease Control and Prevention, showell@cdc.gov |
| Abstract: In 1999, the Centers for Disease Control and Prevention initiated Racial and Ethnic Approaches to Community Health (REACH) 2010 as the cornerstone of its efforts to eliminate racial and ethnic health disparities. REACH aimed to examine and address individual and environmental contexts which included differences across race, ethnicity, education, income, and geographic location. The uniqueness of REACH was its diversity in approaches and strategies and its emphasis on community participation in all aspects of program development, implementation and evaluation. Qualitative methods were utilized to elucidate the three themes presented in this poster: (1) understanding the broader environmental contexts impacting disparities, (2) recognition of culture as an important factor of influence on coalitions, and (3) documenting change across strategies to eliminate disparities. This presentation of four different projects will provide a qualitative framework for understanding the problem and identifying key elements that ensure program sustainability and achieve greater public health impact. |
| A Long-Distance Evaluation of an In-Home Family Services Model: Planning Processes, Successes and Barriers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stephanie Ingram, Boys Town, ingrams@boystown.org |
| Tanya Shaw, Boys Town, shawt@boystown.org |
| Kristin Duppong Hurley, University of Nebraska Lincoln, kdupponghurley2@unl.edu |
| Katy Casey, University of Nebraska Lincoln, katyjcasey@yahoo.com |
| Abstract: Family service interventions are under increasing pressure from funding agencies and clients to demonstrate their effectiveness. Specifically, agencies want to gain a better understanding of what works, for which clients, under what conditions. This difficult task becomes even more complex when conducting a large scale evaluation of at-risk populations in a collaborative research effort, as there are many challenges and processes to plan, including: how to select appropriate measures, how to train staff in complex data collection procedures, how to conduct program fidelity, how to maintain client confidentiality between the service provider and an university research center, and how to monitor data collection, all from 1500 miles away from the intervention site. This paper will discuss the planning processes, challenges and ways to overcome barriers of a long-distance, complex evaluation to help other researchers planning large scale evaluations across the miles. |
| Call for Collaboration in the Midwest: Professional Evaluators in Nebraska Are Seeking Ways to Develop Local Network |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Robert Pawloski, University of Nebraska Medical Center, rwpawloski@unmc.edu |
| Barbara Jackson, University of Nebraska Medical Center, bjjackso@unmc.edu |
| Lisa St Clair, University of Nebraska Medical Center, lstclair@unmc.edu |
| Abstract: Evaluators from the University of Nebraska System (University of Nebraska at Omaha, University of Nebraska Lincoln, University of Nebraska at Kearney, and University of Nebraska Medical Center) have taken initial steps in a collaborative effort to hold a program evaluation conference in Omaha, Nebraska, early 2009. In addition to bringing quality professional development to the Omaha area, we would like to use this as an opportunity to survey the interest level of evaluation colleagues in Iowa and other nearby states about the possibility of forming a network that has potential to evolve into a regional affiliate of the AEA. This poster will present conference details as they develop. Interested evaluators should leave a card, and/or take contact information. We would also appreciate connecting with others who have had experiences and would like to share their successes and challenges in initiating such a conference, as well as launching an affiliate. |
| Continuous Quality Improvement Process: Reinforcing Professional Behavior While Developing Evaluation Skills in Medical Students |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Craig Timm, University of New Mexico, ctimm@salud.unm.edu |
| Paul McGuire, University of New Mexico, pmcguire@salud.unm.edu |
| Roger Jerabek, University of New Mexico, rjerabek@salud.unm.edu |
| Phil Szydlowski, University of New Mexico, pszydlowski@salud.unm.edu |
| Denise Minton, University of New Mexico, dminton@salud.unm.edu |
| Tony Ogburn, University of New Mexico, jogburn@salud.unm.edu |
| Abstract: To engage students as stakeholders, develop their professional competencies, and improve our curriculum, our medical school has implemented a Continuous Quality Improvement process within ten required courses in the first two years of the curriculum. The process involves immersing students in evaluation based on the guiding principles ethical framework adopted by the American Evaluation Association (see Guiding Principles, at http://www.eval.org/Publications/GuidingPrinciples.asp). Students volunteer and five are randomly selected to review a course evaluation report and summarize it with specific recommendations. There are standard evaluation questions used in each course as well as unique questions that course directors write. As evaluators the students are expected to 1) become competent in applying systematic inquiry in their review of data, 2) develop a summary with recommendations that demonstrates honesty and integrity, 3) integrate respect for persons in their work, 4) and consider the common good for the community. |
| Issues Faced when Inquiring with Indigenous Peoples: Views and Thoughts from Different Authors |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Laura Jimenez, Brigham Young University, ljimenezron@gmail.com |
| David Williams, Brigham Young University, ddwbyu@gmail.com |
| Abstract: • Effective action requires understanding the many (e.g., physical and social) contexts in which we seek to act, including what works and what does not (Chambers, 1997) within a particular culture. The general purpose of this research is to identify, understand, and better address the issues inquirers and their collaborating participants face when doing inquiry with indigenous peoples. The study explores several issues to prioritize when doing research or evaluation among people of other cultures. • We need publications that inform inquirers about key issues when evaluating & researching with Indigenous people • The question is: “How should researchers and evaluators collaborate with indigenous peoples most effectively?” |
| Process Evaluation of a Homeless Alcohol Harm Reduction Program: A Voice of Participants |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lisa Dirks, Northern Arizona University, ldirks@pire.org |
| Jennifer Norland, Pacific Institute for Research and Evaluation, jnorland@pire.org |
| Knowlton Johnson, Pacific Institute for Research and Evaluation, kjohnson@pire.org |
| Abstract: Evaluations related to harm reduction (HR) models are limited. The overarching goal of HR is to reduce harm associated with addictive behavior (Marlatt and Tapert, 1993). A main tenet of the HR model is the responsibility of participants for their own goal setting. Since a HR approach is fairly individualized, a participant level perspective is appropriate to consider in evaluating such a program. This presentation of participant experience provides context for what it means to be in a program of this type. The results of interviews conducted as part of a process evaluation of a homeless alcohol harm reduction program are discussed. Utilizing the idea that communal narratives and personal stories are resources for empowerment (Rappaport 1995), the interviews gave voice to participants. This is useful in understanding the goal setting process as well as what it means to be in such a program from the perspective of those being served. The perceptions of the actual participants of the program glean an understanding of what may lead to successful participation. |
| Evaluating the 2nd Year Implementation and Effectiveness of the Kansas Afterschool Leadership Development Initiative to Enhance Afterschool Program Settings |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diane Betzen, Wichita State University, dmbetzen@wichita.edu |
| Tara Gregory, Wichita State University, tara.gregory@wichita.edu |
| Sarah Jolley, Wichita State University, sarah.jolley@wichita.edu |
| Emily Grant, Wichita State University, eagrant@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Debra Renollet, Wichita State University, dlrennolet@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Afterschool programs have long been seen as an appropriate avenue for creating positive youth development. However, there are few models that provide guidance to programs in creating the setting characteristics necessary to positive youth development. For this reason, the Kansas Afterschool Leadership Development Initiative (KALDI) youth leadership capacity-building intervention was implemented as a way to increase organizational effectiveness and promote efforts to achieve positive youth development outcomes through a focus on positive youth-adult relationships, youth-centered norms, and youth participation in decision-making. Results to be presented include the outcomes from measures of KALDI’s effectiveness in its second year, such as surveys of youth and adults in the programs, and observation and interview measures. Insights regarding the successes and challenges of efforts to impact settings through youth leadership capacity-building will be presented. |
| A Fresh Perspective on Evaluating a School-University Partnership |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Steven Thomas, College of Charleston, thomassl@cofc.edu |
| Krysti Webber, College of Charleston, kristina_webber@charleston.k12.sc.us |
| Stephen Jurs, Independent Consultant, sgjurs@charter.net |
| Paula Egelson, College of Charleston, egelsonp@cofc.edu |
| Abstract: This program evaluation report includes information about a partnership effort between the College of Charleston and the Charleston County School District to improve the student outcomes at Burke High School. During the 2007-2008 school year, quantitative and qualitative data were collected to investigate the development of the partnership relationship. Prior evaluation led to several conclusions indicating that stronger relationships have developed between university and school personal. The 2007-2008 report will employ social network analysis to further contextualize the relationships that have developed as a result of this partnership. |
| Steps for Improving Outcome Measurement of a Family-Centered Poverty Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deborah Frater, University of Michigan, dfrater@umich.edu |
| Sue Ann Savas, University of Michigan, ssavas@umich.edu |
| Ann Kalass, Starfish Family Services, akalass@sfish.org |
| Abstract: As a more effective alternative to public poverty programs, Starfish Family Services of Michigan developed the Family Success Program to move families out of poverty into financial stability. The pilot program identified the following expected outcomes: 200% of the Federal Poverty Level, hope for the future, and one person outside of the family supporting their move to independence. This data was collected through face-to-face interviews and phone surveys at program end and 6 months post-program end. A retrospective pre-test survey was administered to capture missing baseline data. The preliminary outcome results indicated that the program produced a number of changes that were not identified as expected outcomes. This led program stakeholders to a re-conceptualization of the program’s theory of change. With the second implementation, evaluators will utilize standardized outcome measures to assess social support, life skills, coping skills, and parenting skills as well as measures of financial stability. |
| Continuous Quality Improvement Process: Reinforcing Professional Behavior While Developing Evaluation Skills in Medical Students |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Craig Timm, University of New Mexico, ctimm@salud.unm.edu |
| Paul McGuire, University of New Mexico, pmcguire@salud.unm.edu |
| Roger Jerabek, University of New Mexico, rjerabek@salud.unm.edu |
| Phil Szydlowski, University of New Mexico, pszydlowski@salud.unm.edu |
| Denise Minton, University of New Mexico, dminton@salud.unm.edu |
| Tony Ogburn, University of New Mexico, jogburn@salud.unm.edu |
| Abstract: To engage students as stakeholders, develop their professional competencies, and improve our curriculum, our medical school has implemented a Continuous Quality Improvement process within ten required courses in the first two years of the curriculum. The process involves immersing students in evaluation based on the guiding principles ethical framework adopted by the American Evaluation Association (see Guiding Principles, at http://www.eval.org/Publications/GuidingPrinciples.asp). Students volunteer and five are randomly selected to review a course evaluation report and summarize it with specific recommendations. There are standard evaluation questions used in each course as well as unique questions that course directors write. As evaluators the students are expected to 1) become competent in applying systematic inquiry in their review of data, 2) develop a summary with recommendations that demonstrates honesty and integrity, 3) integrate respect for persons in their work, 4) and consider the common good for the community. |
| Examining Time-to-Degree within a Multidisciplinary, Multi-Institutional Graduate Training Program--Successes and Challenges |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Dylan Flather, University of Colorado Boulder, dylan.flather@colorado.edu |
| Kristin Kusmierek, University of Michigan, kkusmierek@stanfordalumni.org |
| Tyler Silverman, University of Colorado Boulder, tsilverman1@gmail.com |
| Mary Anne Carroll, University of Michigan, mcarroll@umich.edu |
| David Karowe, Western Michigan University, karowe@wmich.edu |
| Abstract: Time-to-degree is particularly important in interdisciplinary programs, often thought to add time compared to traditional programs. The Biosphere-Atmosphere Research and Training (BART) program, a project of the National Science Foundation’s Integrative Graduate Education and Research Traineeship (IGERT) initiative, is an interdisciplinary program with participants from over a dozen disciplines and universities. This analysis compares time-to-degree of participants in BART’s two-year supplemental program against the time for their non-BART peers from corresponding home institutions. The preliminary results show that former BART students graduate in about the same time as non-BART students. |
| Key Factors for Effective Implementation of School Curricula: Achieving Success in Reading First Outcomes |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Wendy Wolfersteig, Arizona State University, wendy.wolfersteig@asu.edu |
| Arturo Valdivia, Arizona State University, arturo.valdivia@asu.edu |
| Lynn Katz, Arizona State University, lynn.katz@asu.edu |
| Abstract: The goal of Reading First is to impact student achievement via improved instructional practice in the classroom. Eight factors identified in the literature that can impact effective implementation of this program were examined using evaluation measures including test scores, surveys, and fidelity checklists. Since students interact primarily with teachers, this study focused on the impact teachers' instruction, motivation, and beliefs had on students’ reading performance. The ancillary roles of principals, reading specialists, reading coaches, district coordinators, and site-based teams were examined with regard to their roles, support and resources provided. Using a series of multiple regression analyses, this study examined data for 20,000 kindergarten, first, second and third grade students across 61 Arizona schools for the first three years of the program's implementation. Initial findings identified the crucial role the principals’ leadership provides in their ability to enhance teachers’ positive attitudes and instructional improvements as significant predictors of students' performance. |
| A Comprehensive Evaluation of the Active Living by Design Initiative: Innovative Approaches to Increase Physical Activity |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Melissa Hall, Transtria LLC, melissa@transtria.com |
| Laura Brennan Ramirez, Transtria LLC, laura@transtria.com |
| Cheryl Kelly, Saint Louis University, kellycm@slu.edu |
| Abstract: The Active Living by Design (ALbD) initiative establishes innovative approaches to increase physical activity through community design, public policies and communications strategies. Evaluation efforts focus on the ALbD Community Action Model components: preparation, promotions, programs, physical projects, and policies. This evaluation is intended to capture the range of grantee impacts and the strengths and challenges of implementing the ALbD interventions at the local level while focusing on three primary aims: 1) to assess the environmental impacts of physical projects and associated policy changes, 2) to document the range of interventions and accomplishments, and 3) to identify strengths and challenges in the process of planning, developing, and implementing the interventions. Visits to each of the 25 grantee sites will be completed by November 2008. This poster will highlight the methods used in the evaluation as well as findings and recommendations for building community design initiatives to increase physical activity. |
| Popular Empirical Assessment of Community Health (PEACH): An Interactive Methodology |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Teresa Kelly, Health Council of West Central Floirda Inc, tkelly@healthcouncils.org |
| Abstract: Community participation is a crucial component of community planning, but it is often difficult to obtain or impossible to quantify. By using an interactive methodology called the Popular Empirical Assessment of Community Health (PEACH), qualitative data can be captured in a quantifiable way, allowing for integration of community input into evaluations. PEACH utilizes a flexible ranking methodology to solicit information from the community on topics ranging from priority issues; to areas of unmet need; to how scarce resources should be allocated; to the role various agents should play in expanding access. This highly interactive process converts individuals’ descriptive explanations of an issue or problem to quantifiable data. PEACH is conducted during community-based meetings and allows all present to rank five subjects on five variables using a 5 x 5 pocket grid. The tool can be used with any size group and with persons of varying educational backgrounds. |
| Student Association of Global Affairs Uses Empowerment Evaluation Concepts, Techniques, and Findings to Foster Self-Determination, Capacity Building and Program Improvement |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Roberta Eleazer, Oklahoma State University, eleazer@gci.net |
| James Eleazer, Oklahoma State University, eleazer@gci.net |
| Don Eskridge, Oklahoma State University, don.eskridge@okstate.edu |
| Kathleen D Kelsey, Oklahoma State University, kathleen.kelsey@okstate.edu |
| Abstract: This poster will examine how graduate students used empowerment evaluation techniques (Fetterman, 2001) to foster sustainable self-determination for the membership of a graduate student organization, the Student Association of Global Affairs (SAGA) at a land-grant university. SAGA is a nine-year old student organization; however, it stagnated and failed to attract member participation at activities and was unable to fill club officer positions. Using a deliberative democratic process (House, 1993), a graduate student team of evaluators coached the executive board members and interested club members in designing an evaluation that resulted in improved program participation and outcomes. The student evaluators surveyed the membership’s opinions, attitudes, beliefs, behaviors, and preferences using an on-line survey. The evaluators involved the board members at every turn to empower stakeholders. Findings, conclusions, and recommendations were developed in conjunction with the board. The student evaluators believe that the evaluation process transformed SAGA into a sustainable organization with a more engaged board and student membership. References: Fetterman, D. M. (2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage Publications. House, E. R. (1993). Professional Evaluation: Social Impact and Political Consequences. Thousand Oaks: Sage. |
| Evaluation Stew: Multiple Roles of the Evaluator in Education |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chris Haynes, Oklahoma State University, chris.haynes@okstate.edu |
| Tim Kock, Oklahoma State University, tim.kock@okstate.edu |
| Kathleen D Kelsey, Oklahoma State University, kathleen.kelsey@okstate.edu |
| Abstract: This poster will serve as a vehicle to outline an evaluation of AGED 4713 International Programs in Agricultural Education and Extension at Oklahoma State University. Traditionally this course has been taught by tenured faculty members at this institution, and until recently held an enrollment of more than 100 students. Multiple changes in the department have necessitated that this course be assigned to a graduate student working on a Ph.D. in the area of Extension Education and International Development. The focus of this evaluation is to determine if the quality and rigor of the course is being upheld. The evaluation will focus on the effectiveness of the instructors’ teaching methods and how conducive it is to student learning, if the course objectives are being met, and what the actual student perceptions are of the course. Data will be determined through historical documentation, peer assessment, and through student focus group discussion. |
| Raising Up Young Evaluators: Training Social Work Students in Evaluation Methods Through Service Learning |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Adjoa Robinson, University at Buffalo - State University of New York, adr7@buffalo.edu |
| Wooksoo Kim, University at Buffalo - State University of New York, wkim5@buffalo.edu |
| Thomas H Nochajski, University at Buffalo - State University of New York, thn@buffalo.edu |
| Abstract: This poster describes the methods used to develop evaluation practice as an integral part of professional identity in social workers, improve learning outcomes, and serve the community, through the mechanism of sequential research and evaluation courses. Instructors taught the same cohort of students over two semesters and supervised students working in groups to conduct program evaluations. Key to this process was engaging students in service-learning projects through which they planned and implemented program evaluations in partnership with local service agencies. These service-learning projects served the dual purpose of real-world learning of evaluation skills and providing community agencies with needed service. The year-long sequence afforded students the opportunity to work through all the steps in the research process. Students conducted needs assessments and process and outcome evaluations at local agencies. Pre-test data will be analyzed to assess student knowledge and attitudinal outcomes. The poster will present the course components and objectives, instructional methods, measures, and lessons learned from the first year of implementation. |
| Evaluation of the Guide to Clinical Preventive Services: Applying a Multi-Method Approach to the Assessment of a Preventive Medicine Product |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jacqueline King, Johnson, Bassin and Shaw International Inc, jking@jbsinternational.com |
| Julie Nicholson, Johnson, Bassin and Shaw International Inc, jnicholson@jbsinternational.com |
| Lisa Kretz, Johnson, Bassin and Shaw International Inc, lkretz@jbsinternational.com |
| Susan Hayashi, Johnson, Bassin and Shaw International Inc, shayashi@jbsinternational.com |
| Abstract: This evaluation assessed the content, user-friendliness, and usefulness of The Guide to Clinical Preventive Services, an evidence-based product developed and disseminated by the Agency for Healthcare Research and Quality (AHRQ). Three versions of the Guide were evaluated in this project: the printed guide as a handbook, the Preventive Services website, and an electronic version available for download to PDA. A multi-method approach employed both email surveys and focus groups to assess the Guide’s’ use and usefulness among primary care providers (e.g., physicians, physician assistants, and nurse practitioners). Participants were recruited from those individuals who ordered the Guide online or by telephone and who agreed to participate in the study by completing emailed surveys. Focus groups were conducted at selected medical sites and regional conferences in order to gather feedback from primary care providers. Multivariate analyses were used to analyze 1,260 email survey responses, and content analysis was used to assess focus group data. |
| Reflections on Teaching Community Based Research: Recommendations for Preparing Collaborative, Participatory and Empowerment Evaluators |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristen Pozzoboni, University of Colorado Boulder, kristen.pozzoboni@colorado.edu |
| Ben Kirshner, University of Colorado Boulder, ben.kirshner@colorado.edu |
| Abstract: Community based research (CBR) is a collaborative research model in which university researchers and students, partner with community organizations to address needs or issues defined by the community organization. CBR creates opportunities for students to learn research and evaluation skills and benefits the partner agency by producing a product that supports or furthers the work of the community organization (Stocking & Cutforth, 2006). This session features lessons learned by university faculty and students about conducting CBR evaluations of youth development organizations. We concentrate on the methods used to prepare novice researchers and the types of support that assisted student learning as they designed and implemented projects. We also discuss challenges faced by students, such as managing ethical dilemmas, developing relationships, and learning to navigate cultural and socioeconomic differences. This session contributes to evaluation knowledge by addressing how to prepare evaluators and develop collaborative evaluations driven by community goals and interests. |
| Time Gap Model for the Bombings in Region XII, Mindanao, Philippines |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Epimaco Jr Cabanlit, Mindanao State University, maco_727@yahoo.com |
| Robert Kiunisala, South Cotabato Police Provincial Office, robert_bing@yahoo.com |
| Romeo Jr Galgo, Regional Mobile Group, academy158@yahoo.com |
| Steiljes M Cabanlit, Mindanao State University, steilt_727@yahoo.com |
| Abstract: From the year 2000 to 2006, a total of 96 bombing incidents occurred, 101 bombs exploded, 86 were killed and 674 were injured in Region XII, Mindanao, Philippines. This paper presents a time gap model for these bombings. The mixture of exponential and uniform distribution shows a good fit. |
| Validation of a Work/Nonwork Life Employee Retention Model |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tracy Kline, RTI International, tkline@rti.org |
| Michael Schwerin, RTI International, schwerin@rti.org |
| Murrey Olmsted, RTI International, molmsted@rti.org |
| Abstract: Understanding factors affecting employee turnover intent is essential for human resources and organizational development decision makers to develop a strategic human capital plan. Wilcove, Schwerin, and Wolosin (2003) developed an exploratory work/nonwork spillover model that identifies life needs that are related to turnover intent. Additional research was warranted to determine (1) the replicability of the model, (2) the validity of the model when extended to include retention behavior; and (3) generalizability of the model to the commercial sector. This study used data from three independent samples (Sample 1: 2002 Navy QOL Survey [N = 5,114]; Sample 2: Two Southern U.S. Hospitals [N = 984]; Sample 3: Northern U.S. Commercial Trucking Company [N = 884]) to validate the work/nonwork spillover model. Results support the replicability, validity, and generalizability of the Wilcove et al. (2003) model and an approach that integrates both work and nonwork life needs in the commercial sector. |
| Evaluation Stew: Multiple Roles of the Evaluator in Education |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chris Haynes, Oklahoma State University, chris.haynes@okstate.edu |
| Tim Kock, Oklahoma State University, tim.kock@okstate.edu |
| Kathleen D Kelsey, Oklahoma State University, kathleen.kelsey@okstate.edu |
| Abstract: This poster will serve as a vehicle to outline an evaluation of AGED 4713 International Programs in Agricultural Education and Extension at Oklahoma State University. Traditionally this course has been taught by tenured faculty members at this institution, and until recently held an enrollment of more than 100 students. Multiple changes in the department have necessitated that this course be assigned to a graduate student working on a Ph.D. in the area of Extension Education and International Development. The focus of this evaluation is to determine if the quality and rigor of the course is being upheld. The evaluation will focus on the effectiveness of the instructors’ teaching methods and how conducive it is to student learning, if the course objectives are being met, and what the actual student perceptions are of the course. Data will be determined through historical documentation, peer assessment, and through student focus group discussion. |
| Case Study Evaluation of a Professional Development Program for Middle and High School Science Teachers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Karen Koski, University of Northern Colorado, kkconsulting.karen@gmail.com |
| Abstract: Over the course of four years, four doctoral fellows along with three leaders from a Colorado university created and facilitated a professional development program for middle and high school science teachers. The professional development program was evaluated through a case study approach to determine the impact on both the fellows and the leaders who facilitated the program and the science teachers who participated in the program. While the fellows were learning the skills necessary to organize and conduct professional development the teacher participants were learning the skills they needed to implement inquiry based science in their classrooms. |
| Exploring the Role of Evaluation in Community Foundations Grantmaking |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristianna Pettibone, MayaTech Corporation, kpettibone@mayatech.com |
| Shelly Kowalczyk, MayaTech Corporation, skowalczyk@mayatech.com |
| Ayana Perkins, MayaTech Corporation, aperkins@mayatech.com |
| Abstract: Community foundations provide an important source of funding for local health promotion programs that can that result in better health, improved quality of life, and stronger communities. Data from these programs needs to be evaluated in order to determine the nature and extent of these changes. Evaluation of these community health initiatives is critical in assessing successful programs and ensuring that these programs can be replicated and sustained. We will provide an overview of the current status of evaluation policies at Community Health Foundations in Maryland, Virginia and North Carolina. We will examine how the policies are made, the role of evaluation in funded projects and the support provided for evaluation efforts. In addition, we will identify areas in which grantees might benefit from evaluation capacity building. Finally we will examine the grantee perspective to determine challenges or issues associated with the evaluation policies and suggestions for improvement. |
| Evaluation of the Guide to Clinical Preventive Services: Applying a Multi-Method Approach to the Assessment of a Preventive Medicine Product |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jacqueline King, Johnson, Bassin and Shaw International Inc, jking@jbsinternational.com |
| Julie Nicholson, Johnson, Bassin and Shaw International Inc, jnicholson@jbsinternational.com |
| Lisa Kretz, Johnson, Bassin and Shaw International Inc, lkretz@jbsinternational.com |
| Susan Hayashi, Johnson, Bassin and Shaw International Inc, shayashi@jbsinternational.com |
| Abstract: This evaluation assessed the content, user-friendliness, and usefulness of The Guide to Clinical Preventive Services, an evidence-based product developed and disseminated by the Agency for Healthcare Research and Quality (AHRQ). Three versions of the Guide were evaluated in this project: the printed guide as a handbook, the Preventive Services website, and an electronic version available for download to PDA. A multi-method approach employed both email surveys and focus groups to assess the Guide’s’ use and usefulness among primary care providers (e.g., physicians, physician assistants, and nurse practitioners). Participants were recruited from those individuals who ordered the Guide online or by telephone and who agreed to participate in the study by completing emailed surveys. Focus groups were conducted at selected medical sites and regional conferences in order to gather feedback from primary care providers. Multivariate analyses were used to analyze 1,260 email survey responses, and content analysis was used to assess focus group data. |
| Examining Time-to-Degree within a Multidisciplinary, Multi-Institutional Graduate Training Program--Successes and Challenges |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Dylan Flather, University of Colorado Boulder, dylan.flather@colorado.edu |
| Kristin Kusmierek, University of Michigan, kkusmierek@stanfordalumni.org |
| Tyler Silverman, University of Colorado Boulder, tsilverman1@gmail.com |
| Mary Anne Carroll, University of Michigan, mcarroll@umich.edu |
| David Karowe, Western Michigan University, karowe@wmich.edu |
| Abstract: Time-to-degree is particularly important in interdisciplinary programs, often thought to add time compared to traditional programs. The Biosphere-Atmosphere Research and Training (BART) program, a project of the National Science Foundation’s Integrative Graduate Education and Research Traineeship (IGERT) initiative, is an interdisciplinary program with participants from over a dozen disciplines and universities. This analysis compares time-to-degree of participants in BART’s two-year supplemental program against the time for their non-BART peers from corresponding home institutions. The preliminary results show that former BART students graduate in about the same time as non-BART students. |
| Data Acquisition: Considered the Future, but Stuck to the Past |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| John Enger, Nova Southeastern University, enger@bellsouth.net |
| Candace Lacey, Nova Southeastern University, chlacey@bellsouth.net |
| Nydia Cummings, Nova Southeastern University, cnydia@nova.edu |
| Abstract: While moving to online data acquisition to obtain student self-report measures, this presentation shows the problems encountered that resorted in data collection using paper and pencil. Further, in evaluating a comprehensive middle school character education program, two new surveys were created for the project. Their attributes will also be addressed. |
| Evaluation Methods and Findings From a Four-Year Longitudinal Prevention Study to Reduce Caregiver Strain of Adoptive Parents |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Robin Leake, University of Denver, rleake@du.edu |
| Abstract: The Butler Institute for Families at the University of Denver Graduate School of Social Work is conducting a 5-year longitudinal evaluation of the Adoptive Families Initiative (AFI) program through the Colorado Coalition for Adoptive Families. AFI is a Federal grant project to prevent adoption disruption by strengthening relationships of adoptive parents and reducing caregiver strain through intensive relationship building weekend PREP« retreats and follow-up adoption support services in five counties in Colorado. Preliminary results using a repeated-measures design show significant mean differences in positive relationship skills (e.g., better communication strategies) and decreased internalized caregiver strain from baseline to follow-up, and baseline to six-month assessment. These results suggest that participants are learning positive relationship skills such as better communication strategies through the weekend PREP« retreat, and are using these skills effectively six months and even a year after the retreat. |
| Interactive Evaluation as an Essential Component in An Innovative Annual Workshop |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Lynds, University of Colorado, susan.lynds@colorado.edu |
| Tamara Ledley, TERC, tamara_ledley@terc.edu |
| Susan Buhr, University of Colorado, susan.buhr@colorado.edu |
| Abstract: Since 2004, the annual AccessData and DLESE (Digital Library for Earth Systems Education) Data Services workshops have gathered scientists, data managers, technology specialists, teachers, and curriculum developers to work together creating classroom-ready scientific data modules. Teams of five (one participant from each of the five professions) develop topic-specific online educational units for online dissemination Extensive evaluation has been included in the design of each workshop. The evaluation results have been used each year to improve the agenda, topics, and procedures of each subsequent event. Evaluation data has also informed modification of the evaluation protocol each year. Some of the information collected each year reveals attendees’ experiences using scientific data for educational purposes—analysis of multi-year data sets reveals how these experiences have change over time. This poster reviews the evaluation design that has been essential in enhancing this unique and effective annual workshop. |
| Evaluation Capacity Building: Lessons Learned |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anthony Lee, University of South Carolina, abslee@gwm.sc.edu |
| Ruth Saunders, University of South Carolina, rsaunders@sc.edu |
| Abdoulaye Diedhiou, University of South Carolina, diedhiou@mailbox.sc.edu |
| Abstract: The finding in a needs assessment carried out concluded that the overall evaluation needs of a Chronic Disease Prevention Division in a State Health Department needed capacity building. Over a six-month period, planning ensued involving buy in from various organizational leaders, including funding agencies, recruitment of staff and interested partners among, a host of other planning activities. As a result of the planning, chronic disease prevention staff and partner organizations of a state health department immersed themselves in two evaluation capacity training seminars, over another six-month period. Over the longer term, the goal is to inject and reify the role of evaluation among implementing staff by prescribing evaluation into their daily work life. The findings from two seminar series suggest, in the short term, enhanced confidence and capacity relating to evaluation skills. Larger scale, prospective studies with comparison groups are needed to further investigate the efficacy of this intervention. |
| Evaluating the Effectiveness of Mental Health Services in Schools |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Steven Lee, University of Kansas, swlee@ku.edu |
| Jill Lohmeier, University of Massachusetts Lowell, jill_lohmeier@uml.edu |
| Abstract: Schools are the de facto system for the provision of mental health services to children and adolescents in the United States (Gonzalez et al., 2004). There are significant challenges in evaluating the effectiveness of these services. This evaluation was designed to assess outcomes related to increased mental health services in a Midwestern school district as a result of a federally funded grant program. Common factors in counseling as well as measures of effectiveness were obtained from both the counselor and the student for each of 969 cases. The results show discrepancies and similarities between perceptions of each dyad for short and long-term outcomes. These results will be contrasted to previously published attempts at comprehensive and long term evaluation of mental health services for children and youth. Limitations of the study and future directions for improving evaluations for mental health service delivery in the schools will be advanced. |
| An Example of Technical Training Utilization |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sandhya Rao, Pricewaterhousecoopers LLP, sandhya.rao@us.pwc.com |
| James Lehman, Pricewaterhousecoopers LLP, james.lehman@us.pwc.com |
| Daniel Birch, Pricewaterhousecoopers LLP, daniel.birch@us.pwc.com |
| Abstract: An important measure of the impact of training is change in job-related behaviors. Behavior change is Level 3 of the Kirkpatrick training evaluation model (1959); in it one examines changes in job behavior as a result of training. Level 3 needs to be taken together with Reaction, Learning and Business Results for a holistic understanding of training. The study was designed to understand job behavior changes as a result of sending 90 new hires to a six-week long residential technical training program. Follow-up information was gathered from the learners -- employees of PricewaterhouseCoopers, LLP-- 5 months after training. Learners were asked when they applied learning to the job, frequency of accessing training materials, opportunities for applying training and additional training needed. Learners were also asked to identify peers and supervisors who had worked with them post-training, who were in turn surveyed to understand learners' behaviors. Preliminary analyses indicated that in certain technical areas, over 75% of learners indicated that they had not had a chance to utilize any knowledge and skills from training. The results are valuable as they will be used to improve training content and usability for future offerings. |
| The Evaluation Experiences and Needs of County 4-H Educators |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristi Lekies, The Ohio State University, lekies.1@osu.edu |
| Amanda Hamiel Bennett, The Ohio State University, hamiel.3@osu.edu |
| Abstract: Extension 4-H educators are expected to document impacts of their programs on the youth they serve. A web-based survey was conducted with county 4-H educators in one state, Ohio (n=62), to examine their experiences, attitudes, knowledge, and skills regarding evaluation. Also assessed were barriers to conducting evaluation and beneficial supports. Results indicated an emphasis on outcome evaluations, limited variation in designs and methods, overall positive attitudes, and a moderate degree of confidence in conducting evaluations. Time to conduct evaluations and complete Institutional Review Board (IRB) processes were the greatest perceived barriers. Assistance from colleagues, state specialists, inservice trainings, and small group meetings were seen as the most beneficial in improving skills. Some variation by length of employment and job classification was noted for attitudes. Implications include broader involvement of youth in the evaluation process, increased use of logic models, and assistance with statistical analysis, web-based surveys, and publishing. |
| Assessing the Quality of School Improvement Program in Three Special Education Centers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ariana Leo Ramírez, University of Autonoma Yucatan, chinari_17@hotmail.com |
| Edith Cisneros-Cohernour, University of Autonoma Yucatan, cchacon@uady.mx |
| Abstract: This paper presents the findings of a qualitative study assessing the quality of a school improvement program in Mexico. The study examines how the program, developed as part of a standardized approach for school improvement, was responsive to the needs of the main stakeholders at the school level. |
| Evaluation Research as Intervention: Utilizing the Unintended Consequences of an Evaluation of Parent-Child Connectedness in a School-Based Pregnancy Prevention Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carol Lewis, University of Texas at Austin, carolmarie@mail.utexas.edu |
| Emily Wade, University of Texas at Austin, eewade@austin.utexas.edu |
| Amy Pierce, LifeWorks, amy.pierce@lifeworksweb.org |
| Abstract: Background: This evaluation assessed program components designed to enhance parent-child connectedness in a pregnancy prevention intervention for preadolescents conducted by a local non-profit agency. Methods: Parents of preadolescents in pregnancy prevention support groups were invited to attend family events and focus groups to provide feedback about how the program could improve parent-child connectedness. Results: The evaluation impacted participants in at least three significant ways: 1) Positive rapport among agency staff and parent participants was strengthened through increased communication during evaluation recruitment; 2) A preadolescent was provided mentoring services after her mother mentioned in a focus group that the child was participating in gang activity; and 3) Parents experienced the focus groups as opportunities to discuss family problems in a supportive setting. Conclusion: Researchers of human service interventions should consider the impact of their evaluations on all individuals involved to ensure ethical practices and to recognize ways that evaluation activities are a form of interventions. |
| Evaluation of a Reading Intervention Program for Struggling Middle and High School Readers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Debbie Hahs-Vaughn, University of Central Florida, dhahs@mail.ucf.edu |
| Mary Little, University of Central Florida, mlittle@mail.ucf.edu |
| Abstract: Multisite evaluation theory is often understood in the context of federally funded programs (Lawrenz & Huffman, 2003). There is very little published, however, on how multisite evaluation theory can be applied to intervention research in schools when the school district acts as the federal agency and the individual schools implement the program. This paper will add to the knowledge base of multisite evaluation in the context of school settings. The purpose of this session is to share the following: a) the evaluation approach for this large-scale multi-site reading intervention implemented in a large school district in south Florida; b) "lessons learned" for evaluating implementation of interventions in schools; and c) some of the results of the evaluation of the reading intervention program. |
| Assessing the Influence of Non-Health Programs and Policies on the Health of Unemployed Middle-Aged Mothers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Yujia Liu, Stanford University, yujialiu@stanford.edu |
| Abstract: This study examines the impact of unemployment among middle-aged mothers and the influence of unemployment insurance compensation and welfare benefits in ameliorating a possible health effect. We use multivariate regression to analyze the National Longitudinal Survey of Youth. Controlling for socioeconomic status, and baseline information two years before experiencing unemployment, we examine the extensively validated SF-12 health measure, at the time of experiencing the unemployment and two years after experiencing it (in 2004). Preliminary results show that after dropping respondents who were unemployed for health related reasons, of out of 646 mothers over 40 years of age, those who experienced unemployment and received welfare benefits had worse health status that those who were continuously employed, while those who were unemployed but received entitlement benefits were not significantly different. Additional cohorts will be added for our final analysis. Challenges and limitations of this methodology to evaluate social policies will be discussed. |
| Evaluating the Effectiveness of Mental Health Services in Schools |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Steven Lee, University of Kansas, swlee@ku.edu |
| Jill Lohmeier, University of Massachusetts Lowell, jill_lohmeier@uml.edu |
| Abstract: Schools are the de facto system for the provision of mental health services to children and adolescents in the United States (Gonzalez et al., 2004). There are significant challenges in evaluating the effectiveness of these services. This evaluation was designed to assess outcomes related to increased mental health services in a Midwestern school district as a result of a federally funded grant program. Common factors in counseling as well as measures of effectiveness were obtained from both the counselor and the student for each of 969 cases. The results show discrepancies and similarities between perceptions of each dyad for short and long-term outcomes. These results will be contrasted to previously published attempts at comprehensive and long term evaluation of mental health services for children and youth. Limitations of the study and future directions for improving evaluations for mental health service delivery in the schools will be advanced. |
| Evaluating a City-Wide Science Festival |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rucha Londhe, Goodman Research Group Inc, londhe@grginc.com |
| Abstract: The proposed paper is based on a study conducted to evaluate the first ever city-wide science festival in the U.S. The study included both summative evaluation (which focused on establishing the profile of the festival attendees and assessing the success of the festival) and process evaluation (which focused on understanding the challenges faced and the lessons learned by the collaborating organizations with regard to the implementation of the festival). The proposed paper details the study’s methodology and the data collection procedures and makes recommendations for future city-wide initiatives. In addition, the paper also highlights the overall methodology issues related to evaluation of festivals and such large-scope informal education venues. |
| Real World Challenges to Conducting an Implementation Evaluation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Herbert Baum, Macro International Inc, herbert.m.baum@macrointernational.com |
| William Scarbrough, Macro International Inc, william.h.scarbrough.iii@macrointernational.com |
| Felicia Solomon, National Institutes of Health, solomonf@mail.nih.gov |
| Arnold Love, Independent Consultant, arnold.love@gmail.com |
| Abstract: The National Cancer Institute has been working with community partners to disseminate an evidence-based program, Body & Soul, to increase the consumption of fruits and vegetables among Christian African Americans. Macro International was hired to conduct an implementation evaluation, determining what worked well with the dissemination effort and what did not. All evaluations depend on the timely and accurate submission of data, and working with Community-based organizations presents challenges with respect to data collection efforts. This paper reports on some of those challenges and how some were overcome. |
| Interactive Evaluation as an Essential Component in An Innovative Annual Workshop |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Lynds, University of Colorado, susan.lynds@colorado.edu |
| Tamara Ledley, TERC, tamara_ledley@terc.edu |
| Susan Buhr, University of Colorado, susan.buhr@colorado.edu |
| Abstract: Since 2004, the annual AccessData and DLESE (Digital Library for Earth Systems Education) Data Services workshops have gathered scientists, data managers, technology specialists, teachers, and curriculum developers to work together creating classroom-ready scientific data modules. Teams of five (one participant from each of the five professions) develop topic-specific online educational units for online dissemination Extensive evaluation has been included in the design of each workshop. The evaluation results have been used each year to improve the agenda, topics, and procedures of each subsequent event. Evaluation data has also informed modification of the evaluation protocol each year. Some of the information collected each year reveals attendees’ experiences using scientific data for educational purposes—analysis of multi-year data sets reveals how these experiences have change over time. This poster reviews the evaluation design that has been essential in enhancing this unique and effective annual workshop. |
| Capturing Change: Lessons Learned from the Evaluation of an Early Childhood Teacher Quality Intervention |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amanda Moreno, OMNI Institute, amoreno@omni.org |
| Shelby Maier, OMNI Institute, smaier@omni.org |
| Karin McBride, OMNI Institute, kmcbride@omni.org |
| Abstract: This presentation details the experiences and results of the process evaluation of a new curriculum for 0-3 year-olds in disadvantaged early care settings. The central problem to be addressed is that despite teachers’ own reflections as well as those of objective raters that the teachers developed skills over time as a result of implementing the curriculum, the ratings of teacher quality declined over time. That is, in the first year of the evaluation, raters’ own impressions and quantitative ratings were contradictory. Methods and data will be presented from the first year of the evaluation when this conundrum was discovered, and the second year of the evaluation when methods were altered to attempt to correct it. Lessons learned will be discussed in the context of the accountability movement in education, and the particular challenges involved in convincingly capturing quality improvement in classrooms serving the youngest children. |
| The Effect of On-Campus, Academic Year, Laboratory Based Research Experiences on STEM PhD Outcomes for Underrepresented Minorities |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Frances Carter, University of Maryland Baltimore County, frances2@umbc.edu |
| Marv Mandell, University of Maryland Baltimore County, mandell@umbc.edu |
| Abstract: The shortage of underrepresented minorities in science, technology, engineering and mathematics (STEM) research careers is attracting increased attention from evaluators and policymakers. Numerous programs intended to increase the participation of underrepresented minorities in STEM research careers have been implemented to address this problem. A key component in many of these programs is providing undergraduate research opportunities to program participants. However, limited evaluation has been done regarding the impact of participation in undergraduate research on pursuit and attainment of a STEM Ph.D. This paper analyzes the impacts of on-campus, academic year, laboratory based research in STEM fields on STEM PhD outcomes. The analysis uses data from fourteen cohorts of participants in the Meyerhoff Scholars Program at the University of Maryland, Baltimore County. A variety of estimation strategies, including OLS, instrumental variables estimation, and propensity matching are used to estimate the effects and address possible issues of endogeneity. |
| From the Shelf to the Hand: Notes on Becoming a Qualitative Program Evaluator |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ian Martin, University of Massachusetts Amherst, ianmartin.ian@gmail.com |
| Abstract: This paper describes the highs and lows of a first-time qualitative evaluator. While conducting a formative evaluation for a school counseling program in a rural Northeast school, the author confronts the realities of working collaboratively towards program improvement. This reflective process frames the difficult conversations, the apologizing, the ethical dilemmas, the rewriting, and the glory of consensus within rigorous systematic inquiry. Including the bumps and bruises, the story highlights the process of moving an evaluation from one that sits on the shelf, to one that can be used for active program development. |
| Students as Evaluators: Encouraging Them to Use Their (Photo) Voice |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anna Robic, University of South Florida, robic@tempest.coedu.usf.edu |
| Patricia Alvarez McHatton, University of South Florida, mchatton@tempest.coedu.usf.edu |
| Stephanie Martinez, Florida Mental Health Institute, smartinez@fmhi.usf.edu |
| Abstract: A trend in evaluation has been to utilize students as evaluators of programs. What happens when the students who have participated offer little feedback? How can you access what they might not be telling you? This session presents information on this dilemma as well as suggestions on how to overcome the problem. |
| Capturing Change: Lessons Learned from the Evaluation of an Early Childhood Teacher Quality Intervention |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amanda Moreno, OMNI Institute, amoreno@omni.org |
| Shelby Maier, OMNI Institute, smaier@omni.org |
| Karin McBride, OMNI Institute, kmcbride@omni.org |
| Abstract: This presentation details the experiences and results of the process evaluation of a new curriculum for 0-3 year-olds in disadvantaged early care settings. The central problem to be addressed is that despite teachers’ own reflections as well as those of objective raters that the teachers developed skills over time as a result of implementing the curriculum, the ratings of teacher quality declined over time. That is, in the first year of the evaluation, raters’ own impressions and quantitative ratings were contradictory. Methods and data will be presented from the first year of the evaluation when this conundrum was discovered, and the second year of the evaluation when methods were altered to attempt to correct it. Lessons learned will be discussed in the context of the accountability movement in education, and the particular challenges involved in convincingly capturing quality improvement in classrooms serving the youngest children. |
| Continuous Quality Improvement Process: Reinforcing Professional Behavior While Developing Evaluation Skills in Medical Students |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Craig Timm, University of New Mexico, ctimm@salud.unm.edu |
| Paul McGuire, University of New Mexico, pmcguire@salud.unm.edu |
| Roger Jerabek, University of New Mexico, rjerabek@salud.unm.edu |
| Phil Szydlowski, University of New Mexico, pszydlowski@salud.unm.edu |
| Denise Minton, University of New Mexico, dminton@salud.unm.edu |
| Tony Ogburn, University of New Mexico, jogburn@salud.unm.edu |
| Abstract: To engage students as stakeholders, develop their professional competencies, and improve our curriculum, our medical school has implemented a Continuous Quality Improvement process within ten required courses in the first two years of the curriculum. The process involves immersing students in evaluation based on the guiding principles ethical framework adopted by the American Evaluation Association (see Guiding Principles, at http://www.eval.org/Publications/GuidingPrinciples.asp). Students volunteer and five are randomly selected to review a course evaluation report and summarize it with specific recommendations. There are standard evaluation questions used in each course as well as unique questions that course directors write. As evaluators the students are expected to 1) become competent in applying systematic inquiry in their review of data, 2) develop a summary with recommendations that demonstrates honesty and integrity, 3) integrate respect for persons in their work, 4) and consider the common good for the community. |
| Understanding Service Needs of Older Adults |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Understanding and responding to the needs of older adults is a growing interest due to changing demographics, economic turns, and new ways to interact socially. At the same time it can be difficult to understand older adults’ needs for social interaction and community integration other than at a “surface level”. To overcome this challenge, this evaluation explores the needs of older adults in a Midwest community using a mixed method approach including personal interviews with older adults and key informants, focus groups with older adults and caregivers, and archival data. Through these methods, 200 participants provided feedback on available services for seniors as well as input on what they would like to see offered to older adults in the future. Results of this mixed method approach will be presented, including how they have helped guide a local agency and its partners’ plan for future older adult services. |
| Evaluating the 2nd Year Implementation and Effectiveness of the Kansas Afterschool Leadership Development Initiative to Enhance Afterschool Program Settings |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diane Betzen, Wichita State University, dmbetzen@wichita.edu |
| Tara Gregory, Wichita State University, tara.gregory@wichita.edu |
| Sarah Jolley, Wichita State University, sarah.jolley@wichita.edu |
| Emily Grant, Wichita State University, eagrant@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Debra Renollet, Wichita State University, dlrennolet@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Afterschool programs have long been seen as an appropriate avenue for creating positive youth development. However, there are few models that provide guidance to programs in creating the setting characteristics necessary to positive youth development. For this reason, the Kansas Afterschool Leadership Development Initiative (KALDI) youth leadership capacity-building intervention was implemented as a way to increase organizational effectiveness and promote efforts to achieve positive youth development outcomes through a focus on positive youth-adult relationships, youth-centered norms, and youth participation in decision-making. Results to be presented include the outcomes from measures of KALDI’s effectiveness in its second year, such as surveys of youth and adults in the programs, and observation and interview measures. Insights regarding the successes and challenges of efforts to impact settings through youth leadership capacity-building will be presented. |
| Early Childhood Programs and Children’s Behavior and Social Skills: A Longitudinal Investigation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sheretta Barnes, High Scope Educational Research Foundation, sbutler-barnes@highscope.org |
| Marijata Daniel-Echols, High Scope Educational Research Foundation, mdaniel-echols@highscope.org |
| Mary Delcamp, High Scope Educational Research Foundation, mdelcamp@highscope.org |
| Beth Scholz, High Scope Educational Research Foundation, bscholz@highscope.org |
| Olivia McKinney, High Scope Educational Research Foundation, oliviam@highscope.org |
| Rebecca Blisset, High Scope Educational Research Foundation, bblissett@highscope.org |
| Abstract: According to the U.S. Census Bureau (2006-2007) 54% of children between the ages of 3 and 4 are enrolled in school. Furthermore, the positive effects preschool attendance has on young children’s cognitive performance and school readiness is supported (Barnett, Lamy, & Jung, 2005). However, there is a dearth of research examining the different types of programs young children attend and the effects on socioemotional development. Therefore, it is important to explore how the type of early childhood program influences behavioral adjustment and social skills. The present study utilized a longitudinal growth model to examine demographic factors and program type (i.e. federally funded, state funded, or center based program) on children’s behavioral adjustment and social skills across a four year period. Results indicate that gender, parent’s education, food stamp assistance, and program type significantly accounted for the differences in young children’s behavioral adjustment and social skills. |
| Understanding Service Needs of Older Adults |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Understanding and responding to the needs of older adults is a growing interest due to changing demographics, economic turns, and new ways to interact socially. At the same time it can be difficult to understand older adults’ needs for social interaction and community integration other than at a “surface level”. To overcome this challenge, this evaluation explores the needs of older adults in a Midwest community using a mixed method approach including personal interviews with older adults and key informants, focus groups with older adults and caregivers, and archival data. Through these methods, 200 participants provided feedback on available services for seniors as well as input on what they would like to see offered to older adults in the future. Results of this mixed method approach will be presented, including how they have helped guide a local agency and its partners’ plan for future older adult services. |
| Evaluating the 2nd Year Implementation and Effectiveness of the Kansas Afterschool Leadership Development Initiative to Enhance Afterschool Program Settings |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diane Betzen, Wichita State University, dmbetzen@wichita.edu |
| Tara Gregory, Wichita State University, tara.gregory@wichita.edu |
| Sarah Jolley, Wichita State University, sarah.jolley@wichita.edu |
| Emily Grant, Wichita State University, eagrant@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Debra Renollet, Wichita State University, dlrennolet@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Afterschool programs have long been seen as an appropriate avenue for creating positive youth development. However, there are few models that provide guidance to programs in creating the setting characteristics necessary to positive youth development. For this reason, the Kansas Afterschool Leadership Development Initiative (KALDI) youth leadership capacity-building intervention was implemented as a way to increase organizational effectiveness and promote efforts to achieve positive youth development outcomes through a focus on positive youth-adult relationships, youth-centered norms, and youth participation in decision-making. Results to be presented include the outcomes from measures of KALDI’s effectiveness in its second year, such as surveys of youth and adults in the programs, and observation and interview measures. Insights regarding the successes and challenges of efforts to impact settings through youth leadership capacity-building will be presented. |
| An Application of Concept Mapping in a Culturally Responsive Evaluation of an International Education Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rochelle Michel, Educational Testing Service, rmichel@ets.org |
| Abstract: A six-step concept mapping procedure was followed to develop the evaluation questions to be used in this evaluation. The goal of the concept mapping was to generate a set of answerable evaluation questions that were of interest to a number of stakeholders, related to the long term outcomes of the program. Finding answers to the questions will help to ensure that the results will more likely be used to improve the program. This process allowed stakeholders from all sectors of the program to have the chance for input. It is important that all key voices are accurately heard and listened to. If not, the entire evaluation process may be limited in its accuracy and opportunities for meaningful program improvement drastically reduced. |
| Supporting Causal Inferences and Generalization When Evaluating Large-scale, Multi-state, Multi-level Implementations of an Integrated Instructional Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brad Chaney, Westat, bradchaney@westat.com |
| Jane Delgado, College Board, jdelgado@collegeboard.org |
| Doreen Finkelstein, College Board, dfinkelstein@collegeboard.org |
| Joan Michie, Westat, michiej1@westat.com |
| Abstract: In a large-scale, longitudinal, summative evaluation effort, issues of internal validity, external validity, and reliability and were addressed in supporting causal inferences about the impact and efficacy of an integrated instructional program. The program included curriculum, assessment and professional development for both English language arts and mathematics and implementation patterns were very different by subject area, school, district, and state. An interim dataset of more than 1.5 million observations of individual student achievement across multiple years was secured from participating and comparison schools in nine districts and six different states, both before and after program exposure. A repeated measures multi-level modeling approach was utilized in the SAS analyses. Complexities addressed include: using scale scores from different statewide standardized tests as the measure of achievement, determining an effect size when state scale scores differed in type and range, dealing with patterns of missing data, and accounting for varying degrees of exposure at the level of the student and the school. |
| What is Gained by Conducting Randomized Controlled Trials in Education? What is Known About Policy Regarding Conducting Them? |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kim Sprague, Brown University, kim_sprague@brown.edu |
| Candy Miller, Brown University, millerfkcm@comcast.net |
| Abstract: The Education Alliance at Brown University is conducting a randomized controlled trial (RCT) evaluating the effectiveness of two adolescent literacy interventions on the reading achievement of students identified as “struggling readers” (those scoring two levels below grade level), as implemented by two school districts in western Massachusetts. The districts were together awarded one of eight Striving Readers grants by the US Department of Education to conduct this work in collaboration with their evaluator. These studies are considered critical by the department to contribute to the field of adolescent literacy by addressing the primary research question of what interventions are effective. In addition, these studies are also some of the first to address what methods should be employed in such RCT trials, what is required in terms of policy for implementation and collaboration to conduct such efforts, and what is required to appropriately assess their implementation. |
| Continuous Quality Improvement Process: Reinforcing Professional Behavior While Developing Evaluation Skills in Medical Students |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Craig Timm, University of New Mexico, ctimm@salud.unm.edu |
| Paul McGuire, University of New Mexico, pmcguire@salud.unm.edu |
| Roger Jerabek, University of New Mexico, rjerabek@salud.unm.edu |
| Phil Szydlowski, University of New Mexico, pszydlowski@salud.unm.edu |
| Denise Minton, University of New Mexico, dminton@salud.unm.edu |
| Tony Ogburn, University of New Mexico, jogburn@salud.unm.edu |
| Abstract: To engage students as stakeholders, develop their professional competencies, and improve our curriculum, our medical school has implemented a Continuous Quality Improvement process within ten required courses in the first two years of the curriculum. The process involves immersing students in evaluation based on the guiding principles ethical framework adopted by the American Evaluation Association (see Guiding Principles, at http://www.eval.org/Publications/GuidingPrinciples.asp). Students volunteer and five are randomly selected to review a course evaluation report and summarize it with specific recommendations. There are standard evaluation questions used in each course as well as unique questions that course directors write. As evaluators the students are expected to 1) become competent in applying systematic inquiry in their review of data, 2) develop a summary with recommendations that demonstrates honesty and integrity, 3) integrate respect for persons in their work, 4) and consider the common good for the community. |
| An Evaluation Capacity Building Project in a Child Protection Services Agency: The Practice Evaluation Project |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Gilles Mireault, Quebec Youth Center, gilles_mireault@ssss.gouv.qc.ca |
| Abstract: The Practice Evaluation Project (PEP) is an initiative of Centre jeunesse de Québec-Institut universitaire, a Canadian university based child protection services agency based in Quebec City. The project’s main goal is to develop the agency’s evaluation capacity. Over the last 2 years, approximately 100 practitioners from the agency, divided into 6 work teams, were involved in this practice evaluation project. The presentation will focus on the project’s resources, evaluation questions and the process used to select them, research design, data collection and analysis, as well as practical use of results to improve services. After 2 years of activity, results show that practitioners can get involved in practice evaluation, sharing their perception and participating at various stages of the process (e.g. result interpretation). However, teams have shown more difficulties when the time comes to schedule work sessions to discuss results and decide what needs to be done to improve practice. Organizational issues are also a concern for the future. |
| Capturing Change: Lessons Learned from the Evaluation of an Early Childhood Teacher Quality Intervention |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amanda Moreno, OMNI Institute, amoreno@omni.org |
| Shelby Maier, OMNI Institute, smaier@omni.org |
| Karin McBride, OMNI Institute, kmcbride@omni.org |
| Abstract: This presentation details the experiences and results of the process evaluation of a new curriculum for 0-3 year-olds in disadvantaged early care settings. The central problem to be addressed is that despite teachers’ own reflections as well as those of objective raters that the teachers developed skills over time as a result of implementing the curriculum, the ratings of teacher quality declined over time. That is, in the first year of the evaluation, raters’ own impressions and quantitative ratings were contradictory. Methods and data will be presented from the first year of the evaluation when this conundrum was discovered, and the second year of the evaluation when methods were altered to attempt to correct it. Lessons learned will be discussed in the context of the accountability movement in education, and the particular challenges involved in convincingly capturing quality improvement in classrooms serving the youngest children. |
| The Good, The Bad, and The Ugly: Outsourcing Experiences |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristin Pankey, Southern Illinois University at Carbondale, knpankey@siu.edu |
| Joel Nadler, Southern Illinois University at Carbondale, jnadler@siu.edu |
| Nicole Cundiff, Southern Illinois University at Carbondale, karim@siu.edu |
| Nicholas G Hoffman, Southern Illinois University at Carbondale, nghoff@siu.edu |
| Gargi Bhattacharya, Southern Illinois University at Carbondale, gargi@siu.edu |
| Alen Avdic, Southern Illinois University at Carbondale, alen@siu.edu |
| Abstract: The Applied Psychology program at the Southern Illinois University at Carbondale (SIUC) has been providing training in program evaluation since 1982. Perhaps the most unique part of this training is the opportunity to conduct professional evaluations as a part of Applied Research Consultants (ARC), a student-run consulting firm. Over the past few years, ARC has been hired for an increasing amount of outsourcing services and has had a variety of experiences with each of these projects and evaluations. Three specific situations stand out and can be described as “The Good, The Bad, and The Ugly.” A good experience is defined as a project that allows for the development of skills and abilities. “Bad” and “Ugly” experiences are profitable but with low educational value. This presentation will discuss these three situations, ARC’s experiences and relationships with these clients, and tips for dealing with outsourcing. |
| Context, Culture and Change: Examining Racial and Ethnic Health Disparities Through Qualitative Evaluation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pattie Tucker, Centers for Disease Control and Prevention, ptucker1@cdc.gov |
| C Shannon Griffin Blake, Centers for Disease Control and Prevention, sgriffinblake@cdc.gov |
| Amanda Navarro, Centers for Disease Control and Prevention, anavarro2@cdc.gov |
| Shawna Howell, Centers for Disease Control and Prevention, showell@cdc.gov |
| Abstract: In 1999, the Centers for Disease Control and Prevention initiated Racial and Ethnic Approaches to Community Health (REACH) 2010 as the cornerstone of its efforts to eliminate racial and ethnic health disparities. REACH aimed to examine and address individual and environmental contexts which included differences across race, ethnicity, education, income, and geographic location. The uniqueness of REACH was its diversity in approaches and strategies and its emphasis on community participation in all aspects of program development, implementation and evaluation. Qualitative methods were utilized to elucidate the three themes presented in this poster: (1) understanding the broader environmental contexts impacting disparities, (2) recognition of culture as an important factor of influence on coalitions, and (3) documenting change across strategies to eliminate disparities. This presentation of four different projects will provide a qualitative framework for understanding the problem and identifying key elements that ensure program sustainability and achieve greater public health impact. |
| Evaluating Teachers' Professional Development Experiences: Year One Data from a Randomized Control Trial Study |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Andrew Newman, Mid-continent Research for Education and Learning, anewman@mcrel.org |
| Michael Phillips, University of Northern Colorado, michael.m.phillips@gmail.com |
| Abstract: Teachers’ participation in professional development (PD) activities was examined during an ongoing, large-scale randomized control trial (RCT). The intent of this two-year RCT is to investigate the use of a specific PD program focused on classroom assessment in participating Colorado elementary schools. For this poster, baseline and Year 1 data from 337 teachers were analyzed regarding professional development activities other than the RCT intervention. Teachers’ participation in PD was examined, including frequency, type of activity, and the perceived impact of the PD on classroom practice. Additionally, we explored demographic and geographic characteristics of our sample and how they relate to various types of PD in which teachers engage. The majority of PD completed by teachers focused on math and/or literacy (reading and writing); we conducted additional analyses on this subset of data. Evaluators will learn about the survey methodology, data findings, and the implications for data collection and analysis. |
| Sustainability in an Alternative Educational Leadership Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Rogers, University at Albany - State University of New York, bottlecap22@hotmail.com |
| Dianna Newman, University at Albany - State University of New York, dnewman@uamail.albany.edu |
| Abstract: Evaluating post-grant sustainability is an important aspect of determining the success of innovative educational programs. The current study explores this issue through the example of a three-year educational leadership grant which sought to produce new urban administrators, and to provide support for administrators beginning their career in urban districts. This latter goal was identified as a step towards building regional sustainability, and it is the focus of the current paper. In this portion of the program, early career administrators in urban districts were matched with local veteran administrators who acted as mentors for one academic year. Data sources include leadership surveys and interviews collected from program participants, and interviews and surveys collected from program mentors. Findings indicate that this program was successful in establishing a level of sustainable support for early-career urban administrators that transcended the official mentorship period. Data collection, analyses, findings, and lessons learned are discussed. |
| An Evaluation of the Mississippi Women In Agriculture: Annie Project |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paula Threadgill, Mississippi State University, paulat@ext.msstate.edu |
| Michael Newman, Mississippi State University, mnewman@humansci.msstate.edu |
| Abstract: The purpose of the Mississippi Women In Agriculture (MWIA) program was to increase the knowledge and skills of farm women in business management. The objectives of the program were: (1) provide workshops on risk management topics, (2) provide videoconferences that address the issues, and (3)develop a MWIG website that contain information that participants can review and reinforce concepts learned at regional workshops. Three regional three-day workshops and one regional one-day workshop were conducted across the state. Fifty-one pre and posttests were collected from the three regional three-day workshops and 13 were collected from the one-day workshop. All of the participants were female; the participant’s age ranged 40 to 59 years old. Eleven percent was single, 84% were married and four % were widowed. T-Tests were conducted on the means to reveal the significant differences between the pre and posttest. Participants indicated their knowledge level increased on all topics. |
| Strengthening the Evaluation Capacity of the National Institutes of Health |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brandie Taylor, National Institutes of Health, taylorbr@mail.nih.gov |
| Rosanna Ng, National Institutes of Health, ngr@mail.nih.gov |
| Meryl Sufian, National Institutes of Health, sufianm@mail.nih.gov |
| Madeleine Wallace, National Institutes of Health, wallacem2@mail.nih.gov |
| Yuanting Zhang, National Institutes of Health, zhangyuant@mail.nih.gov |
| Abstract: A comprehensive approach has been developed to strengthen the evaluation capacity of the National Institutes of Health (NIH). Led by the Evaluation Branch (EB), a part of the NIH Office of Portfolio Analysis and Strategic Initiatives, efforts are being implemented to build and sustain support for evaluation and evaluation capacity. This poster will showcase these activities (e.g., strategies to maximize the effective use of dedicated funding for evaluations, education and outreach activities, technical support and guidance, collaborative efforts to foster evaluation skills and knowledge) and highlight some of the challenges and lessons learned. |
| Evaluation of the Guide to Clinical Preventive Services: Applying a Multi-Method Approach to the Assessment of a Preventive Medicine Product |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jacqueline King, Johnson, Bassin and Shaw International Inc, jking@jbsinternational.com |
| Julie Nicholson, Johnson, Bassin and Shaw International Inc, jnicholson@jbsinternational.com |
| Lisa Kretz, Johnson, Bassin and Shaw International Inc, lkretz@jbsinternational.com |
| Susan Hayashi, Johnson, Bassin and Shaw International Inc, shayashi@jbsinternational.com |
| Abstract: This evaluation assessed the content, user-friendliness, and usefulness of The Guide to Clinical Preventive Services, an evidence-based product developed and disseminated by the Agency for Healthcare Research and Quality (AHRQ). Three versions of the Guide were evaluated in this project: the printed guide as a handbook, the Preventive Services website, and an electronic version available for download to PDA. A multi-method approach employed both email surveys and focus groups to assess the Guide’s’ use and usefulness among primary care providers (e.g., physicians, physician assistants, and nurse practitioners). Participants were recruited from those individuals who ordered the Guide online or by telephone and who agreed to participate in the study by completing emailed surveys. Focus groups were conducted at selected medical sites and regional conferences in order to gather feedback from primary care providers. Multivariate analyses were used to analyze 1,260 email survey responses, and content analysis was used to assess focus group data. |
| Raising Up Young Evaluators: Training Social Work Students in Evaluation Methods Through Service Learning |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Adjoa Robinson, University at Buffalo - State University of New York, adr7@buffalo.edu |
| Wooksoo Kim, University at Buffalo - State University of New York, wkim5@buffalo.edu |
| Thomas H Nochajski, University at Buffalo - State University of New York, thn@buffalo.edu |
| Abstract: This poster describes the methods used to develop evaluation practice as an integral part of professional identity in social workers, improve learning outcomes, and serve the community, through the mechanism of sequential research and evaluation courses. Instructors taught the same cohort of students over two semesters and supervised students working in groups to conduct program evaluations. Key to this process was engaging students in service-learning projects through which they planned and implemented program evaluations in partnership with local service agencies. These service-learning projects served the dual purpose of real-world learning of evaluation skills and providing community agencies with needed service. The year-long sequence afforded students the opportunity to work through all the steps in the research process. Students conducted needs assessments and process and outcome evaluations at local agencies. Pre-test data will be analyzed to assess student knowledge and attitudinal outcomes. The poster will present the course components and objectives, instructional methods, measures, and lessons learned from the first year of implementation. |
| Process Evaluation of a Homeless Alcohol Harm Reduction Program: A Voice of Participants |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lisa Dirks, Northern Arizona University, ldirks@pire.org |
| Jennifer Norland, Pacific Institute for Research and Evaluation, jnorland@pire.org |
| Knowlton Johnson, Pacific Institute for Research and Evaluation, kjohnson@pire.org |
| Abstract: Evaluations related to harm reduction (HR) models are limited. The overarching goal of HR is to reduce harm associated with addictive behavior (Marlatt and Tapert, 1993). A main tenet of the HR model is the responsibility of participants for their own goal setting. Since a HR approach is fairly individualized, a participant level perspective is appropriate to consider in evaluating such a program. This presentation of participant experience provides context for what it means to be in a program of this type. The results of interviews conducted as part of a process evaluation of a homeless alcohol harm reduction program are discussed. Utilizing the idea that communal narratives and personal stories are resources for empowerment (Rappaport 1995), the interviews gave voice to participants. This is useful in understanding the goal setting process as well as what it means to be in such a program from the perspective of those being served. The perceptions of the actual participants of the program glean an understanding of what may lead to successful participation. |
| Evaluating Smaller Learning Communities in United States High Schools |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Linda Bol, Old Dominion University, lbol@odu.edu |
| Shana Pribesh, Old Dominion University, spribesh@odu.edu |
| Christopher Fischer, Old Dominion University, cfischer@nps.k12.va.us |
| John Nunnery, Old Dominion University, jnunnery@odu.edu |
| Abstract: Creating smaller schools/learning communities (SLCs) have been advocated as a specific reform for improving high school student engagement and graduation rates (NRC, 2002). The linkages of smaller learning communities to student achievement have been found to be promising (Felner, Ginter, and Primavera, 1982; NRC, 2002). However, the research connecting the SLC structure with student performance is tenuous – mostly due to methodological issues. We propose to discuss strategies to evaluate the effect of smaller learning communities in U.S. high schools on student achievement and other outcome measures. This poster will use case studies to propose innovative, rigorous designs to yield more valid evaluation findings. In addition, we would identify other constructs (e.g., school climate, student self-concept and motivation) theoretically linked to achievement that could be employed as additional outcome measures. This poster will be useful for practitioners faced with evaluating smaller schools within schools in public school districts. |
| Continuous Quality Improvement Process: Reinforcing Professional Behavior While Developing Evaluation Skills in Medical Students |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Craig Timm, University of New Mexico, ctimm@salud.unm.edu |
| Paul McGuire, University of New Mexico, pmcguire@salud.unm.edu |
| Roger Jerabek, University of New Mexico, rjerabek@salud.unm.edu |
| Phil Szydlowski, University of New Mexico, pszydlowski@salud.unm.edu |
| Denise Minton, University of New Mexico, dminton@salud.unm.edu |
| Tony Ogburn, University of New Mexico, jogburn@salud.unm.edu |
| Abstract: To engage students as stakeholders, develop their professional competencies, and improve our curriculum, our medical school has implemented a Continuous Quality Improvement process within ten required courses in the first two years of the curriculum. The process involves immersing students in evaluation based on the guiding principles ethical framework adopted by the American Evaluation Association (see Guiding Principles, at http://www.eval.org/Publications/GuidingPrinciples.asp). Students volunteer and five are randomly selected to review a course evaluation report and summarize it with specific recommendations. There are standard evaluation questions used in each course as well as unique questions that course directors write. As evaluators the students are expected to 1) become competent in applying systematic inquiry in their review of data, 2) develop a summary with recommendations that demonstrates honesty and integrity, 3) integrate respect for persons in their work, 4) and consider the common good for the community. |
| Women in Administrative Segregation in Colorado |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Maureen Okeefe, Colorado Department of Corrections, maureen.okeefe@doc.state.co.us |
| Marissa Schnell, University of Colorado Colorado Springs, marissa.schnell@doc.state.co.us |
| Abstract: Only recently have researchers begun to investigate the impact of administrative segregation on male inmates; however, almost no research exists that examines female offenders in this environment. Research reflecting that women have higher rates of mental illness and more severe traumatic histories than their male counterparts implies they will be more negatively impacted by isolation. The purpose of this study is to profile Colorado women in segregation relative to women in the general population and males in segregation. Through analysis of archival data we examine reasons why women are placed in segregation by looking at their index offenses and behavioral violations. The escalating amount of women entering prison requires corrections officials to better understand their programming and security needs. Absent such research, they may continue to rely on systems that were originally developed for men and which may be counterproductive to ensuring safe and humane conditions of confinement for women. |
| Prison Rape: What Do We Know? |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kasey R Weber, University of Colorado Colorado Springs, kasey.weber@doc.state.co.us |
| Marissa Schnell, University of Colorado Colorado Springs, marissa.schnell@doc.state.co.us |
| Maureen O'Keefe, Colorado Department of Corrections, maureen.okeefe@doc.state.co.us |
| Abstract: Facilities across the country are implementing Prison Rape Elimination Act (PREA) programs; as a result, the evaluation of these programs is quickly becoming imperative. The current project analyzes the PREA incidents which occurred in Colorado state prisons from 2005 to 2007. Results indicate the frequency of incidents and further describe the types of offenses, the methods of report, incident location, individuals involved, and investigation results. Using Colorado as an example, this information will serve to help others ask informed questions and focus on important aspects of a PREA evaluation which might otherwise be overlooked. This project also identifies important data to capture and suggests how this information should be defined as it relates to PREA. As these programs multiply the need for evaluation will increase. Accordingly, it is important to share these findings and suggestions with other evaluators who may be embarking on similar evaluations. |
| Validation of a Work/Nonwork Life Employee Retention Model |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tracy Kline, RTI International, tkline@rti.org |
| Michael Schwerin, RTI International, schwerin@rti.org |
| Murrey Olmsted, RTI International, molmsted@rti.org |
| Abstract: Understanding factors affecting employee turnover intent is essential for human resources and organizational development decision makers to develop a strategic human capital plan. Wilcove, Schwerin, and Wolosin (2003) developed an exploratory work/nonwork spillover model that identifies life needs that are related to turnover intent. Additional research was warranted to determine (1) the replicability of the model, (2) the validity of the model when extended to include retention behavior; and (3) generalizability of the model to the commercial sector. This study used data from three independent samples (Sample 1: 2002 Navy QOL Survey [N = 5,114]; Sample 2: Two Southern U.S. Hospitals [N = 984]; Sample 3: Northern U.S. Commercial Trucking Company [N = 884]) to validate the work/nonwork spillover model. Results support the replicability, validity, and generalizability of the Wilcove et al. (2003) model and an approach that integrates both work and nonwork life needs in the commercial sector. |
| The Good, The Bad, and The Ugly: Outsourcing Experiences |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristin Pankey, Southern Illinois University at Carbondale, knpankey@siu.edu |
| Joel Nadler, Southern Illinois University at Carbondale, jnadler@siu.edu |
| Nicole Cundiff, Southern Illinois University at Carbondale, karim@siu.edu |
| Nicholas G Hoffman, Southern Illinois University at Carbondale, nghoff@siu.edu |
| Gargi Bhattacharya, Southern Illinois University at Carbondale, gargi@siu.edu |
| Alen Avdic, Southern Illinois University at Carbondale, alen@siu.edu |
| Abstract: The Applied Psychology program at the Southern Illinois University at Carbondale (SIUC) has been providing training in program evaluation since 1982. Perhaps the most unique part of this training is the opportunity to conduct professional evaluations as a part of Applied Research Consultants (ARC), a student-run consulting firm. Over the past few years, ARC has been hired for an increasing amount of outsourcing services and has had a variety of experiences with each of these projects and evaluations. Three specific situations stand out and can be described as “The Good, The Bad, and The Ugly.” A good experience is defined as a project that allows for the development of skills and abilities. “Bad” and “Ugly” experiences are profitable but with low educational value. This presentation will discuss these three situations, ARC’s experiences and relationships with these clients, and tips for dealing with outsourcing. |
| Broadening Participation of Minority Professionals in Program Evaluation: A Concept Mapping Workshop as Strategy |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ruth Greene, Johnson C Smith University, rgreene@jcsu.edu |
| Frank Parker, Johnson C Smith University, fparker@jcsu.edu |
| Abstract: This poster focuses on the use of concept mapping as a collaborative planning and evaluation process to engage faculty and administrators in effective collaborative efforts . The poster details the framework that was used to guide the process, outlines the major methodological issues and problems and recommends ways in which the tool can be used for training, increasing diversity in the evaluation pipeline and enhancing evaluation and planning in higher education. |
| Teaching Evaluation in the Italian Universities: Political Requirements and Practical Problems |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Serafina Pastore, University of Bari, serafinapastore@vodafone.it |
| Abstract: In the light of the current changes which have affected the University system so far, a new evaluation practice is widely standing out. It mainly aims to embrace the students' opinions about the formative services provided/received within a University degree course. Differently, evaluation for University examinations, that is assessment, does not attract the scholars' current research interests. Nowadays the learning measurement and testing frequently seem to be practiced more as a mere bureaucratic tool than as a pedagogical evaluative method because the shadow of sanction hangs over those educational practices which should lead the teacher to reflect on the student’s un/success and on his/her learning enhancement. Though learning evaluation plays an essential role in the didactic-pedagogical action, we must go beyond any functional logic and question what kind of choices can guide the evaluation action and what values can be carried out by the use of evaluation. |
| Call for Collaboration in the Midwest: Professional Evaluators in Nebraska Are Seeking Ways to Develop Local Network |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Robert Pawloski, University of Nebraska Medical Center, rwpawloski@unmc.edu |
| Barbara Jackson, University of Nebraska Medical Center, bjjackso@unmc.edu |
| Lisa St Clair, University of Nebraska Medical Center, lstclair@unmc.edu |
| Abstract: Evaluators from the University of Nebraska System (University of Nebraska at Omaha, University of Nebraska Lincoln, University of Nebraska at Kearney, and University of Nebraska Medical Center) have taken initial steps in a collaborative effort to hold a program evaluation conference in Omaha, Nebraska, early 2009. In addition to bringing quality professional development to the Omaha area, we would like to use this as an opportunity to survey the interest level of evaluation colleagues in Iowa and other nearby states about the possibility of forming a network that has potential to evolve into a regional affiliate of the AEA. This poster will present conference details as they develop. Interested evaluators should leave a card, and/or take contact information. We would also appreciate connecting with others who have had experiences and would like to share their successes and challenges in initiating such a conference, as well as launching an affiliate. |
| Tag Team Evaluation as Community Outreach: Lessons Learned From the Field |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Laura Pejsa, University of Minnesota, pejs0001@umn.edu |
| Roberta Gibbons, University of Minnesota, gibbo005@sossgw.stu.umn.edu |
| Abstract: The proposed poster will detail the process of evaluating a community-based vocational training program for pregnant and parenting teens. There are several dynamics that are unique to this project: 1) The evaluation was conducted by two graduate students; 2) The evaluation was a “tag-team” effort, with one student planning the evaluation the other carrying it out; and 3) Although the primary intended user of the evaluation was the director of the program, the evaluation was sponsored by a University of Minnesota community outreach center. This presentation will explore barriers to evaluation implementation and lessons learned throughout the process from three perspectives: 1) the authors; 2) the University outreach program; and 3) the primary intended user. In particular, we will discuss how evaluation use prevailed despite numerous structural challenges of the internship and evaluation plan. We will also show how several evaluation concepts learned from coursework were applicable in the case. |
| Exploring the Role of Evaluation in Community Foundations Grantmaking |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristianna Pettibone, MayaTech Corporation, kpettibone@mayatech.com |
| Shelly Kowalczyk, MayaTech Corporation, skowalczyk@mayatech.com |
| Ayana Perkins, MayaTech Corporation, aperkins@mayatech.com |
| Abstract: Community foundations provide an important source of funding for local health promotion programs that can that result in better health, improved quality of life, and stronger communities. Data from these programs needs to be evaluated in order to determine the nature and extent of these changes. Evaluation of these community health initiatives is critical in assessing successful programs and ensuring that these programs can be replicated and sustained. We will provide an overview of the current status of evaluation policies at Community Health Foundations in Maryland, Virginia and North Carolina. We will examine how the policies are made, the role of evaluation in funded projects and the support provided for evaluation efforts. In addition, we will identify areas in which grantees might benefit from evaluation capacity building. Finally we will examine the grantee perspective to determine challenges or issues associated with the evaluation policies and suggestions for improvement. |
| Exploring the Role of Evaluation in Community Foundations Grantmaking |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristianna Pettibone, MayaTech Corporation, kpettibone@mayatech.com |
| Shelly Kowalczyk, MayaTech Corporation, skowalczyk@mayatech.com |
| Ayana Perkins, MayaTech Corporation, aperkins@mayatech.com |
| Abstract: Community foundations provide an important source of funding for local health promotion programs that can that result in better health, improved quality of life, and stronger communities. Data from these programs needs to be evaluated in order to determine the nature and extent of these changes. Evaluation of these community health initiatives is critical in assessing successful programs and ensuring that these programs can be replicated and sustained. We will provide an overview of the current status of evaluation policies at Community Health Foundations in Maryland, Virginia and North Carolina. We will examine how the policies are made, the role of evaluation in funded projects and the support provided for evaluation efforts. In addition, we will identify areas in which grantees might benefit from evaluation capacity building. Finally we will examine the grantee perspective to determine challenges or issues associated with the evaluation policies and suggestions for improvement. |
| Evaluating Teachers' Professional Development Experiences: Year One Data from a Randomized Control Trial Study |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Andrew Newman, Mid-continent Research for Education and Learning, anewman@mcrel.org |
| Michael Phillips, University of Northern Colorado, michael.m.phillips@gmail.com |
| Abstract: Teachers’ participation in professional development (PD) activities was examined during an ongoing, large-scale randomized control trial (RCT). The intent of this two-year RCT is to investigate the use of a specific PD program focused on classroom assessment in participating Colorado elementary schools. For this poster, baseline and Year 1 data from 337 teachers were analyzed regarding professional development activities other than the RCT intervention. Teachers’ participation in PD was examined, including frequency, type of activity, and the perceived impact of the PD on classroom practice. Additionally, we explored demographic and geographic characteristics of our sample and how they relate to various types of PD in which teachers engage. The majority of PD completed by teachers focused on math and/or literacy (reading and writing); we conducted additional analyses on this subset of data. Evaluators will learn about the survey methodology, data findings, and the implications for data collection and analysis. |
| Evaluation Research as Intervention: Utilizing the Unintended Consequences of an Evaluation of Parent-Child Connectedness in a School-Based Pregnancy Prevention Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carol Lewis, University of Texas at Austin, carolmarie@mail.utexas.edu |
| Emily Wade, University of Texas at Austin, eewade@austin.utexas.edu |
| Amy Pierce, LifeWorks, amy.pierce@lifeworksweb.org |
| Abstract: Background: This evaluation assessed program components designed to enhance parent-child connectedness in a pregnancy prevention intervention for preadolescents conducted by a local non-profit agency. Methods: Parents of preadolescents in pregnancy prevention support groups were invited to attend family events and focus groups to provide feedback about how the program could improve parent-child connectedness. Results: The evaluation impacted participants in at least three significant ways: 1) Positive rapport among agency staff and parent participants was strengthened through increased communication during evaluation recruitment; 2) A preadolescent was provided mentoring services after her mother mentioned in a focus group that the child was participating in gang activity; and 3) Parents experienced the focus groups as opportunities to discuss family problems in a supportive setting. Conclusion: Researchers of human service interventions should consider the impact of their evaluations on all individuals involved to ensure ethical practices and to recognize ways that evaluation activities are a form of interventions. |
| Retaining Female Faculty in Higher Education |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| John Dantzler, University of Alabama Birmingham, dantzler@uab.edu |
| Heather Pleasants, University of Alabama, hpleasan@bamaed.ua.edu |
| Abstract: An outcomes evaluation of a National Science Foundation sponsored program to increase the recruitment and retention of female faculty in higher education will be presented. The evaluation utilized qualitative methods within a critical gender framework. A discussion of the unique challenges associated with the evaluation, as well as themes developed from the analysis will be displayed. |
| Using Photovoice to Evaluate an Early College Program: Letting Students Speak |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paula Plonski, Praxis Research Inc, pmplonski@carolina.rr.com |
| Abstract: Photovoice is a relatively new qualitative methodology utilized for program evaluation. It is a participatory community-based approach that allows program participants to communicate and document aspects of their program from their individual perspectives. Dr. Carolyn Wang, who originated Photovoice, describes the method as policy-relevant in terms of shaping and catalyzing change in programs and communities without the distortion of a predetermined paradigm. Early College students in rural North Carolina utilized this method to capture and define their program, its meaning and personal importance. This definition was a relevant piece of the evaluation, as the program had been struggling to define itself and its population in relation to the other schools in the community since its inception. This reflection of the program by the students was utilized not only to impact decisions by policymakers as an evaluation product, but also as a marketing tool to recruit new students. |
| It Works! Findings from the National Evaluation of Parents Anonymous Mutual Support Groups |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Margaret Polinsky, Parents Anonymous Inc, ppolinsky@parentsanonymous.org |
| Abstract: Parents Anonymous® Inc. is an international Network of accredited organizations that implement evidence-based Parents Anonymous® Mutual Support Groups for adults (with co-occurring programs for children and youth) with the goal of addressing risk and protective factors related to the prevention of child maltreatment. A national evaluation funded by the U.S. Office of Juvenile Justice and Delinquency Prevention (OJJDP) and conducted by the National Council on Crime and Delinquency (NCCD) was carried out from 2001-2007. Phases included a process evaluation with a stratified random sample of Network organizations, followed by a quantitative study of 206 parents new to a random sample of Parents Anonymous® groups, and then an interview study of parents attending Spanish-speaking groups. Standardized measures were used to assess change in Risk Factors and Protective Factors. This presentation will discuss the findings from the evaluation, most of which showed positive benefits related to family strengthening and child maltreatment prevention. |
| Reflections on Teaching Community Based Research: Recommendations for Preparing Collaborative, Participatory and Empowerment Evaluators |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristen Pozzoboni, University of Colorado Boulder, kristen.pozzoboni@colorado.edu |
| Ben Kirshner, University of Colorado Boulder, ben.kirshner@colorado.edu |
| Abstract: Community based research (CBR) is a collaborative research model in which university researchers and students, partner with community organizations to address needs or issues defined by the community organization. CBR creates opportunities for students to learn research and evaluation skills and benefits the partner agency by producing a product that supports or furthers the work of the community organization (Stocking & Cutforth, 2006). This session features lessons learned by university faculty and students about conducting CBR evaluations of youth development organizations. We concentrate on the methods used to prepare novice researchers and the types of support that assisted student learning as they designed and implemented projects. We also discuss challenges faced by students, such as managing ethical dilemmas, developing relationships, and learning to navigate cultural and socioeconomic differences. This session contributes to evaluation knowledge by addressing how to prepare evaluators and develop collaborative evaluations driven by community goals and interests. |
| Comparison of Propensity-Matching and IV Methods in Estimating Trauma Center Effects on Return to Work |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sergio Prada, University of Maryland Baltimore County, sprada1@umbc.edu |
| Abstract: Hospitals have difficulty justifying the expense of maintaining trauma centers without strong evidence of their effectiveness. Previous research showed that the risk of death is significantly lower when care is provided in a trauma center than in a non–trauma center. This paper explores whether for persons who were working prior to their injury, was treatment at a trauma center also more likely to result in their returning to work 3 and 12 months later. Outcomes are compared among patients treated in 18 hospitals with a trauma center and 51 hospitals non–trauma centers located in 14 states. Patients 18 to 84 years old with a moderate-to-severe injury were eligible. I use propensity-score weighting to adjust for observable differences and IV methods to adjust for unobservable differences. Covariates include severity of injury, demographics and general economic indicators. Preliminary results favor the use of IV methods in this evaluation context. |
| Evaluation and Youth Development |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brandy Pratt, Western Michigan University, brandypratt@gmail.com |
| Abstract: The youth development field is a burgeoning aspect of non-profit programming. Youth development organizations designed to assist youth in achieving their personal and social needs while building the competencies to become successful adults. As in other social services programs, youth development is fighting to establish a casual linkage between youth development programs and healthy youth development. However, this endeavor is complicated by a lack of resources, in many youth development organizations, that is needed to carry out effective program and organizational evaluation. This paper focuses on the current practice of evaluation in the emerging field of youth development while exploring its strengths, weaknesses, and consistency with evaluation standards. Best practices are suggested and areas for future development are identified. This paper will widen the awareness evaluation in the context of youth development in addition to extending and strengthening the reach of evaluation within the nonprofit sector. |
| Evaluating Smaller Learning Communities in United States High Schools |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Linda Bol, Old Dominion University, lbol@odu.edu |
| Shana Pribesh, Old Dominion University, spribesh@odu.edu |
| Christopher Fischer, Old Dominion University, cfischer@nps.k12.va.us |
| John Nunnery, Old Dominion University, jnunnery@odu.edu |
| Abstract: Creating smaller schools/learning communities (SLCs) have been advocated as a specific reform for improving high school student engagement and graduation rates (NRC, 2002). The linkages of smaller learning communities to student achievement have been found to be promising (Felner, Ginter, and Primavera, 1982; NRC, 2002). However, the research connecting the SLC structure with student performance is tenuous – mostly due to methodological issues. We propose to discuss strategies to evaluate the effect of smaller learning communities in U.S. high schools on student achievement and other outcome measures. This poster will use case studies to propose innovative, rigorous designs to yield more valid evaluation findings. In addition, we would identify other constructs (e.g., school climate, student self-concept and motivation) theoretically linked to achievement that could be employed as additional outcome measures. This poster will be useful for practitioners faced with evaluating smaller schools within schools in public school districts. |
| Work Readiness Assessment Programs: Do They Work? How Do We Know? |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lisa Rajigah, Utah State University, lrajigah@utah.gov |
| Abstract: Utah’s Department of Workforce Services through the state-wide workforce investment boards identified the need for an assessment tool to measure entry-level work readiness. The assessment is designed to measure competencies Utah employers identified for successful retention of employment, from prospective applicants and incumbent workers. The frame work that the DACUM process provided identified work ready entry-level skills, knowledge and abilities for successful obtainment and retention of employment. This paper outlines the steps of the development process for Utah’s work readiness assessment. Many claims have been made regarding the savings to employers using some type of work readiness assessment program. This review of the current literature identified the need for an independent study regarding the value of work readiness assessment programs. Are employees identified as “work ready” saving employers in terms of training time? Are there differences in training times between “work ready” applicants and a control group? |
| The Jellinek Chart Revisited |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Madelaine Ramey, Independent Consultant, madelaine.ramey@att.net |
| Abstract: The poster modifies the Jellinek Chart. The original chart lists steps on the descending path into addiction and on the ascending path to recovery. To make the Chart more compelling, some of the steps – especially those most easily translated into intervention process objectives -- are illustrated with cartoons. |
| An Example of Technical Training Utilization |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sandhya Rao, Pricewaterhousecoopers LLP, sandhya.rao@us.pwc.com |
| James Lehman, Pricewaterhousecoopers LLP, james.lehman@us.pwc.com |
| Daniel Birch, Pricewaterhousecoopers LLP, daniel.birch@us.pwc.com |
| Abstract: An important measure of the impact of training is change in job-related behaviors. Behavior change is Level 3 of the Kirkpatrick training evaluation model (1959); in it one examines changes in job behavior as a result of training. Level 3 needs to be taken together with Reaction, Learning and Business Results for a holistic understanding of training. The study was designed to understand job behavior changes as a result of sending 90 new hires to a six-week long residential technical training program. Follow-up information was gathered from the learners -- employees of PricewaterhouseCoopers, LLP-- 5 months after training. Learners were asked when they applied learning to the job, frequency of accessing training materials, opportunities for applying training and additional training needed. Learners were also asked to identify peers and supervisors who had worked with them post-training, who were in turn surveyed to understand learners' behaviors. Preliminary analyses indicated that in certain technical areas, over 75% of learners indicated that they had not had a chance to utilize any knowledge and skills from training. The results are valuable as they will be used to improve training content and usability for future offerings. |
| Using Immunization Information Systems to Assess Barriers to Full Immunization Coverage for Children |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bobby Rasulnia, Centers for Disease Control and Prevention, bba9@cdc.gov |
| Abstract: Nationally, 57% of children aged 19-35 months are not up-to-date (UTD) on the 4:3:1:3:3:1 vaccine series. Of these, 15% are one dose away from being UTD. The purpose of this assessment was to identify factors/barriers that contribute to remaining one dose away from being UTD. Thirteen IIS with high proportions of children not UTD were selected to participate. Interviews were conducted to outline challenges for series completion. In the 13 IIS, 38% were not UTD on their series. Of those, 26% were missing only one dose: mostly Pertussis-containing vaccine or Varicella. Also, the analysis revealed operational challenges such as: loss of a child to follow-up; inability of the IIS to distinguish which numbered dose is missing by vaccine type; and providers delaying 4th DTaP administration. The discussion will provide information for immunization programs on how to use IIS as a tool to evaluate program activities and identify pockets-of-need. |
| Planning and Policy in the Interpretation and Presentation of Outcomes from Randomized Controlled Trials of Community-Advocacy Programs |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fred Glantz, Kokopelli Associates LLC, fred@kokopelliassociates.com |
| Kim Sprague, Brown University, kim_sprague@brown.edu |
| Pamela Raya-Carlton, RMC Research Corporation, prayacarlton@rmccorp.com |
| Diane Schilder, Evaluation Analysis Solutions, dschilder@eval-inc.com |
| Abstract: These studies are considered critical by the department to contribute to the field by addressing the primary research question “what interventions are effective?” In addition, these studies are also some of the first to address what methods should be employed in such RCT trials, what is required in terms of policy for implementation and collaboration to conduct such efforts, and what is required to appropriately assess their implementation. Larger-scale RCTs are newer to the field of education and the purposes and methods of such studies do not appear widely understood. The goal of this paper is to present more information regarding the benefits of such studies in the context of their complications. |
| If You Know the Scientific Method, Then You Can Understand Evaluation: Building Evaluation Capacity in a Biomedical Research Setting |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Melba C Reed, National Institutes of Health, reedme@niaid.nih.gov |
| Abstract: Assessments have long been part of the decision making process at federal organizations such as the National Institutes of Health (NIH), and developing a systematic approach to program evaluation is an increasing emphasis. This presentation describes the NIH National Institute of Allergy and Infectious Diseases (NIAID) effort to build organizational capacity for providing consultation on the design and implementation of assessments and evaluations that in turn will inform efforts to advance the Institute’s biomedical research mission. The NIAID Strategic Planning and Evaluation Branch is undertaking this work, through a contract with an evaluation company, to: 1) create tools that increase work effectiveness and efficiency among Evaluation Staff, 2) provide educational materials and training on evaluation for managers of NIAID programs, and 3) develop a toolkit for evaluating initiatives and research programs. This project has implications for building capacity for evaluations within and outside the federal government. |
| Evaluating the 2nd Year Implementation and Effectiveness of the Kansas Afterschool Leadership Development Initiative to Enhance Afterschool Program Settings |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diane Betzen, Wichita State University, dmbetzen@wichita.edu |
| Tara Gregory, Wichita State University, tara.gregory@wichita.edu |
| Sarah Jolley, Wichita State University, sarah.jolley@wichita.edu |
| Emily Grant, Wichita State University, eagrant@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Debra Renollet, Wichita State University, dlrennolet@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Afterschool programs have long been seen as an appropriate avenue for creating positive youth development. However, there are few models that provide guidance to programs in creating the setting characteristics necessary to positive youth development. For this reason, the Kansas Afterschool Leadership Development Initiative (KALDI) youth leadership capacity-building intervention was implemented as a way to increase organizational effectiveness and promote efforts to achieve positive youth development outcomes through a focus on positive youth-adult relationships, youth-centered norms, and youth participation in decision-making. Results to be presented include the outcomes from measures of KALDI’s effectiveness in its second year, such as surveys of youth and adults in the programs, and observation and interview measures. Insights regarding the successes and challenges of efforts to impact settings through youth leadership capacity-building will be presented. |
| Practical Considerations for Implementing Strong Evaluation Designs in School-Based Prevention Programs |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Darcy Richardson, LeCroy and Milligan Associates Inc, darcy@lecroymilligan.com |
| Joanne Basta, LeCroy and Milligan Associates Inc, jbasta@lecroymilligan.com |
| DeeAnn Arroyo, Pima Prevention Partnership, darroyo@thepartnership.us |
| Geoff Wood, LeCroy and Milligan Associates Inc, geoff@lecroymilligan.com |
| Abstract: School-based pregnancy prevention programs face numerous challenges with implementation and sustainability. A rigorous evaluation can be advantageous for addressing these challenges because it can provide periodic information on implementation processes and progress. It can also provide reliable and valid outcome data on which program improvement and best-practices can be based. However, an effective evaluation is difficult to accomplish in a school-setting. The strongest evaluation design may be impractical based on the complexities of the program being offered, financial considerations, and/or the responsiveness of the community. The challenge with these programs is to design a rigorous evaluation that is practical in a school-setting. This session will include discussion of the design and implementation of two school-based pregnancy prevention programs, including challenges and lessons learned. The importance of the partnership between program and evaluation will be emphasized throughout this interactive session. |
| Early Childhood Services and Academic Achievement: Shedding Light on the Obstacles to Early School Success |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Eileen Robertson Rehberg, Community Research Council, errehberg@researchcouncil.net |
| Ione Farrar, Community Research Council, ifarrar@researchcouncil.net |
| Abstract: Currently, the State of Tennessee invests in pre-school and Pre-K services for quality child care and education that targets children from low-income families. These services are regarded as a public investment in the future, in which an increased number of children from low-income families are expected to succeed in their early school years. This paper and presentation will discuss the need for broader descriptive and geographically specific baseline data about children and their families at the neighborhood level to better target programs. A case study analysis of neighborhoods in Tennessee’s Hamilton County school district will demonstrate how administrative data from the health department and the school district can be used to assess the variety and diversity of factors that can have an affect on early childhood development and school performance. The case study will also discuss the sustainability of early childhood program effects and the validity of using long-term outcome measures to determine the value of these services. |
| Students as Evaluators: Encouraging Them to Use Their (Photo) Voice |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anna Robic, University of South Florida, robic@tempest.coedu.usf.edu |
| Patricia Alvarez McHatton, University of South Florida, mchatton@tempest.coedu.usf.edu |
| Stephanie Martinez, Florida Mental Health Institute, smartinez@fmhi.usf.edu |
| Abstract: A trend in evaluation has been to utilize students as evaluators of programs. What happens when the students who have participated offer little feedback? How can you access what they might not be telling you? This session presents information on this dilemma as well as suggestions on how to overcome the problem. |
| Results of a Baseline Survey of Barriers and Facilitators of Clinical Translational Research |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fredric Wolf, University of Washington, wolf@u.washington.edu |
| Douglas Brock, University of Washington, dmbrock@u.washington.edu |
| Lynne Robins, University of Washington, lynner@u.washington.edu |
| Pamela Nagasawa, University of Washington, pnaga22@u.washington.edu |
| Abstract: Barriers to the conduct of clinical and translational research were identified as part of a mixed-methods evaluation of a newly created Institute of Translational Health Sciences (ITHS). The ITHS was created as part of the National Institute of Health’s Roadmap Clinical and Translational Sciences Award program. Ten barriers identified in an earlier open-ended survey were rated in terms of 1) current level of satisfaction and 2) degree to which each represented a barrier to research and were then 3) “prioritized” in terms of urgency for addressing each (Not a priority, > 10 years, 5-10 years, 1-5 years, Year 1, Immediate). Modal responses of 215 researchers indicated an “Immediate” need to address two areas: Institutional Review Board (IRB) operations and need for more local grant support to gather preliminary data to prepare larger grant applications. All other areas received modal responses suggesting they be addressed within five years. |
| Raising Up Young Evaluators: Training Social Work Students in Evaluation Methods Through Service Learning |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Adjoa Robinson, University at Buffalo - State University of New York, adr7@buffalo.edu |
| Wooksoo Kim, University at Buffalo - State University of New York, wkim5@buffalo.edu |
| Thomas H Nochajski, University at Buffalo - State University of New York, thn@buffalo.edu |
| Abstract: This poster describes the methods used to develop evaluation practice as an integral part of professional identity in social workers, improve learning outcomes, and serve the community, through the mechanism of sequential research and evaluation courses. Instructors taught the same cohort of students over two semesters and supervised students working in groups to conduct program evaluations. Key to this process was engaging students in service-learning projects through which they planned and implemented program evaluations in partnership with local service agencies. These service-learning projects served the dual purpose of real-world learning of evaluation skills and providing community agencies with needed service. The year-long sequence afforded students the opportunity to work through all the steps in the research process. Students conducted needs assessments and process and outcome evaluations at local agencies. Pre-test data will be analyzed to assess student knowledge and attitudinal outcomes. The poster will present the course components and objectives, instructional methods, measures, and lessons learned from the first year of implementation. |
| Assessing the Influence of Non-Health Programs and Policies on the Health of Unemployed Middle-Aged Mothers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Yujia Liu, Stanford University, yujialiu@stanford.edu |
| Abstract: This study examines the impact of unemployment among middle-aged mothers and the influence of unemployment insurance compensation and welfare benefits in ameliorating a possible health effect. We use multivariate regression to analyze the National Longitudinal Survey of Youth. Controlling for socioeconomic status, and baseline information two years before experiencing unemployment, we examine the extensively validated SF-12 health measure, at the time of experiencing the unemployment and two years after experiencing it (in 2004). Preliminary results show that after dropping respondents who were unemployed for health related reasons, of out of 646 mothers over 40 years of age, those who experienced unemployment and received welfare benefits had worse health status that those who were continuously employed, while those who were unemployed but received entitlement benefits were not significantly different. Additional cohorts will be added for our final analysis. Challenges and limitations of this methodology to evaluate social policies will be discussed. |
| Sustainability in an Alternative Educational Leadership Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Rogers, University at Albany - State University of New York, bottlecap22@hotmail.com |
| Dianna Newman, University at Albany - State University of New York, dnewman@uamail.albany.edu |
| Abstract: Evaluating post-grant sustainability is an important aspect of determining the success of innovative educational programs. The current study explores this issue through the example of a three-year educational leadership grant which sought to produce new urban administrators, and to provide support for administrators beginning their career in urban districts. This latter goal was identified as a step towards building regional sustainability, and it is the focus of the current paper. In this portion of the program, early career administrators in urban districts were matched with local veteran administrators who acted as mentors for one academic year. Data sources include leadership surveys and interviews collected from program participants, and interviews and surveys collected from program mentors. Findings indicate that this program was successful in establishing a level of sustainable support for early-career urban administrators that transcended the official mentorship period. Data collection, analyses, findings, and lessons learned are discussed. |
| Evaluating the Impact of a Three-Hour Neuroscience of Addiction Curriculum on College Students’ Knowledge and Attitudes: Results From the National Institute on Drug Abuse Enters College Project |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Joyce Hartje, University of Nevada Reno, jhartje@casat.org |
| Michelle Berry, University of Nevada Reno, mberry@casat.org |
| Nancy Roget, University of Nevada Reno, nroget@casat.org |
| Wendy Woods, University of Nevada Reno, wwoods@casat.org |
| Angela Broadus, University of Nevada Reno, abroadus@casat.org |
| Abstract: The purpose of this project was to evaluate the impact of a 3-hour science of addiction research-based curriculum infusion on university undergraduate students’ knowledge and attitudes about substance use disorders. Two sections of existing introductory courses in three pre-professional disciplines (criminal justice, nursing, and social work) were selected. One section served as the curriculum infusion (implementation) group (N = 122); the other as the control (non-implementation) group (N = 178). Pre- and post-test measures were developed based on curriculum objectives to assess the effect on knowledge and attitudes. Results showed a significant increase in knowledge about addiction and a corresponding decrease in stigmatizing attitudes in 1) the implementation vs. non-implementation groups; and 2) the implementation group pretest vs. post-test measures. These findings lend support to the potential effectiveness of infusing a brief research-based addiction curriculum into existing undergraduate courses and could have long-term implications for preparing helping professionals. |
| Popular Opinion Leader (POL) On the Internet: Adapting an Evidence Based Intervention for MSM Who Seek Sex Online |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sara Gillen, Harlem United Community AIDS Center Inc, sgillen@harlemunited.org |
| George Gates, Harlem United Community AIDS Center Inc, ggates@harlemunited.org |
| Rashi Rohatgi, Harlem United Community AIDS Center Inc, rrohatgi@harlemunited.org |
| Michael Coley, Harlem United Community AIDS Center Inc, mcoley@harlemunited.org |
| Abstract: To address the increasing HIV rates among YMCSM and the need for more prolific prevention efforts, the Popular Opinion Leader (POL) program has been adapted and will be implemented and evaluated by Harlem United Community AIDS Center in New York City. POL is a community-level HIV prevention intervention designed to enlist and train the most influential peers to encourage safer sex norms and behaviors through informal risk-reduction conversations. Objectives of the formative evaluation include identifying service needs and HIV-related risk factors among community members, and innovative ways to properly adapt the POL intervention for the internet. Preliminary, formative findings, identified a community ostracized for engaging in unprotected anal sex, despite simultaneously practicing safer sex; a lack of online sex positive and harm reduction prevention messaging; effective methods for engaging men online (e.g. most popular websites), powerful alternative prevention messaging, and behavioral norms (e.g. lack of discussion concerning unsafe sex practices) |
| Evaluation of an Obesity Prevention Intervention for Food Pantry Clients: The WELL program (Walk Eat Learn Live) |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Laura Runnels, Transtria LLC, lrunnels@transtria.com |
| Julie Claus, Transtria LLC, julie@transtria.com |
| Laura Brennan Ramirez, Transtria LLC, laura@transtria.com |
| Abstract: Obesity in the United States is an economic issue related to limited social and economic resources and linked to disparities in access to healthy foods. The W.E.L.L. program (Walk. Eat. Learn. Live.) aims to improve health and reduce rates of obesity among food pantry clients through a variety of strategies including cooking classes, taste tests, walking groups, and community education. A variety of measures have been utilized to evaluate the program, including surveys, interviews, focus groups, feedback forms, and attendance sheets, which assess improvements in knowledge, beliefs, and behaviors of participants. Pantry audits measure improvements in healthy food donations. Preliminary findings indicate that while participants’ knowledge is high, they face numerous barriers to making healthy food choices. Pantry audits have shown an increase in the number of healthy foods donated by the broader community. |
| Real World Challenges to Conducting an Implementation Evaluation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Herbert Baum, Macro International Inc, herbert.m.baum@macrointernational.com |
| William Scarbrough, Macro International Inc, william.h.scarbrough.iii@macrointernational.com |
| Felicia Solomon, National Institutes of Health, solomonf@mail.nih.gov |
| Arnold Love, Independent Consultant, arnold.love@gmail.com |
| Abstract: The National Cancer Institute has been working with community partners to disseminate an evidence-based program, Body & Soul, to increase the consumption of fruits and vegetables among Christian African Americans. Macro International was hired to conduct an implementation evaluation, determining what worked well with the dissemination effort and what did not. All evaluations depend on the timely and accurate submission of data, and working with Community-based organizations presents challenges with respect to data collection efforts. This paper reports on some of those challenges and how some were overcome. |
| Planning and Policy in the Interpretation and Presentation of Outcomes from Randomized Controlled Trials of Community-Advocacy Programs |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fred Glantz, Kokopelli Associates LLC, fred@kokopelliassociates.com |
| Kim Sprague, Brown University, kim_sprague@brown.edu |
| Pamela Raya-Carlton, RMC Research Corporation, prayacarlton@rmccorp.com |
| Diane Schilder, Evaluation Analysis Solutions, dschilder@eval-inc.com |
| Abstract: These studies are considered critical by the department to contribute to the field by addressing the primary research question “what interventions are effective?” In addition, these studies are also some of the first to address what methods should be employed in such RCT trials, what is required in terms of policy for implementation and collaboration to conduct such efforts, and what is required to appropriately assess their implementation. Larger-scale RCTs are newer to the field of education and the purposes and methods of such studies do not appear widely understood. The goal of this paper is to present more information regarding the benefits of such studies in the context of their complications. |
| Prison Rape: What Do We Know? |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kasey R Weber, University of Colorado Colorado Springs, kasey.weber@doc.state.co.us |
| Marissa Schnell, University of Colorado Colorado Springs, marissa.schnell@doc.state.co.us |
| Maureen O'Keefe, Colorado Department of Corrections, maureen.okeefe@doc.state.co.us |
| Abstract: Facilities across the country are implementing Prison Rape Elimination Act (PREA) programs; as a result, the evaluation of these programs is quickly becoming imperative. The current project analyzes the PREA incidents which occurred in Colorado state prisons from 2005 to 2007. Results indicate the frequency of incidents and further describe the types of offenses, the methods of report, incident location, individuals involved, and investigation results. Using Colorado as an example, this information will serve to help others ask informed questions and focus on important aspects of a PREA evaluation which might otherwise be overlooked. This project also identifies important data to capture and suggests how this information should be defined as it relates to PREA. As these programs multiply the need for evaluation will increase. Accordingly, it is important to share these findings and suggestions with other evaluators who may be embarking on similar evaluations. |
| Women in Administrative Segregation in Colorado |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Maureen Okeefe, Colorado Department of Corrections, maureen.okeefe@doc.state.co.us |
| Marissa Schnell, University of Colorado Colorado Springs, marissa.schnell@doc.state.co.us |
| Abstract: Only recently have researchers begun to investigate the impact of administrative segregation on male inmates; however, almost no research exists that examines female offenders in this environment. Research reflecting that women have higher rates of mental illness and more severe traumatic histories than their male counterparts implies they will be more negatively impacted by isolation. The purpose of this study is to profile Colorado women in segregation relative to women in the general population and males in segregation. Through analysis of archival data we examine reasons why women are placed in segregation by looking at their index offenses and behavioral violations. The escalating amount of women entering prison requires corrections officials to better understand their programming and security needs. Absent such research, they may continue to rely on systems that were originally developed for men and which may be counterproductive to ensuring safe and humane conditions of confinement for women. |
| Early Childhood Programs and Children’s Behavior and Social Skills: A Longitudinal Investigation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sheretta Barnes, High Scope Educational Research Foundation, sbutler-barnes@highscope.org |
| Marijata Daniel-Echols, High Scope Educational Research Foundation, mdaniel-echols@highscope.org |
| Mary Delcamp, High Scope Educational Research Foundation, mdelcamp@highscope.org |
| Beth Scholz, High Scope Educational Research Foundation, bscholz@highscope.org |
| Olivia McKinney, High Scope Educational Research Foundation, oliviam@highscope.org |
| Rebecca Blisset, High Scope Educational Research Foundation, bblissett@highscope.org |
| Abstract: According to the U.S. Census Bureau (2006-2007) 54% of children between the ages of 3 and 4 are enrolled in school. Furthermore, the positive effects preschool attendance has on young children’s cognitive performance and school readiness is supported (Barnett, Lamy, & Jung, 2005). However, there is a dearth of research examining the different types of programs young children attend and the effects on socioemotional development. Therefore, it is important to explore how the type of early childhood program influences behavioral adjustment and social skills. The present study utilized a longitudinal growth model to examine demographic factors and program type (i.e. federally funded, state funded, or center based program) on children’s behavioral adjustment and social skills across a four year period. Results indicate that gender, parent’s education, food stamp assistance, and program type significantly accounted for the differences in young children’s behavioral adjustment and social skills. |
| Validation of a Work/Nonwork Life Employee Retention Model |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tracy Kline, RTI International, tkline@rti.org |
| Michael Schwerin, RTI International, schwerin@rti.org |
| Murrey Olmsted, RTI International, molmsted@rti.org |
| Abstract: Understanding factors affecting employee turnover intent is essential for human resources and organizational development decision makers to develop a strategic human capital plan. Wilcove, Schwerin, and Wolosin (2003) developed an exploratory work/nonwork spillover model that identifies life needs that are related to turnover intent. Additional research was warranted to determine (1) the replicability of the model, (2) the validity of the model when extended to include retention behavior; and (3) generalizability of the model to the commercial sector. This study used data from three independent samples (Sample 1: 2002 Navy QOL Survey [N = 5,114]; Sample 2: Two Southern U.S. Hospitals [N = 984]; Sample 3: Northern U.S. Commercial Trucking Company [N = 884]) to validate the work/nonwork spillover model. Results support the replicability, validity, and generalizability of the Wilcove et al. (2003) model and an approach that integrates both work and nonwork life needs in the commercial sector. |
| A Randomized Experiment Comparing Random to Cutoff-Based Assignment |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rudolfo Galindo, University of California Merced, rgalindo@ucmerced.edu |
| William Shadish, University of California Merced, wshadish@ucmerced.edu |
| Abstract: Several studies have attempted to show that nonrandomized experiments can approximate results from randomized experiments for testing the effects of policies and practices in fields such as education, medicine, public health, job training, and psychology. The methods used in past studies all compared results from randomized and nonrandomized experiments, the latter usually subject to adjustments like econometric selection bias models or propensity score analysis. Several years ago, a laboratory analogue paradigm was developed to improve this design by randomly assigning participants to be in a randomized or nonrandomized experiment in which they could choose their training, and where they were otherwise treated identically. The current design follows that paradigm by randomizing participants into a randomized experiment (RE) or a regression discontinuity design (RDD). Regression discontinuity designs are cut-off based designs which assign participants to a treatment dependent upon a single score on a pre-test. Preliminary results are presented here. |
| Evaluating the Partnership of Child Welfare and Mental Health Systems in Florida |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amy Vargo, University of South Florida, avargo@fmhi.usf.edu |
| Patty Sharrock, University of South Florida, psharrock@fmhi.usf.edu |
| Abstract: Through a collaboration between Florida’s child welfare and Medicaid mental health systems, the Child Welfare Prepaid Mental Health Plan (CW-PMHP) was implemented nearly statewide in February 2007. The CW-PMHP evaluation methodology has evolved to include three components. In Year One, an implementation analysis captured the early experiences that various stakeholders had in the development and early operation of the plan, and a mail survey obtained foster parents’ perspectives regarding the mental health needs of children and their experiences in accessing mental health services. In Year Two, a quality of care component was added consisting of interviews with mental health service providers. Interviews with foster parents will be added in Year Three to glean qualitative data to further investigate the quantitative findings of the mail survey. This multi-level, multi-method evaluation approach provides a comprehensive view of the implementation, its effects on multiple stakeholders, and methodological implications for future evaluative work. |
| A Long-Distance Evaluation of an In-Home Family Services Model: Planning Processes, Successes and Barriers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stephanie Ingram, Boys Town, ingrams@boystown.org |
| Tanya Shaw, Boys Town, shawt@boystown.org |
| Kristin Duppong Hurley, University of Nebraska Lincoln, kdupponghurley2@unl.edu |
| Katy Casey, University of Nebraska Lincoln, katyjcasey@yahoo.com |
| Abstract: Family service interventions are under increasing pressure from funding agencies and clients to demonstrate their effectiveness. Specifically, agencies want to gain a better understanding of what works, for which clients, under what conditions. This difficult task becomes even more complex when conducting a large scale evaluation of at-risk populations in a collaborative research effort, as there are many challenges and processes to plan, including: how to select appropriate measures, how to train staff in complex data collection procedures, how to conduct program fidelity, how to maintain client confidentiality between the service provider and an university research center, and how to monitor data collection, all from 1500 miles away from the intervention site. This paper will discuss the planning processes, challenges and ways to overcome barriers of a long-distance, complex evaluation to help other researchers planning large scale evaluations across the miles. |
| Striking a Balance Between Public Health Protection and Individual Freedom: Using Diffusion of Innovation Theory in Vaccine Exemption Policy |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jo Ann Shoup, Kaiser Permanente, jo.ann.shoup@kp.org |
| Abstract: Vaccine policy is the backbone of successful elimination of childhood infectious disease, and rising political controversy. School vaccine exemption type and date of enactment by each state was obtained through review of state laws available from state government web sites, consumer web sites, LexisNexis, and consultation with each state Health Department. By 2003, 18 (36%) states had adopted vaccine philosophical exemption policies. The cumulative number of adopter states was graphed against time, expressed as an adoption curve. Next, the date of exemption was plotted on to the map of the United States and categorized into five stages of adoption (innovator; early adopter; early majority; late majority; or laggard), resulting in 1 innovator state; 2 early adopter states; 7 early majority states; 4 late majority states; and 4 laggard states. The results were analyzed for regional adoption and model of adoption. Using diffusion of innovation model revealed patterns of geographical adoption. |
| Urbanicity, Race/Ethnicity, and School Level Analyses from the Communities In Schools (CIS) National Evaluation: Informing Policy and Practice |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Julie Gdula, ICF International, jgdula@icfi.com |
| Susan Siegel, Communities In Schools Inc, ssiegel@cisnet.org |
| Abstract: Communities In Schools, Inc. (CIS) is a nationwide initiative to connect schools with community resources to help at-risk students successfully learn, stay in school, and prepare for life. As part of a five-year comprehensive national evaluation of CIS, various studies were conducted to determine the impact of the CIS model on school-level outcomes and the extent of program implementation. In addition to the main findings, subgroup analyses were conducted to assess the performance of different types of CIS schools. These subgroups were divided by urbanicity (urban, suburban, rural), race/ethnicity (African American, Hispanic/Latino, white, and diverse), and school level (elementary, middle, high). To better understand the context of the findings, profiles of each subgroup were created based on survey responses regarding such domains as planning, needs assessment, service delivery, and monitoring. Results are presented as they pertain to the different challenges of implementing the CIS model to serve different student subgroups. |
| Examining Time-to-Degree within a Multidisciplinary, Multi-Institutional Graduate Training Program--Successes and Challenges |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Dylan Flather, University of Colorado Boulder, dylan.flather@colorado.edu |
| Kristin Kusmierek, University of Michigan, kkusmierek@stanfordalumni.org |
| Tyler Silverman, University of Colorado Boulder, tsilverman1@gmail.com |
| Mary Anne Carroll, University of Michigan, mcarroll@umich.edu |
| David Karowe, Western Michigan University, karowe@wmich.edu |
| Abstract: Time-to-degree is particularly important in interdisciplinary programs, often thought to add time compared to traditional programs. The Biosphere-Atmosphere Research and Training (BART) program, a project of the National Science Foundation’s Integrative Graduate Education and Research Traineeship (IGERT) initiative, is an interdisciplinary program with participants from over a dozen disciplines and universities. This analysis compares time-to-degree of participants in BART’s two-year supplemental program against the time for their non-BART peers from corresponding home institutions. The preliminary results show that former BART students graduate in about the same time as non-BART students. |
| Building Capacity in Science: Evaluation of a Science Inquiry Center |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bridget Cotner, University of South Florida, bcotner@cas.usf.edu |
| Chrystal Smith, University of South Florida, casmith5@cas.usf.edu |
| Kathryn Borman, University of South Florida, kborman@cas.usf.edu |
| Abstract: This evaluation used an objective oriented approach to determine if the Center's projects are accomplishing the four goals set forth when founded. These four goals are 1) provide research opportunities in science education and learning, 2) rebuild and diversify the human resource base for science, technology, engineering and mathematic (STEM) education, 3) increase the number of K-12 STEM educators with the knowledge to implement standards-based science instruction, and 4) create synergy between research and practice through research based practice and translate results into practical suggestions for science educators. The evaluation provided formative and summative information to the Center to assist them in their quest to build capacity in science. |
| Real World Challenges to Conducting an Implementation Evaluation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Herbert Baum, Macro International Inc, herbert.m.baum@macrointernational.com |
| William Scarbrough, Macro International Inc, william.h.scarbrough.iii@macrointernational.com |
| Felicia Solomon, National Institutes of Health, solomonf@mail.nih.gov |
| Arnold Love, Independent Consultant, arnold.love@gmail.com |
| Abstract: The National Cancer Institute has been working with community partners to disseminate an evidence-based program, Body & Soul, to increase the consumption of fruits and vegetables among Christian African Americans. Macro International was hired to conduct an implementation evaluation, determining what worked well with the dissemination effort and what did not. All evaluations depend on the timely and accurate submission of data, and working with Community-based organizations presents challenges with respect to data collection efforts. This paper reports on some of those challenges and how some were overcome. |
| What is Gained by Conducting Randomized Controlled Trials in Education? What is Known About Policy Regarding Conducting Them? |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kim Sprague, Brown University, kim_sprague@brown.edu |
| Candy Miller, Brown University, millerfkcm@comcast.net |
| Abstract: The Education Alliance at Brown University is conducting a randomized controlled trial (RCT) evaluating the effectiveness of two adolescent literacy interventions on the reading achievement of students identified as “struggling readers” (those scoring two levels below grade level), as implemented by two school districts in western Massachusetts. The districts were together awarded one of eight Striving Readers grants by the US Department of Education to conduct this work in collaboration with their evaluator. These studies are considered critical by the department to contribute to the field of adolescent literacy by addressing the primary research question of what interventions are effective. In addition, these studies are also some of the first to address what methods should be employed in such RCT trials, what is required in terms of policy for implementation and collaboration to conduct such efforts, and what is required to appropriately assess their implementation. |
| Planning and Policy in the Interpretation and Presentation of Outcomes from Randomized Controlled Trials of Community-Advocacy Programs |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fred Glantz, Kokopelli Associates LLC, fred@kokopelliassociates.com |
| Kim Sprague, Brown University, kim_sprague@brown.edu |
| Pamela Raya-Carlton, RMC Research Corporation, prayacarlton@rmccorp.com |
| Diane Schilder, Evaluation Analysis Solutions, dschilder@eval-inc.com |
| Abstract: These studies are considered critical by the department to contribute to the field by addressing the primary research question “what interventions are effective?” In addition, these studies are also some of the first to address what methods should be employed in such RCT trials, what is required in terms of policy for implementation and collaboration to conduct such efforts, and what is required to appropriately assess their implementation. Larger-scale RCTs are newer to the field of education and the purposes and methods of such studies do not appear widely understood. The goal of this paper is to present more information regarding the benefits of such studies in the context of their complications. |
| Call for Collaboration in the Midwest: Professional Evaluators in Nebraska Are Seeking Ways to Develop Local Network |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Robert Pawloski, University of Nebraska Medical Center, rwpawloski@unmc.edu |
| Barbara Jackson, University of Nebraska Medical Center, bjjackso@unmc.edu |
| Lisa St Clair, University of Nebraska Medical Center, lstclair@unmc.edu |
| Abstract: Evaluators from the University of Nebraska System (University of Nebraska at Omaha, University of Nebraska Lincoln, University of Nebraska at Kearney, and University of Nebraska Medical Center) have taken initial steps in a collaborative effort to hold a program evaluation conference in Omaha, Nebraska, early 2009. In addition to bringing quality professional development to the Omaha area, we would like to use this as an opportunity to survey the interest level of evaluation colleagues in Iowa and other nearby states about the possibility of forming a network that has potential to evolve into a regional affiliate of the AEA. This poster will present conference details as they develop. Interested evaluators should leave a card, and/or take contact information. We would also appreciate connecting with others who have had experiences and would like to share their successes and challenges in initiating such a conference, as well as launching an affiliate. |
| Local Outcomes from the Rhode Island State Incentive Grant |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| John Stevenson, University of Rhode Island, jsteve@uri.edu |
| Nathan Stein, University of Rhode Island, nstein@mail.uri.edu |
| Israel Beren, University of Rhode Island, chezsruli@mail.uri.edu |
| Paul Florin, University of Rhode Island, pflorin@uri.edu |
| Abstract: This paper looks at outcomes from science-based substance abuse prevention initiatives funded in seven school districts in RI. Despite the development of a state-level policy linking local and state evaluations in a collaborative approach, evidence for local program effects was limited. Changes over time from 6th grade pre-tests to 7th grade post-tests on risk and protective factor measures (MANOVA and Chi-square analyses) were negligible. However, when “high implementation” schools were compared to “low implementation” schools as well as to the statewide picture, there were notably greater reductions in self-reported substance use for 8th graders in high implementation schools. Comparable fidelity of delivery was achieved but youth participation rates were starkly different. Conclusions are drawn about the importance of the delivery system for prevention (curricular insertion vs. after-school) as compared to the particular evidence-based curriculum (All Stars, Project Northland, Life Skills Training) for producing effects with practical significance for a community. |
| Local Outcomes from the Rhode Island State Incentive Grant |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| John Stevenson, University of Rhode Island, jsteve@uri.edu |
| Nathan Stein, University of Rhode Island, nstein@mail.uri.edu |
| Israel Beren, University of Rhode Island, chezsruli@mail.uri.edu |
| Paul Florin, University of Rhode Island, pflorin@uri.edu |
| Abstract: This paper looks at outcomes from science-based substance abuse prevention initiatives funded in seven school districts in RI. Despite the development of a state-level policy linking local and state evaluations in a collaborative approach, evidence for local program effects was limited. Changes over time from 6th grade pre-tests to 7th grade post-tests on risk and protective factor measures (MANOVA and Chi-square analyses) were negligible. However, when “high implementation” schools were compared to “low implementation” schools as well as to the statewide picture, there were notably greater reductions in self-reported substance use for 8th graders in high implementation schools. Comparable fidelity of delivery was achieved but youth participation rates were starkly different. Conclusions are drawn about the importance of the delivery system for prevention (curricular insertion vs. after-school) as compared to the particular evidence-based curriculum (All Stars, Project Northland, Life Skills Training) for producing effects with practical significance for a community. |
| A Utilization-Focused Evaluation of a Training Program for Early Intervention Service Providers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Barbara Stroud, Los Angeles Child Guidance Clinic, bstroud@lacgc.org |
| Rupanwita Gupta, Claremont Graduate University, rupanwita@gmail.com |
| Tony Wu, Los Angeles Child Guidance Clinic, tonyw@lacgc.org |
| Abstract: This paper discusses the evaluation of a training program for professionals in early intervention (0-5 mental health services) conducted by a child guidance clinic. Trainings and concurrent supervisions were intended to equip participants to serve clients (0-5) and their families effectively. Surveys elicited quantitative and qualitative feedback to learn about participants’ perceptions of the workshops and utilization of presented topics. Follow-up telephone interviews with eight participants were completed to understand the influence of the training program on changes in clinical practice. The trainings and trainers were rated highly on several aspects. Additionally, phone interview participants mentioned that training topics had informed them of factors to pay attention to when assisting clients, while supervision sessions had provided additional support. Noteworthy are this evaluation's implications, given the dearth of ongoing professional development for early intervention professionals. |
| The Domino Effect: Informing Program Policy Through Evaluative Practices- A Day Treatment Program Case Study |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jacquelyn Christensen, Los Angeles Child Guidance Clinic, jacquelynsheac@hotmail.com |
| Barbara Stroud, Los Angeles Child Guidance Clinic, bstroud@lacgc.org |
| Tony Wu, Los Angeles Child Guidance Clinic, tonyw@lacgc.org |
| Abstract: This paper highlights the importance of evaluating early intervention programs, and provides a case study that describes how utilization-focused evaluation not only resulted in increased program accountability, but also elicited program improvement at both the program level as well as overall organization policy. In evaluating a nonprofit early childhood day-treatment program, it was demonstrated that parental stress was significantly reduced after children’s participation in the program. Data from this preliminary study suggested the benefits of an early intervention program in addition to how the outcomes were utilized to guide and inform administrative policy in terms of program implementation and service delivery. Implications from this study revealed the necessity of evaluation and speak to the reciprocal interaction between evaluation and program improvement. |
| Utilizing Reflection to Promote Learning and Change during Evaluation: One School District’s Experience |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Aubrie Swan, Rutgers the State University of New Jersey, aswan@eden.rutgers.edu |
| Abstract: Designing evaluation tools that promote reflection by evaluation participants can greatly assist evaluators in eliciting useful ideas regarding the impact of program changes and how the program can be improved, while promoting future program awareness and learning by participants. The use of reflection by participants is likely to increase the chance changes will be made and sustained. This poster will present the topic of reflection in evaluation through the context of an evaluation of a school district’s state-funded preschool and Kindergarten program. A request for this evaluation was made by district administrators to examine grant- and state-mandated changes to these programs, and to learn how to better serve the district’s increasing low-income population. However, through the open reflection of teachers during focus groups, unexpected consequences of the program changes emerged, such as how the recent changes had negatively impacted instruction, provoking further reflection and learning by the Superintendent and teachers. |
| Continuous Quality Improvement Process: Reinforcing Professional Behavior While Developing Evaluation Skills in Medical Students |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Craig Timm, University of New Mexico, ctimm@salud.unm.edu |
| Paul McGuire, University of New Mexico, pmcguire@salud.unm.edu |
| Roger Jerabek, University of New Mexico, rjerabek@salud.unm.edu |
| Phil Szydlowski, University of New Mexico, pszydlowski@salud.unm.edu |
| Denise Minton, University of New Mexico, dminton@salud.unm.edu |
| Tony Ogburn, University of New Mexico, jogburn@salud.unm.edu |
| Abstract: To engage students as stakeholders, develop their professional competencies, and improve our curriculum, our medical school has implemented a Continuous Quality Improvement process within ten required courses in the first two years of the curriculum. The process involves immersing students in evaluation based on the guiding principles ethical framework adopted by the American Evaluation Association (see Guiding Principles, at http://www.eval.org/Publications/GuidingPrinciples.asp). Students volunteer and five are randomly selected to review a course evaluation report and summarize it with specific recommendations. There are standard evaluation questions used in each course as well as unique questions that course directors write. As evaluators the students are expected to 1) become competent in applying systematic inquiry in their review of data, 2) develop a summary with recommendations that demonstrates honesty and integrity, 3) integrate respect for persons in their work, 4) and consider the common good for the community. |
| Strengthening the Evaluation Capacity of the National Institutes of Health |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brandie Taylor, National Institutes of Health, taylorbr@mail.nih.gov |
| Rosanna Ng, National Institutes of Health, ngr@mail.nih.gov |
| Meryl Sufian, National Institutes of Health, sufianm@mail.nih.gov |
| Madeleine Wallace, National Institutes of Health, wallacem2@mail.nih.gov |
| Yuanting Zhang, National Institutes of Health, zhangyuant@mail.nih.gov |
| Abstract: A comprehensive approach has been developed to strengthen the evaluation capacity of the National Institutes of Health (NIH). Led by the Evaluation Branch (EB), a part of the NIH Office of Portfolio Analysis and Strategic Initiatives, efforts are being implemented to build and sustain support for evaluation and evaluation capacity. This poster will showcase these activities (e.g., strategies to maximize the effective use of dedicated funding for evaluations, education and outreach activities, technical support and guidance, collaborative efforts to foster evaluation skills and knowledge) and highlight some of the challenges and lessons learned. |
| A Fresh Perspective on Evaluating a School-University Partnership |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Steven Thomas, College of Charleston, thomassl@cofc.edu |
| Krysti Webber, College of Charleston, kristina_webber@charleston.k12.sc.us |
| Stephen Jurs, Independent Consultant, sgjurs@charter.net |
| Paula Egelson, College of Charleston, egelsonp@cofc.edu |
| Abstract: This program evaluation report includes information about a partnership effort between the College of Charleston and the Charleston County School District to improve the student outcomes at Burke High School. During the 2007-2008 school year, quantitative and qualitative data were collected to investigate the development of the partnership relationship. Prior evaluation led to several conclusions indicating that stronger relationships have developed between university and school personal. The 2007-2008 report will employ social network analysis to further contextualize the relationships that have developed as a result of this partnership. |
| An Evaluation of the Mississippi Women In Agriculture: Annie Project |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paula Threadgill, Mississippi State University, paulat@ext.msstate.edu |
| Michael Newman, Mississippi State University, mnewman@humansci.msstate.edu |
| Abstract: The purpose of the Mississippi Women In Agriculture (MWIA) program was to increase the knowledge and skills of farm women in business management. The objectives of the program were: (1) provide workshops on risk management topics, (2) provide videoconferences that address the issues, and (3)develop a MWIG website that contain information that participants can review and reinforce concepts learned at regional workshops. Three regional three-day workshops and one regional one-day workshop were conducted across the state. Fifty-one pre and posttests were collected from the three regional three-day workshops and 13 were collected from the one-day workshop. All of the participants were female; the participant’s age ranged 40 to 59 years old. Eleven percent was single, 84% were married and four % were widowed. T-Tests were conducted on the means to reveal the significant differences between the pre and posttest. Participants indicated their knowledge level increased on all topics. |
| Continuous Quality Improvement Process: Reinforcing Professional Behavior While Developing Evaluation Skills in Medical Students |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Craig Timm, University of New Mexico, ctimm@salud.unm.edu |
| Paul McGuire, University of New Mexico, pmcguire@salud.unm.edu |
| Roger Jerabek, University of New Mexico, rjerabek@salud.unm.edu |
| Phil Szydlowski, University of New Mexico, pszydlowski@salud.unm.edu |
| Denise Minton, University of New Mexico, dminton@salud.unm.edu |
| Tony Ogburn, University of New Mexico, jogburn@salud.unm.edu |
| Abstract: To engage students as stakeholders, develop their professional competencies, and improve our curriculum, our medical school has implemented a Continuous Quality Improvement process within ten required courses in the first two years of the curriculum. The process involves immersing students in evaluation based on the guiding principles ethical framework adopted by the American Evaluation Association (see Guiding Principles, at http://www.eval.org/Publications/GuidingPrinciples.asp). Students volunteer and five are randomly selected to review a course evaluation report and summarize it with specific recommendations. There are standard evaluation questions used in each course as well as unique questions that course directors write. As evaluators the students are expected to 1) become competent in applying systematic inquiry in their review of data, 2) develop a summary with recommendations that demonstrates honesty and integrity, 3) integrate respect for persons in their work, 4) and consider the common good for the community. |
| Context, Culture and Change: Examining Racial and Ethnic Health Disparities Through Qualitative Evaluation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pattie Tucker, Centers for Disease Control and Prevention, ptucker1@cdc.gov |
| C Shannon Griffin Blake, Centers for Disease Control and Prevention, sgriffinblake@cdc.gov |
| Amanda Navarro, Centers for Disease Control and Prevention, anavarro2@cdc.gov |
| Shawna Howell, Centers for Disease Control and Prevention, showell@cdc.gov |
| Abstract: In 1999, the Centers for Disease Control and Prevention initiated Racial and Ethnic Approaches to Community Health (REACH) 2010 as the cornerstone of its efforts to eliminate racial and ethnic health disparities. REACH aimed to examine and address individual and environmental contexts which included differences across race, ethnicity, education, income, and geographic location. The uniqueness of REACH was its diversity in approaches and strategies and its emphasis on community participation in all aspects of program development, implementation and evaluation. Qualitative methods were utilized to elucidate the three themes presented in this poster: (1) understanding the broader environmental contexts impacting disparities, (2) recognition of culture as an important factor of influence on coalitions, and (3) documenting change across strategies to eliminate disparities. This presentation of four different projects will provide a qualitative framework for understanding the problem and identifying key elements that ensure program sustainability and achieve greater public health impact. |
| Key Factors for Effective Implementation of School Curricula: Achieving Success in Reading First Outcomes |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Wendy Wolfersteig, Arizona State University, wendy.wolfersteig@asu.edu |
| Arturo Valdivia, Arizona State University, arturo.valdivia@asu.edu |
| Lynn Katz, Arizona State University, lynn.katz@asu.edu |
| Abstract: The goal of Reading First is to impact student achievement via improved instructional practice in the classroom. Eight factors identified in the literature that can impact effective implementation of this program were examined using evaluation measures including test scores, surveys, and fidelity checklists. Since students interact primarily with teachers, this study focused on the impact teachers' instruction, motivation, and beliefs had on students’ reading performance. The ancillary roles of principals, reading specialists, reading coaches, district coordinators, and site-based teams were examined with regard to their roles, support and resources provided. Using a series of multiple regression analyses, this study examined data for 20,000 kindergarten, first, second and third grade students across 61 Arizona schools for the first three years of the program's implementation. Initial findings identified the crucial role the principals’ leadership provides in their ability to enhance teachers’ positive attitudes and instructional improvements as significant predictors of students' performance. |
| Evaluating the Partnership of Child Welfare and Mental Health Systems in Florida |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amy Vargo, University of South Florida, avargo@fmhi.usf.edu |
| Patty Sharrock, University of South Florida, psharrock@fmhi.usf.edu |
| Abstract: Through a collaboration between Florida’s child welfare and Medicaid mental health systems, the Child Welfare Prepaid Mental Health Plan (CW-PMHP) was implemented nearly statewide in February 2007. The CW-PMHP evaluation methodology has evolved to include three components. In Year One, an implementation analysis captured the early experiences that various stakeholders had in the development and early operation of the plan, and a mail survey obtained foster parents’ perspectives regarding the mental health needs of children and their experiences in accessing mental health services. In Year Two, a quality of care component was added consisting of interviews with mental health service providers. Interviews with foster parents will be added in Year Three to glean qualitative data to further investigate the quantitative findings of the mail survey. This multi-level, multi-method evaluation approach provides a comprehensive view of the implementation, its effects on multiple stakeholders, and methodological implications for future evaluative work. |
| Evaluation Research as Intervention: Utilizing the Unintended Consequences of an Evaluation of Parent-Child Connectedness in a School-Based Pregnancy Prevention Program |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carol Lewis, University of Texas at Austin, carolmarie@mail.utexas.edu |
| Emily Wade, University of Texas at Austin, eewade@austin.utexas.edu |
| Amy Pierce, LifeWorks, amy.pierce@lifeworksweb.org |
| Abstract: Background: This evaluation assessed program components designed to enhance parent-child connectedness in a pregnancy prevention intervention for preadolescents conducted by a local non-profit agency. Methods: Parents of preadolescents in pregnancy prevention support groups were invited to attend family events and focus groups to provide feedback about how the program could improve parent-child connectedness. Results: The evaluation impacted participants in at least three significant ways: 1) Positive rapport among agency staff and parent participants was strengthened through increased communication during evaluation recruitment; 2) A preadolescent was provided mentoring services after her mother mentioned in a focus group that the child was participating in gang activity; and 3) Parents experienced the focus groups as opportunities to discuss family problems in a supportive setting. Conclusion: Researchers of human service interventions should consider the impact of their evaluations on all individuals involved to ensure ethical practices and to recognize ways that evaluation activities are a form of interventions. |
| Strengthening the Evaluation Capacity of the National Institutes of Health |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brandie Taylor, National Institutes of Health, taylorbr@mail.nih.gov |
| Rosanna Ng, National Institutes of Health, ngr@mail.nih.gov |
| Meryl Sufian, National Institutes of Health, sufianm@mail.nih.gov |
| Madeleine Wallace, National Institutes of Health, wallacem2@mail.nih.gov |
| Yuanting Zhang, National Institutes of Health, zhangyuant@mail.nih.gov |
| Abstract: A comprehensive approach has been developed to strengthen the evaluation capacity of the National Institutes of Health (NIH). Led by the Evaluation Branch (EB), a part of the NIH Office of Portfolio Analysis and Strategic Initiatives, efforts are being implemented to build and sustain support for evaluation and evaluation capacity. This poster will showcase these activities (e.g., strategies to maximize the effective use of dedicated funding for evaluations, education and outreach activities, technical support and guidance, collaborative efforts to foster evaluation skills and knowledge) and highlight some of the challenges and lessons learned. |
| Achieving a Successful Semester-Long Program Evaluation Experience: Tips from a One-on-One Course |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carrie Brown, Saint Louis University, cbrown53@slu.edu |
| Angela Walmsley, Saint Louis University, walmsley@slu.edu |
| Abstract: Graduate students can feel discouraged to conduct a program evaluation by thinking it will take too long to complete. The graduate student and professor involved with this presentation recently completed a program evaluation together, from start to finish, within a semester time-frame. The evaluation was conducted to fulfill the student’s requirement for a one-on-one program evaluation course, and investigated an economics educational program for 3rd and 4th graders at a children’s museum. Both qualitative and quantitative methods were implemented. Methods triangulation was employed through paper-and-pencil surveys, telephone interviews, e-mail correspondence, and site visits. Participant triangulation was also utilized. Upon conclusion of the semester, a final written report was given to the museum. Graduate students interested in conducting a program evaluation, and professors interested in giving a program evaluation course, may find this experience helpful in thinking of ways to develop a semester-long program evaluation that is both complete and satisfying. |
| A Fresh Perspective on Evaluating a School-University Partnership |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Steven Thomas, College of Charleston, thomassl@cofc.edu |
| Krysti Webber, College of Charleston, kristina_webber@charleston.k12.sc.us |
| Stephen Jurs, Independent Consultant, sgjurs@charter.net |
| Paula Egelson, College of Charleston, egelsonp@cofc.edu |
| Abstract: This program evaluation report includes information about a partnership effort between the College of Charleston and the Charleston County School District to improve the student outcomes at Burke High School. During the 2007-2008 school year, quantitative and qualitative data were collected to investigate the development of the partnership relationship. Prior evaluation led to several conclusions indicating that stronger relationships have developed between university and school personal. The 2007-2008 report will employ social network analysis to further contextualize the relationships that have developed as a result of this partnership. |
| Prison Rape: What Do We Know? |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kasey R Weber, University of Colorado Colorado Springs, kasey.weber@doc.state.co.us |
| Marissa Schnell, University of Colorado Colorado Springs, marissa.schnell@doc.state.co.us |
| Maureen O'Keefe, Colorado Department of Corrections, maureen.okeefe@doc.state.co.us |
| Abstract: Facilities across the country are implementing Prison Rape Elimination Act (PREA) programs; as a result, the evaluation of these programs is quickly becoming imperative. The current project analyzes the PREA incidents which occurred in Colorado state prisons from 2005 to 2007. Results indicate the frequency of incidents and further describe the types of offenses, the methods of report, incident location, individuals involved, and investigation results. Using Colorado as an example, this information will serve to help others ask informed questions and focus on important aspects of a PREA evaluation which might otherwise be overlooked. This project also identifies important data to capture and suggests how this information should be defined as it relates to PREA. As these programs multiply the need for evaluation will increase. Accordingly, it is important to share these findings and suggestions with other evaluators who may be embarking on similar evaluations. |
| Benefits and Opportunities in Implicit Evaluation: A Case Study in Approaching Sustainability at the Australian National University |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Su Wild River, Australian National University, su.wild-river@anu.edu.au |
| Abstract: Implicit evaluation policy and practice can be successful in achieving both robust evaluations and substantive outcomes. This has been the case at the Australian National University (ANU), which completes a five-year environmental policy in 2008. The policy includes explicit targets, each of which imply an evaluation methodology. The ANUgreen team implements this environment policy and is largely free to establish its own evaluation practices. Results are reported regularly to an Environment Management Planning Committee and the Vice Chancellor. There are at least five strategies that mitigate the risks of implicit evaluation policies and self-evaluation practices. • Striving for excellence, • Taking part in external environmental programs, • Nominating for grants and awards, • Being transparent and inclusive in sustainability practices, and • Publishing methods and results as peer reviewed papers. The paper describes the evaluation practices and outcomes and defends a grounded and gradual move towards formal evaluation. |
| Issues Faced when Inquiring with Indigenous Peoples: Views and Thoughts from Different Authors |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Laura Jimenez, Brigham Young University, ljimenezron@gmail.com |
| David Williams, Brigham Young University, ddwbyu@gmail.com |
| Abstract: • Effective action requires understanding the many (e.g., physical and social) contexts in which we seek to act, including what works and what does not (Chambers, 1997) within a particular culture. The general purpose of this research is to identify, understand, and better address the issues inquirers and their collaborating participants face when doing inquiry with indigenous peoples. The study explores several issues to prioritize when doing research or evaluation among people of other cultures. • We need publications that inform inquirers about key issues when evaluating & researching with Indigenous people • The question is: “How should researchers and evaluators collaborate with indigenous peoples most effectively?” |
| Evaluating the 2nd Year Implementation and Effectiveness of the Kansas Afterschool Leadership Development Initiative to Enhance Afterschool Program Settings |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diane Betzen, Wichita State University, dmbetzen@wichita.edu |
| Tara Gregory, Wichita State University, tara.gregory@wichita.edu |
| Sarah Jolley, Wichita State University, sarah.jolley@wichita.edu |
| Emily Grant, Wichita State University, eagrant@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Debra Renollet, Wichita State University, dlrennolet@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Afterschool programs have long been seen as an appropriate avenue for creating positive youth development. However, there are few models that provide guidance to programs in creating the setting characteristics necessary to positive youth development. For this reason, the Kansas Afterschool Leadership Development Initiative (KALDI) youth leadership capacity-building intervention was implemented as a way to increase organizational effectiveness and promote efforts to achieve positive youth development outcomes through a focus on positive youth-adult relationships, youth-centered norms, and youth participation in decision-making. Results to be presented include the outcomes from measures of KALDI’s effectiveness in its second year, such as surveys of youth and adults in the programs, and observation and interview measures. Insights regarding the successes and challenges of efforts to impact settings through youth leadership capacity-building will be presented. |
| Understanding Service Needs of Older Adults |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Trish McKenney, Wichita State University, tlmckenney@wichita.edu |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Greg Meissen, Wichita State University, greg.meissen@wichita.edu |
| Abstract: Understanding and responding to the needs of older adults is a growing interest due to changing demographics, economic turns, and new ways to interact socially. At the same time it can be difficult to understand older adults’ needs for social interaction and community integration other than at a “surface level”. To overcome this challenge, this evaluation explores the needs of older adults in a Midwest community using a mixed method approach including personal interviews with older adults and key informants, focus groups with older adults and caregivers, and archival data. Through these methods, 200 participants provided feedback on available services for seniors as well as input on what they would like to see offered to older adults in the future. Results of this mixed method approach will be presented, including how they have helped guide a local agency and its partners’ plan for future older adult services. |
| Results of a Baseline Survey of Barriers and Facilitators of Clinical Translational Research |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fredric Wolf, University of Washington, wolf@u.washington.edu |
| Douglas Brock, University of Washington, dmbrock@u.washington.edu |
| Lynne Robins, University of Washington, lynner@u.washington.edu |
| Pamela Nagasawa, University of Washington, pnaga22@u.washington.edu |
| Abstract: Barriers to the conduct of clinical and translational research were identified as part of a mixed-methods evaluation of a newly created Institute of Translational Health Sciences (ITHS). The ITHS was created as part of the National Institute of Health’s Roadmap Clinical and Translational Sciences Award program. Ten barriers identified in an earlier open-ended survey were rated in terms of 1) current level of satisfaction and 2) degree to which each represented a barrier to research and were then 3) “prioritized” in terms of urgency for addressing each (Not a priority, > 10 years, 5-10 years, 1-5 years, Year 1, Immediate). Modal responses of 215 researchers indicated an “Immediate” need to address two areas: Institutional Review Board (IRB) operations and need for more local grant support to gather preliminary data to prepare larger grant applications. All other areas received modal responses suggesting they be addressed within five years. |
| Key Factors for Effective Implementation of School Curricula: Achieving Success in Reading First Outcomes |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Wendy Wolfersteig, Arizona State University, wendy.wolfersteig@asu.edu |
| Arturo Valdivia, Arizona State University, arturo.valdivia@asu.edu |
| Lynn Katz, Arizona State University, lynn.katz@asu.edu |
| Abstract: The goal of Reading First is to impact student achievement via improved instructional practice in the classroom. Eight factors identified in the literature that can impact effective implementation of this program were examined using evaluation measures including test scores, surveys, and fidelity checklists. Since students interact primarily with teachers, this study focused on the impact teachers' instruction, motivation, and beliefs had on students’ reading performance. The ancillary roles of principals, reading specialists, reading coaches, district coordinators, and site-based teams were examined with regard to their roles, support and resources provided. Using a series of multiple regression analyses, this study examined data for 20,000 kindergarten, first, second and third grade students across 61 Arizona schools for the first three years of the program's implementation. Initial findings identified the crucial role the principals’ leadership provides in their ability to enhance teachers’ positive attitudes and instructional improvements as significant predictors of students' performance. |
| Practical Considerations for Implementing Strong Evaluation Designs in School-Based Prevention Programs |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Darcy Richardson, LeCroy and Milligan Associates Inc, darcy@lecroymilligan.com |
| Joanne Basta, LeCroy and Milligan Associates Inc, jbasta@lecroymilligan.com |
| DeeAnn Arroyo, Pima Prevention Partnership, darroyo@thepartnership.us |
| Geoff Wood, LeCroy and Milligan Associates Inc, geoff@lecroymilligan.com |
| Abstract: School-based pregnancy prevention programs face numerous challenges with implementation and sustainability. A rigorous evaluation can be advantageous for addressing these challenges because it can provide periodic information on implementation processes and progress. It can also provide reliable and valid outcome data on which program improvement and best-practices can be based. However, an effective evaluation is difficult to accomplish in a school-setting. The strongest evaluation design may be impractical based on the complexities of the program being offered, financial considerations, and/or the responsiveness of the community. The challenge with these programs is to design a rigorous evaluation that is practical in a school-setting. This session will include discussion of the design and implementation of two school-based pregnancy prevention programs, including challenges and lessons learned. The importance of the partnership between program and evaluation will be emphasized throughout this interactive session. |
| Evaluating the Impact of a Three-Hour Neuroscience of Addiction Curriculum on College Students’ Knowledge and Attitudes: Results From the National Institute on Drug Abuse Enters College Project |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Joyce Hartje, University of Nevada Reno, jhartje@casat.org |
| Michelle Berry, University of Nevada Reno, mberry@casat.org |
| Nancy Roget, University of Nevada Reno, nroget@casat.org |
| Wendy Woods, University of Nevada Reno, wwoods@casat.org |
| Angela Broadus, University of Nevada Reno, abroadus@casat.org |
| Abstract: The purpose of this project was to evaluate the impact of a 3-hour science of addiction research-based curriculum infusion on university undergraduate students’ knowledge and attitudes about substance use disorders. Two sections of existing introductory courses in three pre-professional disciplines (criminal justice, nursing, and social work) were selected. One section served as the curriculum infusion (implementation) group (N = 122); the other as the control (non-implementation) group (N = 178). Pre- and post-test measures were developed based on curriculum objectives to assess the effect on knowledge and attitudes. Results showed a significant increase in knowledge about addiction and a corresponding decrease in stigmatizing attitudes in 1) the implementation vs. non-implementation groups; and 2) the implementation group pretest vs. post-test measures. These findings lend support to the potential effectiveness of infusing a brief research-based addiction curriculum into existing undergraduate courses and could have long-term implications for preparing helping professionals. |
| Evaluating the Role of Policy Change in Transforming the Mental Health System in Oklahoma |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| David Wright, Oklahoma Department of Mental Health and Substance Abuse Services, dwright@odmhsas.org |
| Abstract: In implementing and monitoring the Mental Health Transformation State Incentive Grant (TSIG) program, SAMHSA’s Center for Mental Health Services (CMHS) has guided States in evaluating the effectiveness of their programs. As part of this overall evaluation, CMHS asked States to evaluate mental health policy and regulatory changes. This presentation summarizes and documents the methodology and findings from the state agencies in Oklahoma that are involved in TSIG. |
| A Utilization-Focused Evaluation of a Training Program for Early Intervention Service Providers |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Barbara Stroud, Los Angeles Child Guidance Clinic, bstroud@lacgc.org |
| Rupanwita Gupta, Claremont Graduate University, rupanwita@gmail.com |
| Tony Wu, Los Angeles Child Guidance Clinic, tonyw@lacgc.org |
| Abstract: This paper discusses the evaluation of a training program for professionals in early intervention (0-5 mental health services) conducted by a child guidance clinic. Trainings and concurrent supervisions were intended to equip participants to serve clients (0-5) and their families effectively. Surveys elicited quantitative and qualitative feedback to learn about participants’ perceptions of the workshops and utilization of presented topics. Follow-up telephone interviews with eight participants were completed to understand the influence of the training program on changes in clinical practice. The trainings and trainers were rated highly on several aspects. Additionally, phone interview participants mentioned that training topics had informed them of factors to pay attention to when assisting clients, while supervision sessions had provided additional support. Noteworthy are this evaluation's implications, given the dearth of ongoing professional development for early intervention professionals. |
| The Domino Effect: Informing Program Policy Through Evaluative Practices- A Day Treatment Program Case Study |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jacquelyn Christensen, Los Angeles Child Guidance Clinic, jacquelynsheac@hotmail.com |
| Barbara Stroud, Los Angeles Child Guidance Clinic, bstroud@lacgc.org |
| Tony Wu, Los Angeles Child Guidance Clinic, tonyw@lacgc.org |
| Abstract: This paper highlights the importance of evaluating early intervention programs, and provides a case study that describes how utilization-focused evaluation not only resulted in increased program accountability, but also elicited program improvement at both the program level as well as overall organization policy. In evaluating a nonprofit early childhood day-treatment program, it was demonstrated that parental stress was significantly reduced after children’s participation in the program. Data from this preliminary study suggested the benefits of an early intervention program in addition to how the outcomes were utilized to guide and inform administrative policy in terms of program implementation and service delivery. Implications from this study revealed the necessity of evaluation and speak to the reciprocal interaction between evaluation and program improvement. |
| Use of Two Focus Group Interview Formats to Evaluate Public Health Grand Rounds: Methods and Results |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bayram Yilmaz, The Ohio State University, yilmaz.12@osu.edu |
| James Altschuld, The Ohio State University, altschuld.1@osu.edu |
| Abstract: While face-to-face focus group interviews (FGI) have been widely used for program evaluation, online FGIs have recently become popular among evaluators. In an evaluation of Public Health Grand Rounds (PHGR), a series of seminars on current public health issues, we used both techniques and conducted three FGI sessions (one face-to-face, two cyber). They included individuals from stakeholding groups such as public health students, local public health practitioners, and Ohio Center for Public Health Preparedness (O-CPHP) staff. The focus was on how well the program worked, what should be changed, what key public health topics could be emphasized in subsequent grand rounds, reasons for attending PHGR presentations, barriers to attending, reaching public health audiences other than students, and similar concerns. We compared the findings of the two data collection procedures, and made suggestions regarding the effective use of them and for changes in PHGR. Although the cyber FGI produced about same amount of data, took less time, needed less arrangements and a smaller budget than that of face-to-face, it has its own difficulties in that it is not easy to probe for responses, slow typing skills may prevent some from fully joining the process, and the possibility of response bias. |
| Culturally Responsive Early Education: Hispanic Early Childhood Education and Pacific Island PreK Boost Projects |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| JoAnn Yuen, University of Hawaii Manoa, joyuen@hawaii.edu |
| Arthur Hernandez, University of Texas San Antonio, art.hernandez@utsa.edu |
| Abstract: Shaped by federal and state funding, early education programs target the most at-risk: underserved families and those unable to afford or access early education. These families are more likely to be heterogeneous and early education programs and evaluations will need to respond to diversity. Evaluation research is poised to inform, clarify the vision and shape early childhood education policy. This panel will contrast two different early childhood programs and research evaluations: 1. Early education and health curriculum; and, 2. PreK 20-day learning behavior and literacy transition. The activities associated with the 'Healthy and Ready to Learn' and 'PreK-ACE Boost' projects were designed in a manner intended to ensure cultural appropriateness (Hispanic and Pacific Islanders) and used an "evaluation" rubric to guide program development and assess fidelity. Both programs are culturally responsive and effective in preparing young children and families for school. |
| Effects of Three State Health Programs on Hospital Uncompensated Care Provision: Guiding Principles of an Impact Evaluation |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lei Zhang, Georgia State University, alhlzx@langate.gsu.edu |
| Abstract: Our impact evaluation of three state regulatory programs provides an applied example of how specific evaluation policies have guided our evaluation practice. This evaluation seeks to examine the effects of various state policies/programs (Certificate-of-Need program, uncompensated care pool program, and community benefit requirement regulation) and their interactions on hospital provision of uncompensated care. Our experience with this evaluation project helps to illustrate how three guiding principles: (1) stakeholder oriented; (2) quality centered; and (3) utilization focused influenced the design, methodology, reporting and dissemination of the impact evaluation. These guiding principles reflected the professional standards set by the Joint Committee and have been field-tested to promote effective evaluation. |
| Strengthening the Evaluation Capacity of the National Institutes of Health |
| Poster Presentation 128 to be held in Centennial Section D on Wednesday, Nov 5, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brandie Taylor, National Institutes of Health, taylorbr@mail.nih.gov |
| Rosanna Ng, National Institutes of Health, ngr@mail.nih.gov |
| Meryl Sufian, National Institutes of Health, sufianm@mail.nih.gov |
| Madeleine Wallace, National Institutes of Health, wallacem2@mail.nih.gov |
| Yuanting Zhang, National Institutes of Health, zhangyuant@mail.nih.gov |
| Abstract: A comprehensive approach has been developed to strengthen the evaluation capacity of the National Institutes of Health (NIH). Led by the Evaluation Branch (EB), a part of the NIH Office of Portfolio Analysis and Strategic Initiatives, efforts are being implemented to build and sustain support for evaluation and evaluation capacity. This poster will showcase these activities (e.g., strategies to maximize the effective use of dedicated funding for evaluations, education and outreach activities, technical support and guidance, collaborative efforts to foster evaluation skills and knowledge) and highlight some of the challenges and lessons learned. |