| How Graduate Students Gain Evaluation Skills: What Worked for Us |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Margot Ackermann, Old Dominion University, margot.ackermann@gmail.com |
| Abstract: There are few universities in the United States that offer graduate programs either directly in evaluation or with a concentration in evaluation. Therefore, students looking for practical experience are often left on their own to create opportunities for themselves. The focus of this presentation will be a description of what we have done at our institution to train our students in program evaluation and a discussion of what has worked for us. A description of how students can find opportunities to conduct their own evaluation research at their institution, strategies for fitting evaluation into their typical graduate work load, as well as how they can obtain education on evaluation practices outside of their educational institution will be presented. |
| Assessing the Quality of Program for Improving Reading in the Southeast of Mexico |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Victor Aguilar-Fernandez, Universidad Autonoma de Yucatan, afernand@uady.mx |
| Edith J Cisneros-Cohernour, Universidad Autonoma de Yucatan, cchacon@uady.mx |
| Abstract: This paper presents the findings of an evaluation of a program developed for improving the quality of reading in basic education in Southern Mexico. The study included the main stakeholders involved in program implementation: Department of Education officials, school principals, teachers and school supervisors. Data collection included: semi-structured interviews, site observations and document analysis.Findings of the study indicate that although the program design is culturally and academically relevant to the schools, implementation problems and lack of training and adequate communication among different levels is affecting program results and implementation. |
| An Evaluation of Techniques for Recruiting Parents into HIV Prevention Research in Rural Communities: Teen Recruitment of Adults Versus Traditional Methods: What Works Best? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Josephine Allen, Cornell University, jaa7@cornell.edu |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Jennifer Tiffany, Cornell University, jst5@cornell.edu |
| Deladem Kusi-Appouh, Cornell University, dnk7@cornell.edu |
| Wendy Vonhof, Cornell University, wmv2@cornell.edu |
| Abstract: Reducing adolescent sexual risk-taking behaviors is one of the leading national priorities under the “Healthy People 2010” initiative, and adolescents in rural areas are particularly at high risk of acquiring STIs, including HIV. Adolescents reporting positive communication with parents are less likely to engage in sexual risk-taking behaviors. Yet, engaging teenagers and their parents in HIV prevention interventions remains a challenge. We report our experience with recruiting parents or guardians into an HIV prevention research project through teenage participants who had been engaged using a Participant-Driven Recruitment approach. Our results indicate that recruitment of adults into HIV preventive interventions may work better when done using traditional techniques, rather than by asking adolescents to recruit adults. The reasons and applications to other interventions and evaluations will be discussed. |
| HIV/AIDS/STD-related Worry and Sexual Activity Within Adolescent Social Networks |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deladem Kusi-Appouh, Cornell University, dnk7@cornell.edu |
| Josephine Allen, Cornell University, jaa7@cornell.edu |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Jennifer Tiffany, Cornell University, jst5@cornell.edu |
| Lindy Williams, Cornell University, lbw@cornell.edu |
| Abstract: This paper examines the relationship of four personal worry factors to sexual activity among 15-19 year-olds. Adolescents were recruited by their peers through Participant-Driven Recruitment (PDR), a methodology that combines participatory research and Respondent-Driven Sampling (RDS). The social networks illustrated by the PDR/RDS recruitment data enabled us to evaluate the tendency for adolescents to form social ties based on similarity (homophily). The recruiter/recruitee social network was our unit of analysis. The survey instrument included measures of worry about 1) getting HIV/AIDS; 2) becoming pregnant/getting someone pregnant; 3) having sex; and 4) getting a sexually transmitted disease. Worry was measured as a continuous variable: never worried, sometimes worried, often worried and constantly worried. Results suggest that those adolescents who worried about these factors had increased odds of engaging in sexual activity during the month preceding the survey. Color graphics displaying the social networks according to each worry factor complement the analysis. |
| The Effects of a Series of Early Literacy Workshops for Parents on Parents, Teachers and Students: An Evaluation of the Pre-Kindergarten Parent Project |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Katy Allen, University of Rochester, flygirl@rochester.rr.com |
| Abstract: The Parent Project is a workshop approach to parent involvement designed by James Vopat. In the current study, parents participated in five workshops designed to enhance knowledge, attitudes, and behaviors, with regards to children's learning and literacy development. Program goals included the development of trusting relationships between parents and educators, as well as improved outcomes for children. Parents who participated advanced in their levels of knowledge regarding child development, pro-school/pro-learning attitudes, and literacy-supporting behaviors. Children of participants showed a statistically significant increase in enthusiasm for learning. However, results of the PALS screening did not indicate that children of participants showed greater academic success than children whose parents did not participate. As such, The Parent Project may be a promising program for increasing positive parental attitudes towards educators and school, with a corresponding increase in behaviors that support children's early literacy development and enthusiasm for learning. |
| Glowing in the Dark in the Cave of Information: Organizational Learning and Transtheoretical Models |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Marisa Allen, Case Western Reserve University, marisa.allen@case.edu |
| Abstract: Factors which predict the use and influence of evaluation have been studied since the 1960s. Since this time, theories and models have had limited success in fully explaining how organizations can best use evaluation to ultimately improve organizational functioning. The evaluation field still does not fully understand to what extent various individual and organizational factors lead to evaluation use and program improvement. This paper discusses the ways in which the transtheoretical model and theories of organization learning might aid in our understanding of the consequences of evaluation. The key concepts of these models and their application to evaluation practice will be presented. A new integrated model which incorporates individual and organizational characteristics will be proposed. The paper discusses the use of this integrated model as an organizing premise for evaluation practice. |
| Evaluation: A Tool for Teaching and Learning High School Mathematics |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anane Olatunji, George Washington University, dr_o@gwu.edu |
| Paige Allison, Alachua County Public Schools, plado@cox.net |
| Abstract: Baseline survey results from this pilot project in spring 2006 indicated that at a regular high school in Florida most students reported mathematics not only was less interesting than other subjects but also more difficult to learn. Our aim, therefore, was to create an effective method of teaching mathematics. Efficacy was based on two criteria: student engagement and achievement. The evaluators used students' weekly test scores to illustrate a unit on statistical concepts. Graphs were generated and displayed using SPSS and PowerPoint slides. Student incentives also were provided. Final survey results indicated that the “Data Analysis Project” was highly effective. Ninety-four percent of students reported that they enjoyed the project, 70 percent said that it helped them to learn mathematical concepts, and 85 percent agreed that the project helped them to understand the utility of mathematics. In addition, student comments bolstered survey results. |
| Evaluator Learning Curve: Is it a Gamble for Stakeholders? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Caroline Lippy, Georgia State University, clippy1@student.gsu.edu |
| Leanne Valentine, Georiga State University, leannevalentine@comcast.net |
| Jim Emshoff, Georgia State University, psyjge@langate.gsu.edu |
| Beth Anthony, Georgia State University, eanthony1@student.gsu.edu |
| Dary Enkhtor, Georgia State University, denkhtor1@student.gsu.edu |
| Ayana Perkins, Georgia State University, ayanaperkins@msn.com |
| Lindsey Zimmerman, Georgia State University, lindsetzimmerman@gmail.com |
| Abstract: This poster examines a series of research projects funded by the Georgia Department of Human Resources, focusing on a comparison of two prevalence studies of problematic gambling behavior. Serving as needs assessments, the studies, conducted in 1999 and 2007, each completed approximately 1600 randomized telephone interviews of Georgia residents and helped to outline the need for problem gambling services in the state. This poster examines several modifications between the first and second study, indicating an evaluator learning curve. Specifically, evaluators utilized an alternative measure of pathological gambling and also became more aware of race/ethnicity interviewer effects on the sample. While such modifications improved the quality of the evaluation, they potentially induced unintended negative consequences for the state, including a decreased understanding of the effectiveness of previous efforts implementing evaluator recommendations from the first study. The implications and recommendations for future research are made for both evaluators and state officials. |
| How Learning About Evaluation Can Better Support Teaching and Learning in a Postsecondary Environment |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stanley Varnhagen, University of Alberta, stanley.varnhagen@ualberta.ca |
| Brad Arkison, University of Alberta, bradley.arkison@ualberta.ca |
| Jason Daniels, University of Alberta, jason.daniels@ualberta.ca |
| Abstract: Although evaluation is common in post-secondary environments, it is often limited to being summative and judgmental in nature, which restricts its effectiveness as a tool to enhance teaching and learning. In the post-secondary environment, and in a broader scholarship context, it is necessary for academics and researchers to better understand evaluation and applied research methods. In order to enhance the effectiveness of evaluation at the post-secondary level, academics and researchers need to understand the discipline of formative evaluation, its tools, and potential benefits. In addition, the evaluative process needs to be safe from external judgment and comparisons. This presentation examines some current evaluative projects that allow various users of innovative instructional approaches to learn evaluative methods, to receive formative feedback in a safe environment, and to participate actively in the evaluation process. |
| Evaluating Outreach Strategies to Promote HIV Testing Among Young African-American Men who Have Sex With Men: The Case for Youth Empowerment |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| George Greene, Working For Togetherness, george.wft@sbcglobal.net |
| Clifford Armstead, Working For Togetherness, cliffordarmstead@sbcglobal.net |
| Gregory Norels, Working For Togetherness, gnorels@hotmail.com |
| Sean Morgan, Working For Togetherness, smorgan.wft@sbcglobal.net |
| Abstract: Increasing HIV rates among young, African-American men who have sex with men (YAAMSM) suggest this population is not being effectively served by traditional prevention programs. This poster articulates our organizational response to the epidemic and presents an evaluation of our multi-pronged outreach approach. Incorporating urban marketing strategies into HIV prevention for YAAMSM, our agency provides both mobile community-wide and fixed-site outreach in HIV-impacted neighborhoods and high-risk settings by offering HIV/STD education, counseling, and testing. Additionally, our agency opened a youth empowerment center to equip participants with resources to manage their healthcare needs. Preliminary evaluation data indicate that mobile- and fixed-site outreach located large numbers of YAAMSM for information and condom dissemination; however, youth-centered services in a GLBTQ safe space better promoted HIV testing among youth. Taken together, this service delivery approach minimizes cultural barriers to health-seeking behaviors and delivers HIV prevention services to groups disproportionately affected by the HIV epidemic. |
| Evaluation and Learning: Project ECHO, a Training and Capacity Building Model for Hepatitis C |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Denise Dion, University of New Mexico, dmdion@salud.unm.edu |
| Jan Mines, University of New Mexico, jmines@salud.unm.edu |
| Karla Thornton, University of New Mexico, kthornton@salud.unm.edu |
| Sanjeev Arora, University of New Mexico, sarora@salud.unm.edu |
| Abstract: Using telemedicine infrastructure for clinical learning, Project ECHO (Extension for Community Healthcare Outcomes) trains primary care providers who serve rural, underserved and prison populations in providing Hepatitis C (HCV) treatment for their patients with the guidance of specialists. This model is relevant to populations with chronic, complex diseases or with limited access to specialists (Arora et al, 2007). Using cased-based and iterative learning, primary care providers present HCV patients during weekly telemedicine clinics. Specialist physicians co-manage patients with primary care providers utilizing evidence-based treatment protocols. Learning loops are created through iterative and collaborative case-based discussion during each clinic. The evaluation views the provider as learning a complex set of behaviors with extensive support in a community of similar practitioners. The evaluation draws from self efficacy, situated learning and social cognitive theory. Providers endorse their knowledge growth and self-efficacy about Hepatitis C screening, symptom management, and treatment. |
| Evaluating an Intensive Outpatient Treatment Program for Parolees With Mental Health and Substance Abuse Disorders |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pamela Walsh, Eastern Michigan University, pwalsh@emich.edu |
| David Atkinson, Community Care Services, ripsman1214@yahoo.com |
| Abstract: This is a collaborative project between a community mental health and substance abuse services agency located in Lincoln Park, MI and an university in evaluating the agency's' M-COIT Program, which is an intensive, multi-disciplinary outpatient mental health and substance abuse treatment program for parolees who are mentally ill and/or have substance disorders. M-COIT's goal is to assist parolees' in adjusting to community living and to live a crime-free life style. The premise is that parolees who comply with treatment will have lower recidivism rates, reduced inpatient psychiatric hospitalizations and will comply with parole conditions. Empowerment evaluation is the evaluation method using client record review of parolees who have been discharged from the M-COIT Program from 2001-2005 to identify their recidivism and inpatient psychiatric hospitalization rates and to identify factors (e.g., type of interventions, length of interventions, presence/absence of social supports) that have been significant in affecting recidivism and hospitalization rates. |
| Let Me Tell You What I Need! Using Multiple Perspectives to Explore the Needs of Adolescents After a Concussion |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Isabelle Gagnon, University of Montreal, isabelle.gagnon6@sympatico.ca |
| Swaine Bonnie, University of Montreal, bonnie.swaine@umontreal |
| François Champagne, University of Montreal, francois.champagne@umontreal.ca |
| Helene Lefebvre, University of Montreal, |
| Debbie Feldman, University of Montreal, |
| Jeff Atkinson, Montreal Children's Hospital, |
| Abstract: Practice guidelines have been published concerning the treatment of individuals after a concussion, but most target adults or children, with no particular emphasis on adolescents. Furthermore, most intervention programs put in place have been developed through expert consensus without the specific input of adolescents themselves. As an initial step in improving the care provided to adolescents after a concussion, their specific needs were assessed using different perspectives including those of the literature, of service providers and, especially, that of adolescents as users of the existing services. Data collection strategies included a systematic review of the literature, interviews with adolescents and with parents, focus groups of expert service providers and a postal survey. Our results demonstrate that, even after an event considered by many as trivial, adolescents have service needs that are unique and particularly important during adolescence. Clear benefits can be obtained by including adolescents themselves in evaluations addressing issues related to them and preconceived ideas about their needs should be discarded in favor of a more collaborative approach. |
| Validity of Patient-reported Health-related Quality of Life Global Ratings of Change Using Structural Equation Modeling |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-generated single-item global ratings of change (GRCs) are often used as anchors of health-related quality of life (HRQoL) since they are easy for clinicians to interpret and provide clinicians with change score interpretations that may guide clinical decision-making and assist in determining intervention efficacy. Although this approach may be preferred, the validity of the anchor-based approach is currently under scrutiny. We used structural equation modeling to estimate the explained variation in domain-specific GRCs that is accounted for by Time 1 (T1) and Time 2 (T2) domain-specific summary scores from the Short-Form 36, V2 (SF-36) Health Survey in 356 asthma outpatients. Correlations revealed that domain-specific GRCs were more strongly related to T2 than T1 domain summary scores, indicating that patients were not equally relying on T1 and T2 to generate the GRCs. Furthermore, T1-domain summary scores were not of equal magnitude and opposite sign as compared to T2 scores. There is insufficient evidence to establish SF-36 domain-specific GRC validity in asthma outpatients. |
| A Comparison of Traditional and Rasch Cut Points for Assessing Clinically Important Change in Health-Related Quality of Life Among Patients with Asthma |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-perceived change in health-related quality of life (HRQoL) domains has often been classified using a 15-point patient transition rating scale with the arbitrarily-derived change levels of minimal (2, 3 or -2, -3), moderate (4, 5 or -4, -5) and large (6, 7 or -6, -7). Therefore, in order to determine the adequacy of these traditional change levels, item response theory methodology was employed to derive empirically-based cut points for the change levels. Our sample included 396 asthmatic outpatients who completed bimonthly telephone interviews on the Asthma Quality of Life Questionnaire and transition rating items over one year. After computing Rasch partial credit model transition rating cut points, we compared the mean domain-specific HRQoL changes from Time 1 to Time 2 under each traditional transition rating change level with those under Rasch-derived change levels. Although traditional and Rasch categorizations for small, moderate, and large changes slightly differed, nearly all mean changes between classification approaches were comparable. Traditional transition rating cut points remain suitable to assess HRQoL clinical significance in outpatients with asthma. |
| Impact of AgrAbility Program: Helping Farmers to Stay in Farming |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Connie Baggett, Pennsylvania State University, bbc@psu.edu |
| Rama Radhakrishna, Pennsylvania State University, brr100@psu.edu |
| Linda Fetzer, Pennsylvania State University, lmf8@psu,edu |
| Abstract: Agriculture work is ranked as one of the most dangerous occupations in the nation (RTC, 2000). AgrAbility is a four year grant project through the U.S. Department of Agriculture that provides direct services to farmers with a disability or injury. Services provided by AgrAbility include disabled farmer evaluation, work sites assessments, modification recommendations, suggesting equipment adaptation, task restructuring, mobilizing and coordinating community resources and services, and facilitating rural independent living, etc. The overall purpose of the study was to assess the impact of Agrability Program. Overall, several of the targets of the project were met. Assessments and site visits conducted by AgrAbility staff has helped farmers to perform farm related tasks and continue farming. During the implementation phase, we encountered several challenges and barriers. In order to address these challenges, several changes are underway in the new project cycle (2006-10) which was recently funded by USDA. |
| Successful Organization Learning and Evaluation Capacity Building Among Nonprofits |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michele Graham, JVA Consulting LLC, michele@jvaconsulting.com |
| Rebecca Baggett, JVA Consulting LLC, rebecca@jvaconsulting.com |
| Sheridan Green, JVA Consulting LLC, sheridan@jvaconsulting.com |
| Abstract: JVA Consulting, LLC, developed a capacity-building program for community-and faith-based organizations with funding from the U.S. Department of Health and Human Services. The program was designed to increase organizational capacity in numerous areas, including evaluation. Capacity building services were provided for nine months and organizations took part in baseline and follow-up assessments of evaluation capacity. The purpose of the study was to examine factors related to capacity growth in evaluation and to subsequent organizational learning. Research questions include: 1) were there significant gains in evaluation capacity? 2) what organizational factors were associated with gains in evaluation capacity; and 3) how did organizational learning with regard to evaluation manifest at the individual or organizational level? A mixed method design was used for the study; a quantitative assessment tool measured capacity and organization characteristics and organizational learning was tapped through informal interview notes. Findings indicate significant gains in capacity and revealed important successes with regard to organizational learning. |
| Assertive Community Treatment in Atlanta: Exploring Particular Outcomes of a Grady Healthcare System's Approach |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Angela Mooss, Georgia State University, amooss1@student.gsu.edu |
| John Barile, Georgia State University, jbarile1@student.gsu.edu |
| Joanna Weinberg, Georgia State University, jweinberg3@student.gsu.edu |
| Doyanne Darnell, Georgia State University, dhorst1@student.gsu.edu |
| Brandeis Green, Georgia State University, bgreen8@student.gsu.edu |
| Abstract: Extensive research indicates that the Assertive Community Treatment (ACT) model of mental health service delivery is effective in treating persons with severe mental illness. The Atlanta based ACT team (ECSEL) at the Grady Healthcare System employs a “housing first” philosophy which holds that effective service delivery cannot occur without stable housing. Supportive housing for homeless people with severe mental illness has been found to reduce overall service use, incarcerations and hospitalizations (Culhane, et. al. 2001). The ACT model is predicted to positively impact overall factors of wellbeing, such as housing satisfaction, familial relations, employment opportunities, prevalence of psychiatric symptoms, and reductions in substance abuse. This presentation examines the outcomes of the above factors among a homeless sample in Atlanta, and explores the unique challenges and successes of data collection in a variety of settings. Subsequent outcomes and issues of diversity among program participants and between the participants and members of the ACT team will also be addressed regarding fidelity to the original ACT model. |
| Influences on the Career Paths of Underrepresented Minority Students in a Biology Enrichment Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jennifer Sweeney, University of California, Davis, jksweeney@ucdavis.edu |
| Merna Villarejo, University of California, Davis, mrvillarejo@ucdavis.edu |
| Amy Barlow, University of California, Davis, aebarlow@ucdavis.edu |
| Abstract: A two-stage survey/interview evaluation design was used to test the premise that science enrichment activities, particularly participation in undergraduate research, encourage underrepresented minority students to pursue biology-based professions. Alumni who participated in an educational enrichment program at the University of California, Davis responded to a survey and a follow-up telephone interview regarding influences on their career choices. The survey and interview complement a statistical study associating undergraduate research participation and positive graduation outcomes including graduation in a biology major. Survey respondents (n=202) detailed undergraduate experiences and career choices. Over 100 interviewees reflected on the contribution of pre-college, undergraduate, and post-graduate experiences to their subsequent career choices in the CATI follow-up interview. Preliminary analysis suggests that undergraduate research experience affects career choice by providing an opportunity for students to discover, refine, or confirm their interests. The value of this experience appears to persist regardless of whether students pursue a research career. |
| Evaluating the Virginia Community College System Professional Development Program: Lessons in Conducting a Formal Assessment of a Large-scale, Centralized Professional Development Initiative |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kelle Basta, Caliber an ICF International Company, kbasta@icfcaliber.com |
| Thomas J Horwood, Caliber an ICF International Company, thorwood@icfcaliber.com |
| Mary Ann Hanson, Caliber an ICF International Company, mhanson@icfcaliber.com |
| Nan Ottenritter, Virginia Community College System, nottenritter@vccs.edu |
| Abstract: This presentation will showcase the methodology and findings related to the evaluation of the Virginia Community College System (VCCS) Professional Development (PD) program. The evaluation was conducted to provide the VCCS with information on participation, needs, satisfaction, preferred format for delivery, and outcomes related to the professional development of faculty and staff. A mixed-methods approach was used, both qualitative and quantitative data were collected, and each project phase was designed to build on the previous steps. Evaluators conducted interviews and focus groups, designed a logic model, and administered a survey to VCCS employees. The evaluators integrated the results to develop recommendations for the system-wide PD program and for future program assessment activities. Overall, the conclusions of this assessment demonstrate that the VCCS PD program is highly effective in meeting many of the constituents' professional development needs. |
| Mentoring Adult Ex-offenders in Ready4Work: Mentors' Perspectives |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Danijela Korom-Djakovic, Public/Private Ventures, dkoromdjakovic@ppv.org |
| Igor Holas, Public/Private Ventures, iholas@ppv.org |
| Shawn Bauldry, University of North Carolina, sbauldry@email.unc.edu |
| Abstract: Ready4Work, a three-year national demonstration, was designed to address barriers ex-offenders face as they transition back into their communities. In addition to typical components of a reentry program (case management and employment), Ready4Work sites also sought to provide program participants with mentors. This paper explores the experiences of volunteer mentors who mentored adult ex-offenders. We analyzed interviews of 31 mentors from seven cities, focusing on their motivations for mentoring ex-offenders and their perceived roles as mentors. Preliminary data analyses indicate that mentors' ex-offender status (i.e., whether a mentor was ever incarcerated or not) as well as religious orientation shape their construction of what it means and what it takes to mentor ex-offenders. Mentoring programs for adult ex-offenders are sparse and so is data about them. This study informs future reentry program evaluations and provides insights about evaluation designs that take into account mentors' perspectives and a range of their experiences. |
| Mentoring Adult Ex-offenders in Ready4Work: Assessing the Effects of Mentoring |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shawn Bauldry, University of North Carolina, sbauldry@email.unc.edu |
| Igor Holas, Public/Private Ventures, iholas@ppv.org |
| Danijela Korom-Djakovic, Public/Private ventures, dkoromdjakovic@ppv.org |
| Abstract: Ready4Work, a three-year national demonstration, was designed to address barriers people coming out of prison face as they transition back into their communities. In addition to typical components of a reentry program (case management and employment), Ready4Work sites also sought to provide program participants with mentors. An analysis of numerous sources of data suggests meeting with mentors may improve program retention and the employment prospects of participants. As a demonstration, however, this research lacked a comparison group. Particular attention is given to acknowledging the limitations of non-comparison group research, while making the most of the available data. |
| Learning Through Focus Groups: Evaluation of the Treatment Planning Process by Clinicians Within a Community Health Center |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Erica Gosselin, Mental Health Center of Denver, erica.gosselin@mhcd.org |
| Steve Baumer, Mental Health Center of Denver, steve.baumer@mhcd.org |
| Kristi Helvig, Mental Health Center of Denver, kristi.helvig@mhcd.org |
| Antonio Olmos, Mental Health Center of Denver, antonio.olmos@mhcd.org |
| Abstract: At the 2006 AEA conference, the Mental Health Center of Denver (MHCD) presented a paper regarding mental health consumers' perceived participation in their treatment plans. Results from these focus groups yielded specific areas for follow-up regarding the perceived utility of the Individual Service Plan (ISP) and consumers' perception of involvement in the process. Following this study, we wanted to elicit additional information in order to create a comprehensive plan for change with the aim of improving quality. Based upon an identified area for follow-up, we conducted 12 focus groups with clinicians who work with the consumers at MHCD. These focus groups were conducted to explore 1) the clinician's perception of consumer's participation in treatment, 2) the clinician's perception of the ISP and ISP process, and 3) changes clinicians would like to see in this process. Results from the clinician focus groups were correlated with the prior data from consumer focus groups to formulate targeted changes in the ISP process. |
| Collaborative Evaluation of Project Quest: A Comprehensive Program Preparing Middle and High School Students for Academic Success |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Monifa Beverly, University of North Carolina, Chapel Hill, mngreen@email.unc.edu |
| Rita O'Sullivan, University of North Carolina, Chapel Hill, ritao@unc.edu |
| Abstract: This session presents the results of a two-year summative evaluation of a three-year grant from the U.S. Department of Education's Advanced Placement Incentive Program. Project Quest is a three-school partnership designed to increase the successful participation of low-income students in academically rigorous, higher–level academic courses. To address this goal, Project Quest has implemented a complex and integrated strategy comprising four major programs: Advanced Placement (AP), Advancement via Individual Determination (AVID), International Baccalaureate (IB), and Paideia. A collaborative evaluation was undertaken as a means of building the evaluation capacity of program staff leading to increased utility of evaluation findings. The results of this evaluation, guided by a collaborative evaluation approach, support the notion that programs with limited and ample resources alike can secure sound, effective evaluations. Moreover, the approach's focus on acknowledging and valuing a staff's own program expertise leads to better program management and execution of future evaluation responsibilities. |
| The Use of a Standardized Exercise to Evaluate Emergency Preparedness Training Effectiveness for Health Care Workers |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Eileen Blake, Yale New Haven Health, eileen.blake@ynhh.org |
| Michael Mozzer, Yale New Haven Health, michael.mozzer@ynhh.org |
| Abstract: The Yale New Haven Center for Emergency Preparedness and Disaster Response National Alliance for Training HealthCare for Emergency Response (NATHCER) is a nationwide education and training program for the healthcare delivery workforce. The project provides healthcare emergency preparedness training to ten partner organizations throughout the nation. The effectiveness of the NATHCER project will be determined in part by a standardized process where each regional partner hosts a Tabletop Exercise facilitated by YNH-CEPDR staff. The exercise will consist of a subset of staff that has completed either Introduction to Emergency Management with NIMS or Best Practices for the Protection of Healthcare Based First Receivers. The exercise scenario and objectives will be consistent at all locations. The outcomes of this exercise will inform the ability of these courses to increase the emergency preparedness competency of the participants and test a strategy for use of exercises to validate training content. |
| Immersion in School-based Evaluation Communities |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Douglas Huffman, University of Kansas, huffman@ku.edu |
| Kelli Thomas, University of Kansas, kthomas@ku.edu |
| Karen Lombardi, University of Kansas, lombaka6@yahoo.com |
| Carrie Hohl, University of Kansas, cmhohl@salvajor.com |
| Abstract: This poster session will focus on the results of the Collaborative Evaluation Communities in Urban Schools project (CEC Project) at the University of Kansas. The project was designed to enhance the evaluation capacity of K-8 schools through collaborative evaluation communities comprised of teachers, instructional coaches, graduate students, and university faculty. The goals of the project were to improve the evaluation capacity of urban schools, develop graduate level educational leaders with the knowledge and skills to evaluate science and mathematics education programs, and develop the evaluation capacity of K-8 teachers. The poster will focus on the impact of using evaluation to develop elementary mathematics and science instruction in schools. The poster will also address the broader question: In what ways can collaborative evaluation create learning communities of practice? |
| Mentoring Adult Ex-offenders in Ready4Work: Mentors' Perspectives |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Danijela Korom-Djakovic, Public/Private Ventures, dkoromdjakovic@ppv.org |
| Igor Holas, Public/Private Ventures, iholas@ppv.org |
| Shawn Bauldry, University of North Carolina, sbauldry@email.unc.edu |
| Abstract: Ready4Work, a three-year national demonstration, was designed to address barriers ex-offenders face as they transition back into their communities. In addition to typical components of a reentry program (case management and employment), Ready4Work sites also sought to provide program participants with mentors. This paper explores the experiences of volunteer mentors who mentored adult ex-offenders. We analyzed interviews of 31 mentors from seven cities, focusing on their motivations for mentoring ex-offenders and their perceived roles as mentors. Preliminary data analyses indicate that mentors' ex-offender status (i.e., whether a mentor was ever incarcerated or not) as well as religious orientation shape their construction of what it means and what it takes to mentor ex-offenders. Mentoring programs for adult ex-offenders are sparse and so is data about them. This study informs future reentry program evaluations and provides insights about evaluation designs that take into account mentors' perspectives and a range of their experiences. |
| Mentoring Adult Ex-offenders in Ready4Work: Assessing the Effects of Mentoring |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shawn Bauldry, University of North Carolina, sbauldry@email.unc.edu |
| Igor Holas, Public/Private Ventures, iholas@ppv.org |
| Danijela Korom-Djakovic, Public/Private ventures, dkoromdjakovic@ppv.org |
| Abstract: Ready4Work, a three-year national demonstration, was designed to address barriers people coming out of prison face as they transition back into their communities. In addition to typical components of a reentry program (case management and employment), Ready4Work sites also sought to provide program participants with mentors. An analysis of numerous sources of data suggests meeting with mentors may improve program retention and the employment prospects of participants. As a demonstration, however, this research lacked a comparison group. Particular attention is given to acknowledging the limitations of non-comparison group research, while making the most of the available data. |
| Collaboration and Evaluation Capacity Building: Lessons From a Learning Environment |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Randi K Nelson, University of Minnesota, nelso326@umn.edu |
| David Fischer, University of Minnesota, fisch413@umn.edu |
| Herbert Struss, University of Minnesota, strus0101@umn.edu |
| Abstract: Evaluation, data based decision making, and collaboration have tremendous potential for building the capacity of teachers and program evaluators to learn from evaluation. This poster highlights the views of teachers and program staff toward collegiality and data use to inform practice, during a three-year collaborative evaluation capacity building project between a university and two public schools. Program evaluation traditionally focuses on stakeholders who receive program services. In collaborations, the views of both the service providers and recipients must be considered. This poster illustrates how annual surveys of teachers and follow up interviews with teachers and program staff have been used to provide all stakeholders the opportunity to learn from their experiences. In this project, use of data from teachers and program staff demonstrates collaboration and data based decision making among all participants and models evaluation to promote collaborative capacity and learning in addition to its more traditional purposes. |
| Let Me Tell You What I Need! Using Multiple Perspectives to Explore the Needs of Adolescents After a Concussion |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Isabelle Gagnon, University of Montreal, isabelle.gagnon6@sympatico.ca |
| Swaine Bonnie, University of Montreal, bonnie.swaine@umontreal |
| François Champagne, University of Montreal, francois.champagne@umontreal.ca |
| Helene Lefebvre, University of Montreal, |
| Debbie Feldman, University of Montreal, |
| Jeff Atkinson, Montreal Children's Hospital, |
| Abstract: Practice guidelines have been published concerning the treatment of individuals after a concussion, but most target adults or children, with no particular emphasis on adolescents. Furthermore, most intervention programs put in place have been developed through expert consensus without the specific input of adolescents themselves. As an initial step in improving the care provided to adolescents after a concussion, their specific needs were assessed using different perspectives including those of the literature, of service providers and, especially, that of adolescents as users of the existing services. Data collection strategies included a systematic review of the literature, interviews with adolescents and with parents, focus groups of expert service providers and a postal survey. Our results demonstrate that, even after an event considered by many as trivial, adolescents have service needs that are unique and particularly important during adolescence. Clear benefits can be obtained by including adolescents themselves in evaluations addressing issues related to them and preconceived ideas about their needs should be discarded in favor of a more collaborative approach. |
| Impact of a Financial Program on Knowledge and Practices of Adult Participants |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Cathy Bowen, Pennsylvania State University, cfb4@psu.edu |
| Marilyn Furry, Pennsylvania State University, mfurry@psu.edu |
| Rama Radhakrishna, Pennsylvania State University, brr100@psu.edu |
| Abstract: Reports continue to document the financial illiteracy of Americans. Lack of financial skills is illustrated by bankruptcy rates, increased credit card debt, and low savings rate. The purpose of this study was to assess the effectiveness of a financial education program on the knowledge and practices of participants. A pretest-posttest comparison group design was used to determine if participation in the program would lead to: 1) using a spending plans and recordkeeping system 2) starting a savings program or increase the amount contributed to an existing savings plan, 3) developing a plan to reduce the current debt load, and 4) using techniques to reduce their cost of using credit. Significant differences were found between the treatment and comparison groups' knowledge score on the posttest and delayed posttest. Treatment group participants not only made changes in behaviors relative to managing personal finances but also sustained those changes over a longer period. |
| Ensuring Program Assessment Rating Tool (PART) Effectiveness: The Performance Measure Analytic Tool (PMAT) |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ian Boyd, DFI Government Services, iboyd@dfi-intl.com |
| Christopher Cihlar, DFI Government Services, ccihlar@dfi-intl.com |
| Abstract: The Office of Management and Budget's (OMB) Program Assessment Rating Tool (PART) alters how federal programs are required to report on accomplishments. However, evaluators have done little to determine how PART performance measures and the scores they generate compare across government programs. Without such analysis, the consistency of the PART process is unknown. Whether PART scores are driven by an ability to demonstrate successful progress through quality measures or whether other variables, unrelated to performance, affect these scores becomes a pivotal issue. To address this, there needs to be a demonstrated correlation between the quality of performance measures and program ratings. This poster introduces an analytical tool that identifies key variables in each performance measure for the 800+ programs that have been analyzed through the PART process and then uses these variables to develop cross-cutting statistical comparisons between the quality of performance measures, program achievement, and the program's PART scores. |
| Having Graduate Students as Evaluators: A Description of a Community Based Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shawn Dickerson, Old Dominion University, sdick003@odu.edu |
| Abby Braitman, Old Dominion University, abraitma@odu.edu |
| Krystall Dunaway, Old Dominion University, kdunaway@odu.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Abstract: The current evaluation was a team evaluation conducted by graduate students which utilized an online survey to collect information from the stakeholders of interest. Evaluators gathered and examined information concerning different aspects of the job roles for Uniform Patrol Officers (UPOs) and Neighborhood Impact Officers (NIOs) from a local community's Police Department. The researchers assessed attitudes and behavior in regards to the officers' training and responsibilities, mainly pertaining to the Weed & Seed program. The Weed & Seed program was established to assist the city in reducing violent crime, the trafficking of illegal drugs and reducing juvenile crime rates. This outcome-focused evaluation gained vital information for community policing and increased the understanding of police officers beliefs and increase knowledge about their training. |
| Where Do We Go From Here? Lessons on Organizational Change and Development From an Assessment of a School Resource Officer Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kathleen Crowley, Circle Solutions, kcrowley@circlesolutions.com |
| Anna Laszlo, Circle Solutions, alaszlo@circlesolutions.com |
| Deanna Breslin, Circle Solutions, dbreslin@circlesolutions.com |
| Abstract: The purpose of this paper is to present findings from an evaluation of the Montgomery County Police Department's (MCPD) Educational Facilities Officers (EFO) program. The evaluation is based on a mixed method approach that primarily focused on EFO activities at 26 Montgomery County high schools. Multiple sources of data were collected including(1) official program documentation (e.g., grant application, progress reports, etc.); (2) interviews with key stakeholders; (3) observation of EFO activities; (4) surveys with school staff; and (5) secondary school data. We will discuss policy-relevant recommendations around effective management structures and program operations and present key points from a compendium of best practice intended for other SRO programs nationally. |
| Using Formative Evaluation to Culturally Adapt a Parenting Program for Black Fathers of Children With Special Needs |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| James C Bridgers Jr, University of Maryland, College Park, jbridgers@mayatech.com |
| Suzanne M Randolph, University of Maryland, College Park, suzanner@umd.edu |
| Cecil H Doggette, Health Services for Children with Special Needs Inc, cdoggette@hscsn.org |
| Sally Koblinsky, University of Maryland, koblinsk@umd.edu |
| Abstract: Ecological systems theory was used in a formative evaluation to assess the feasibility of a model program, developed for Black mothers' normative parenting, for use with Black fathers of children with special needs. There is scant literature on Black fathering; and parenting programs often focus on parents' challenges rather than strengths. This evaluation examined the rewards, strengths, challenges, and support needs of African American fathers parenting children with special needs. The theoretical framework guided development of focus group guides and analyses. Data were used to culturally adapt the model parenting program and select evaluation measures for the locally implemented intervention. |
| Evaluating an Organizational Capacity Building Program Designed to Increase Diversity Among Health Services Researchers |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Andrea S Burling, American Institutes for Research, aburling@air.org |
| Roger E Levine, American Institutes for Research, rlevine@air.org |
| Darlene Russ-Eft, Oregon State University, darlene.russeft@oregonstate.edu |
| Jennnifer Stephens, American Institutes for Research, jstephens@air.org |
| Abstract: This paper will describe a large-scale organizational capacity building evaluation, discuss challenges faced and strategies for overcoming them, describe a unique methodology being used, and may present initial findings. The American Institutes for Research is evaluating two programs funded by the Agency for Healthcare Research and Quality (AHRQ) that are designed to improve the organizational capacities to conduct health services research of academic institutions that predominantly serve racial/ethnic minorities, and institutions located in geographic areas that have had historically low levels of AHRQ funding. The evaluation is determining how, and to what extent, grantees have strengthened their institutional infrastructures and enhanced the capabilities of individual faculty in relation to their ability to undertake health services research. Multiple methods are being used including a unique document data abstraction approach, as well as telephone interviews and site visits. Key challenges include defining and measuring success for organizational capacity building programs and efficiently accessing organizational data. |
| How Graduate Students Gain Evaluation Skills: What Worked for Us |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Margot Ackermann, Old Dominion University, margot.ackermann@gmail.com |
| Abstract: There are few universities in the United States that offer graduate programs either directly in evaluation or with a concentration in evaluation. Therefore, students looking for practical experience are often left on their own to create opportunities for themselves. The focus of this presentation will be a description of what we have done at our institution to train our students in program evaluation and a discussion of what has worked for us. A description of how students can find opportunities to conduct their own evaluation research at their institution, strategies for fitting evaluation into their typical graduate work load, as well as how they can obtain education on evaluation practices outside of their educational institution will be presented. |
| How Good is Our Advice? An Evaluation of an Undergraduate Advising Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jessica Ladage, Old Dominion University, jladage@odu.edu |
| Catherine Butler, Old Dominion University, cbbutler@odu.edu |
| Kelly Carpenter, Old Dominion University, kcarp005@odu.edu |
| Karen Otero-Fisher, Old Dominion University, kfish003@odu.edu |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Abstract: Advising programs at the university level allow students to gain advice on class offerings, school activities, future career plans, and more. The purpose of this evaluation is to determine the effectiveness of our university's undergraduate Psychology advising program. This evaluation is interested in students' awareness of the advising process, how well students are using the available resources, and student satisfaction towards the advising program. Quality of faculty advisors and advising sessions will also be examined. Undergraduate declared Psychology majors will complete an on-line questionnaire regarding the advising program. Both quantitative and qualitative data will be used to assess students' views on the advising program. Correlational analyses will be conducted to examine relationships, as well as analyses of variance to see if any gender or class year differences are present. |
| Having Graduate Students as Evaluators: A Description of a Community Based Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shawn Dickerson, Old Dominion University, sdick003@odu.edu |
| Abby Braitman, Old Dominion University, abraitma@odu.edu |
| Krystall Dunaway, Old Dominion University, kdunaway@odu.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Abstract: The current evaluation was a team evaluation conducted by graduate students which utilized an online survey to collect information from the stakeholders of interest. Evaluators gathered and examined information concerning different aspects of the job roles for Uniform Patrol Officers (UPOs) and Neighborhood Impact Officers (NIOs) from a local community's Police Department. The researchers assessed attitudes and behavior in regards to the officers' training and responsibilities, mainly pertaining to the Weed & Seed program. The Weed & Seed program was established to assist the city in reducing violent crime, the trafficking of illegal drugs and reducing juvenile crime rates. This outcome-focused evaluation gained vital information for community policing and increased the understanding of police officers beliefs and increase knowledge about their training. |
| How Good is Our Advice? An Evaluation of an Undergraduate Advising Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jessica Ladage, Old Dominion University, jladage@odu.edu |
| Catherine Butler, Old Dominion University, cbbutler@odu.edu |
| Kelly Carpenter, Old Dominion University, kcarp005@odu.edu |
| Karen Otero-Fisher, Old Dominion University, kfish003@odu.edu |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Abstract: Advising programs at the university level allow students to gain advice on class offerings, school activities, future career plans, and more. The purpose of this evaluation is to determine the effectiveness of our university's undergraduate Psychology advising program. This evaluation is interested in students' awareness of the advising process, how well students are using the available resources, and student satisfaction towards the advising program. Quality of faculty advisors and advising sessions will also be examined. Undergraduate declared Psychology majors will complete an on-line questionnaire regarding the advising program. Both quantitative and qualitative data will be used to assess students' views on the advising program. Correlational analyses will be conducted to examine relationships, as well as analyses of variance to see if any gender or class year differences are present. |
| Internal Auditing As Evaluation: Quality Assurance and Performance Improvement Through An Internal Social Work Auditing Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| William Cabin, Youth Consultation Service, williamcabin@yahoo.com |
| Abstract: Auditing historically has been viewed as a method reserved for the accounting and business accountability in the public and private sectors. Evaluation has been used in business, accounting, and many other public and private sectors (Rossi, 2006). However, the literature does not indicate auditing as an accepted method within evaluation. The paper presents an internal auditing program used to assure quality and improve performance of social workers in a multi-site, non-profit child welfare organization. The paper is based on two years of program implementation using a sampling-based clinical chart review based on JCAHo and state regulatory requirements. The paper relates the origins, goals, design, implementation, results, and current status of the program.Samples of the audit tool, exit reports, and graphic trend reports are included. |
| Amplifying Community Voices: Enhancing the Needs Assessment Process |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Catherine Carey, Practical Research, catherine@practical-research.com |
| Ede Taylor, Belair-Edison Healthy Community Coalition, etaylor@becoalition.org |
| Abstract: Needs assessments are commonly conducted during the program development process. Needs assessments generally include a review publicly available data sets and / or a survey. Seeking a better understanding of survey results, a community activist and researcher added a focus group where community leaders reflected on survey results and added their opinions about priorities for services. Focus group participants were invited due to their leadership positions in the community. They received a summary of the survey results prior to the meeting and the survey results were presented at the meeting. The purpose of this poster is to present the focus group methodology and to engage in discussion about its usefulness: Would needs assessment practice be improved by the addition of a focus group process? When? Where there unique features that facilitated a useful experience with lengthening the needs assessment process? |
| How Good is Our Advice? An Evaluation of an Undergraduate Advising Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jessica Ladage, Old Dominion University, jladage@odu.edu |
| Catherine Butler, Old Dominion University, cbbutler@odu.edu |
| Kelly Carpenter, Old Dominion University, kcarp005@odu.edu |
| Karen Otero-Fisher, Old Dominion University, kfish003@odu.edu |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Abstract: Advising programs at the university level allow students to gain advice on class offerings, school activities, future career plans, and more. The purpose of this evaluation is to determine the effectiveness of our university's undergraduate Psychology advising program. This evaluation is interested in students' awareness of the advising process, how well students are using the available resources, and student satisfaction towards the advising program. Quality of faculty advisors and advising sessions will also be examined. Undergraduate declared Psychology majors will complete an on-line questionnaire regarding the advising program. Both quantitative and qualitative data will be used to assess students' views on the advising program. Correlational analyses will be conducted to examine relationships, as well as analyses of variance to see if any gender or class year differences are present. |
| Involving Stakeholders to Improve Acceptance |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Nancy Carrillo, Albuquerque Public Schools, carrillo_n@aps.edu |
| Abstract: A large urban school district requested an outcome evaluation of the many reading programs implemented among its elementary schools in order to possibly reduce the number of programs schools may choose. Previous research of school-level data had suggested that the type of reading program did not influence assessment results; but these results were not well accepted and a more thorough evaluation was requested. Stakeholders did not see the research as being completely unbiased and some believed that key factors had not been considered. I developed a committee of stakeholders with disparate opinions to assist in designing the main data collection tool. The evaluation included some interim deliverables useful to some stakeholders. Results are quite similar to those found in the past. When these results are presented within the next few weeks, I hope that the disappointing results will be more widely accepted due to stakeholder inclusion. |
| Let Me Tell You What I Need! Using Multiple Perspectives to Explore the Needs of Adolescents After a Concussion |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Isabelle Gagnon, University of Montreal, isabelle.gagnon6@sympatico.ca |
| Swaine Bonnie, University of Montreal, bonnie.swaine@umontreal |
| François Champagne, University of Montreal, francois.champagne@umontreal.ca |
| Helene Lefebvre, University of Montreal, |
| Debbie Feldman, University of Montreal, |
| Jeff Atkinson, Montreal Children's Hospital, |
| Abstract: Practice guidelines have been published concerning the treatment of individuals after a concussion, but most target adults or children, with no particular emphasis on adolescents. Furthermore, most intervention programs put in place have been developed through expert consensus without the specific input of adolescents themselves. As an initial step in improving the care provided to adolescents after a concussion, their specific needs were assessed using different perspectives including those of the literature, of service providers and, especially, that of adolescents as users of the existing services. Data collection strategies included a systematic review of the literature, interviews with adolescents and with parents, focus groups of expert service providers and a postal survey. Our results demonstrate that, even after an event considered by many as trivial, adolescents have service needs that are unique and particularly important during adolescence. Clear benefits can be obtained by including adolescents themselves in evaluations addressing issues related to them and preconceived ideas about their needs should be discarded in favor of a more collaborative approach. |
| Learning About Use and Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jeanne Hubelbank, Independent Consultant, jhubel@evalconsult.com |
| Lauren Chapman, Boston College, chapmala@bc.edu |
| Maureen Kavanaugh, Boston College, maureen.kavanaugh@marist.edu |
| Abstract: This paper will share the experiences of teaching and learning about evaluation in a graduate-level course on the practical aspects of evaluation. Specifically, the paper will concentrate on evaluation use and how understanding of it was translated into practical evaluation activities. An assignment to interview someone who had been involved in an evaluation about the use of its results was used also to teach the evaluation process and to provide experience in gathering and presenting results. Students summarized their results, created a poster, and analyzed the summaries to practice content analysis. Student input was sought throughout the course. This paper will discuss the instructor's approaches and students' reactions and learning from the assignment. The materials, successes and the areas in need of improvement will be shared. |
| Promoting Science, Technology, Engineering, Mathematics, and Geography (STEM-G) Through Professional Development: Learning From Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Karen Chen, Wheeling Jesuit University, kchen@cet.edu |
| Laurie Ruberg, Wheeling Jesuit University, lruberg@cet.edu |
| Judy Martin, Wheeling Jesuit University, jmartin@cet.edu |
| Abstract: This poster presents an evaluation project that was conducted to assess the effectiveness of a nationwide science, technology, engineering, mathematics, and geography (STEM-G) educational program established in 2003. The project highlights the need for sound evaluation to document the quality and impact of the program. An interpretive, multiple case study approach was employed to review the program's impact on students. Multiple data sources were collected and analyzed, including teachers' observations of students, e-Portfolios, and student interest surveys. The evaluation team analyzed the data for emerging themes from a set of cluster-based, randomly selected case study schools. The poster presentation will discuss evaluation themes and challenges encountered. |
| National Cancer Institute's Small Grants Program for Behavioral Research in Cancer Control Boosting Careers for New Investigators: Lessons on Assessing Program Success in Meeting National Institutes of Health (NIH) Research Priorities |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Veronica Chollette, National Institutes of Health, vc24a@nih.gov |
| Kathleen Crowley, Circle Solutions, kcrowley@circlesolutions.com |
| Abstract: This paper presents findings from an evaluation of the National Cancer Institute's (NCI) Small Grants Program for Behavioral Research in Cancer Control; explores potential implications on policy, and the inherent challenges assessing effectiveness of programs intended to promote scientific discovery and innovation. Methods: Conducted a descriptive analysis of data from a grantee survey and post-award activities. Results: Findings indicate the program was a factor in supporting the careers of new investigators. Grantees reported a 37% success rate in obtaining additional NIH grant funding including 18% being awarded an R01 and 21% an R21. Approximately 72% of grantees published or had in press at least one article resulting from this award. . These articles were cited a total of 134 times in 85 journals. Conclusions: Evaluation results suggest that the program provides early career investigators the opportunity to continue “independent” research work and fulfill the NIH goal of supporting new investigators. |
| Ensuring Program Assessment Rating Tool (PART) Effectiveness: The Performance Measure Analytic Tool (PMAT) |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ian Boyd, DFI Government Services, iboyd@dfi-intl.com |
| Christopher Cihlar, DFI Government Services, ccihlar@dfi-intl.com |
| Abstract: The Office of Management and Budget's (OMB) Program Assessment Rating Tool (PART) alters how federal programs are required to report on accomplishments. However, evaluators have done little to determine how PART performance measures and the scores they generate compare across government programs. Without such analysis, the consistency of the PART process is unknown. Whether PART scores are driven by an ability to demonstrate successful progress through quality measures or whether other variables, unrelated to performance, affect these scores becomes a pivotal issue. To address this, there needs to be a demonstrated correlation between the quality of performance measures and program ratings. This poster introduces an analytical tool that identifies key variables in each performance measure for the 800+ programs that have been analyzed through the PART process and then uses these variables to develop cross-cutting statistical comparisons between the quality of performance measures, program achievement, and the program's PART scores. |
| Assessing the Quality of Program for Improving Reading in the Southeast of Mexico |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Victor Aguilar-Fernandez, Universidad Autonoma de Yucatan, afernand@uady.mx |
| Edith J Cisneros-Cohernour, Universidad Autonoma de Yucatan, cchacon@uady.mx |
| Abstract: This paper presents the findings of an evaluation of a program developed for improving the quality of reading in basic education in Southern Mexico. The study included the main stakeholders involved in program implementation: Department of Education officials, school principals, teachers and school supervisors. Data collection included: semi-structured interviews, site observations and document analysis.Findings of the study indicate that although the program design is culturally and academically relevant to the schools, implementation problems and lack of training and adequate communication among different levels is affecting program results and implementation. |
| "Apples to Apples": The Experience of Developing an Evaluation Tool for the Workforce Development Performance Benchmarking Project |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacy Woodruff-Bolte, Public/Private Ventures, swoodruff-bolte@ppv.org |
| Martha Miles, Training Inc, mmiles@traininginc.org |
| Carol Clymer, Public/Private Ventures, cclymer@ppv.org |
| Abstract: The Workforce Development Performance Benchmarks Project represents an attempt to determine the feasibility of identifying standards of performance for workforce development organizations and use the benchmarks to improve performance. Following the second wave of survey data collection from workforce organizations across the country serving TANF recipients, lessons related to creating a performance measurement evaluation tool are emerging. Using methods to account for differences among employment and training organizations, we will discuss the development of a much needed evaluation tool in the field for workforce development. |
| Evaluating the Capacity and Quality of an Early Care and Education System |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rob Fischer, Case Western Reserve University, fischer@case.edu |
| Kristen Mikelbank, Case Western Reserve University, kristen.mikelbank@case.edu |
| Claudia Coulton, Case Western Reserve University, claudia.coulton@case.edu |
| Abstract: Since 1999, a public/private partnership in Cuyahoga County, Ohio, has been engaged in a comprehensive effort focused on the well-being of children under the age of six and their families. As part of its strategic planning process, the partnership identified goals around enhancing the early care and education system for the County's children. The present study provides a detailed assessment of the County's regulated early care system in regard to its capacity and quality, by combining administrative and observational data. Using geographic analyses, areas within the County exhibiting relative need for additional child care slots are identified. The study demonstrates the effective integration of a range of data sources in a geographic analysis designed to assist in County planning for the needs of young children and their families. |
| Where Do We Go From Here? Lessons on Organizational Change and Development From an Assessment of a School Resource Officer Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kathleen Crowley, Circle Solutions, kcrowley@circlesolutions.com |
| Anna Laszlo, Circle Solutions, alaszlo@circlesolutions.com |
| Deanna Breslin, Circle Solutions, dbreslin@circlesolutions.com |
| Abstract: The purpose of this paper is to present findings from an evaluation of the Montgomery County Police Department's (MCPD) Educational Facilities Officers (EFO) program. The evaluation is based on a mixed method approach that primarily focused on EFO activities at 26 Montgomery County high schools. Multiple sources of data were collected including(1) official program documentation (e.g., grant application, progress reports, etc.); (2) interviews with key stakeholders; (3) observation of EFO activities; (4) surveys with school staff; and (5) secondary school data. We will discuss policy-relevant recommendations around effective management structures and program operations and present key points from a compendium of best practice intended for other SRO programs nationally. |
| National Cancer Institute's Small Grants Program for Behavioral Research in Cancer Control Boosting Careers for New Investigators: Lessons on Assessing Program Success in Meeting National Institutes of Health (NIH) Research Priorities |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Veronica Chollette, National Institutes of Health, vc24a@nih.gov |
| Kathleen Crowley, Circle Solutions, kcrowley@circlesolutions.com |
| Abstract: This paper presents findings from an evaluation of the National Cancer Institute's (NCI) Small Grants Program for Behavioral Research in Cancer Control; explores potential implications on policy, and the inherent challenges assessing effectiveness of programs intended to promote scientific discovery and innovation. Methods: Conducted a descriptive analysis of data from a grantee survey and post-award activities. Results: Findings indicate the program was a factor in supporting the careers of new investigators. Grantees reported a 37% success rate in obtaining additional NIH grant funding including 18% being awarded an R01 and 21% an R21. Approximately 72% of grantees published or had in press at least one article resulting from this award. . These articles were cited a total of 134 times in 85 journals. Conclusions: Evaluation results suggest that the program provides early career investigators the opportunity to continue “independent” research work and fulfill the NIH goal of supporting new investigators. |
| How Learning About Evaluation Can Better Support Teaching and Learning in a Postsecondary Environment |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stanley Varnhagen, University of Alberta, stanley.varnhagen@ualberta.ca |
| Brad Arkison, University of Alberta, bradley.arkison@ualberta.ca |
| Jason Daniels, University of Alberta, jason.daniels@ualberta.ca |
| Abstract: Although evaluation is common in post-secondary environments, it is often limited to being summative and judgmental in nature, which restricts its effectiveness as a tool to enhance teaching and learning. In the post-secondary environment, and in a broader scholarship context, it is necessary for academics and researchers to better understand evaluation and applied research methods. In order to enhance the effectiveness of evaluation at the post-secondary level, academics and researchers need to understand the discipline of formative evaluation, its tools, and potential benefits. In addition, the evaluative process needs to be safe from external judgment and comparisons. This presentation examines some current evaluative projects that allow various users of innovative instructional approaches to learn evaluative methods, to receive formative feedback in a safe environment, and to participate actively in the evaluation process. |
| Three Decades of Program Evaluation in Brazil: Lessons Learned and the Case of Roberto Marinho Foundation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Thereza Penna Firme, Cesgranrio Foundation, therezapf@uol.com.br |
| Vilma Guimarães, Roberto Marinho Foundation, vilma@frm.org.br |
| Angela Dannemann, D&S Engenheiras Associadas Ltda, angcdann@terra.com.br |
| Ana Carolina Letichevsky, Cesgranrio Foundation, anacarolina@cesgranrio.org.br |
| Abstract: This paper sheds light on lessons learned over thirty years of evaluation in Brazil. A crucial aspect of this journey was learning the do's and the don'ts of evaluation. Some findings: respect stakeholders values, take community knowledge into account, report outcomes with appropriate language including potential and limitations, maintain an attitude of social responsibility. Also, never impose evaluations, hurt values in the name of accuracy or breach agreements without consent. The permanent concern with program evaluation standards while meeting the Brazilian challenges of institutional diversity in the face of social vulnerability has demanded creativity and courage from all evaluators. This apparent dilemma requires the courage to blend and use multiple approaches tuned to the changing reality in order to promote inclusion, dialogue, and appreciation. Hence, this paper focuses on the evaluation case of one of the most renowned Brazilian foundations committed to adult education and cultural patrimony – Roberto Marinho Foundation. |
| Using Latent Growth Modeling to Test Effectiveness of Georgia Family Connection Collaboratives |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Adam Darnell, EMSTAR Research Inc, darnelladam@hotmail.com |
| James Emshoff, EMSTAR Research Inc, jemshoff@gsu.edu |
| Steve Erickson, EMSTAR Research Inc, ericksoneval@att.net |
| Abstract: The Family Connection initiative is a statewide network of community collaboratives that aims to address serious challenges facing Georgia's children and families. Currently 159 collaboratives, one in each county of the state, are implementing community-based support systems in accordance with a set of operating principles (a Theory of Change) designed to improve health-, education- and economic-related outcomes. Outcomes have been measured annually for each of the 159 counties over the eleven year period from 1994 to 2004. Longitudinal change in outcomes was modeled using latent growth curve modeling. Predictors, also measured longitudinally, include characteristics of collaborative structure and process. Alternative analytic approaches for demonstrating causal effects between collaborative characteristics and county-level outcomes will be presented. Discussion will address the strengths and weaknesses of the different approaches to associating change between multiple longitudinal variables. |
| Assertive Community Treatment in Atlanta: Exploring Particular Outcomes of a Grady Healthcare System's Approach |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Angela Mooss, Georgia State University, amooss1@student.gsu.edu |
| John Barile, Georgia State University, jbarile1@student.gsu.edu |
| Joanna Weinberg, Georgia State University, jweinberg3@student.gsu.edu |
| Doyanne Darnell, Georgia State University, dhorst1@student.gsu.edu |
| Brandeis Green, Georgia State University, bgreen8@student.gsu.edu |
| Abstract: Extensive research indicates that the Assertive Community Treatment (ACT) model of mental health service delivery is effective in treating persons with severe mental illness. The Atlanta based ACT team (ECSEL) at the Grady Healthcare System employs a “housing first” philosophy which holds that effective service delivery cannot occur without stable housing. Supportive housing for homeless people with severe mental illness has been found to reduce overall service use, incarcerations and hospitalizations (Culhane, et. al. 2001). The ACT model is predicted to positively impact overall factors of wellbeing, such as housing satisfaction, familial relations, employment opportunities, prevalence of psychiatric symptoms, and reductions in substance abuse. This presentation examines the outcomes of the above factors among a homeless sample in Atlanta, and explores the unique challenges and successes of data collection in a variety of settings. Subsequent outcomes and issues of diversity among program participants and between the participants and members of the ACT team will also be addressed regarding fidelity to the original ACT model. |
| Developmental Evaluation as a Learning Tool for Novice Evaluators |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chris De Luca, Queen's University, 2cd16@qlink.queensu.ca |
| Laura McEwen, Queen's University, laura@education.concordia.ca |
| Hana Saab, Queen's University, saabh@educ.queensu.ca |
| Abstract: Organizations are frequently required to respond on short notice to emergent needs and policy decisions. This presentation examines the contribution of evaluation to the expansion and implementation of a new course in student assessment at a Faculty of Education, initiated in response to Ministry recommendations. Patton's developmental evaluation (1994) provides a framework for the evaluation conducted by a team of graduate students despite recommendations that such an approach may not be feasible for novice evaluators. The paper will discuss how the evaluation contributed to the professional learning of the novice evaluation team, to the positioning of evaluation as an ongoing iterative and interpretive process that contributed to the emergent nature of the course, and to creating a level of sustained interactivity among the stakeholders. Suggested accommodations and adjustments to the developmental evaluation model will be discussed in anticipation of its continued use as a learning tool for students in evaluation. |
| Having Graduate Students as Evaluators: A Description of a Community Based Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shawn Dickerson, Old Dominion University, sdick003@odu.edu |
| Abby Braitman, Old Dominion University, abraitma@odu.edu |
| Krystall Dunaway, Old Dominion University, kdunaway@odu.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Abstract: The current evaluation was a team evaluation conducted by graduate students which utilized an online survey to collect information from the stakeholders of interest. Evaluators gathered and examined information concerning different aspects of the job roles for Uniform Patrol Officers (UPOs) and Neighborhood Impact Officers (NIOs) from a local community's Police Department. The researchers assessed attitudes and behavior in regards to the officers' training and responsibilities, mainly pertaining to the Weed & Seed program. The Weed & Seed program was established to assist the city in reducing violent crime, the trafficking of illegal drugs and reducing juvenile crime rates. This outcome-focused evaluation gained vital information for community policing and increased the understanding of police officers beliefs and increase knowledge about their training. |
| Evaluation and Learning: Project ECHO, a Training and Capacity Building Model for Hepatitis C |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Denise Dion, University of New Mexico, dmdion@salud.unm.edu |
| Jan Mines, University of New Mexico, jmines@salud.unm.edu |
| Karla Thornton, University of New Mexico, kthornton@salud.unm.edu |
| Sanjeev Arora, University of New Mexico, sarora@salud.unm.edu |
| Abstract: Using telemedicine infrastructure for clinical learning, Project ECHO (Extension for Community Healthcare Outcomes) trains primary care providers who serve rural, underserved and prison populations in providing Hepatitis C (HCV) treatment for their patients with the guidance of specialists. This model is relevant to populations with chronic, complex diseases or with limited access to specialists (Arora et al, 2007). Using cased-based and iterative learning, primary care providers present HCV patients during weekly telemedicine clinics. Specialist physicians co-manage patients with primary care providers utilizing evidence-based treatment protocols. Learning loops are created through iterative and collaborative case-based discussion during each clinic. The evaluation views the provider as learning a complex set of behaviors with extensive support in a community of similar practitioners. The evaluation draws from self efficacy, situated learning and social cognitive theory. Providers endorse their knowledge growth and self-efficacy about Hepatitis C screening, symptom management, and treatment. |
| Using Formative Evaluation to Culturally Adapt a Parenting Program for Black Fathers of Children With Special Needs |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| James C Bridgers Jr, University of Maryland, College Park, jbridgers@mayatech.com |
| Suzanne M Randolph, University of Maryland, College Park, suzanner@umd.edu |
| Cecil H Doggette, Health Services for Children with Special Needs Inc, cdoggette@hscsn.org |
| Sally Koblinsky, University of Maryland, koblinsk@umd.edu |
| Abstract: Ecological systems theory was used in a formative evaluation to assess the feasibility of a model program, developed for Black mothers' normative parenting, for use with Black fathers of children with special needs. There is scant literature on Black fathering; and parenting programs often focus on parents' challenges rather than strengths. This evaluation examined the rewards, strengths, challenges, and support needs of African American fathers parenting children with special needs. The theoretical framework guided development of focus group guides and analyses. Data were used to culturally adapt the model parenting program and select evaluation measures for the locally implemented intervention. |
| The Use of Evaluation to Impact Comprehensive School Reform Over Time: Michigan Coalition of Essential Schools 2002-2007 |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tara Donahue, Learning Point Associates, tara.donahue@learningpt.org |
| Abstract: In 2002, the Michigan Coalition of Essential Schools (MCES) received their first cohort of schools that chose them as the model provider for a Comprehensive School Reform (CSR) grant. In subsequent years, three more cohorts followed. This paper analyzes the role of utilization evaluation over five years of evaluation work on the CSR model used by MCES. Through survey data and interviews with teachers and MCES staff, this paper will examine the growth that each experienced with the model and how the use of evaluation impacted the work done by the MCES organization, the Leadership Teams in the CSR schools, and the School Reform Coaches assigned to each school. Although schools from each cohort faced similar challenges to implementing the practices espoused by MCES, MCES staff used the evaluation data to make changes to the implementation model. This paper reflects how the evaluation ultimately changed the CSR model. |
| Evaluating an Online New Teacher Mentorship Program: Science, Technology, Engineering, and Mathematics Mentoring Program (STEMMP) |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Herbert Struss, University of Minnesota, strus010@umn.edu |
| Joel Donna, University of Minnesota, donna010@umn.edu |
| Gillian Roehrig, University of Minnesota, roehr0013@umn.edu |
| Abstract: The subject of the evaluation was a virtual learning community – Science, Technology, Engineering, and Mathematics Mentoring Program (STEMMP). The goals of STEMMP were to improve teacher job satisfaction and to increase teachers' use of reform-based teaching strategies, with the ultimate goal being to improve student achievement. Besides outlining the evaluation, the poster highlights the challenges of evaluating online learning communities. Approximately sixty beginning (untenured) STEM teachers throughout Minnesota participated in the STEMMP program, along with sixty experienced STEM teachers who served as mentors to the untenured teachers. Teachers represented a wide range of settings from urban to rural, and public to charter. Electronic communication features of STEMMP included private chat rooms and larger forums. For example, the inquiry discussion area was developed as a learning cycle where the teacher planned, implemented, and reflected on a lesson or unit with their peers and mentor. |
| Implementation of Evidence-based Interventions in Community Mental Health Settings: Organizational Factors and Provider Perspectives Impacting Implementation Processes and Child and Family Outcomes |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Elizabeth Douglas, Macro International Inc, elizabeth.b.douglas@orcmacro.com |
| Christine Walrath, Macro International Inc, christine.m.walrath-greene@orcmacro.com |
| Bhuvana Sukumar, Macro International Inc, bhuvana.sukumar@orcmacro.com |
| Abstract: A growing body of knowledge exists about evidence-based practices (EBPs) that work to improve outcomes for children with mental health needs. However, as much of this research is conducted in tightly controlled settings (Burns & Hoagwood, 2002), it often reflects intervention efficacy while leaving questions about effectiveness in diverse "real-world" settings. Moreover, reports indicate that even in the case of interventions found to be effective through evaluation in clinical settings, the protocols are not widely understood by clinicians, and are not being translated into practice nearly often enough (NIMH, 2000). As part of an effort to contribute to the knowledge base regarding EBP implementation and effectiveness, this symposium will present findings related to organizational factors and provider perspectives impacting implementation as well as the results of a study examining the effectiveness of Parent-Child Interaction Therapy in community mental health settings. Implications for future implementation and effectiveness research will be discussed. |
| Having Graduate Students as Evaluators: A Description of a Community Based Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shawn Dickerson, Old Dominion University, sdick003@odu.edu |
| Abby Braitman, Old Dominion University, abraitma@odu.edu |
| Krystall Dunaway, Old Dominion University, kdunaway@odu.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Abstract: The current evaluation was a team evaluation conducted by graduate students which utilized an online survey to collect information from the stakeholders of interest. Evaluators gathered and examined information concerning different aspects of the job roles for Uniform Patrol Officers (UPOs) and Neighborhood Impact Officers (NIOs) from a local community's Police Department. The researchers assessed attitudes and behavior in regards to the officers' training and responsibilities, mainly pertaining to the Weed & Seed program. The Weed & Seed program was established to assist the city in reducing violent crime, the trafficking of illegal drugs and reducing juvenile crime rates. This outcome-focused evaluation gained vital information for community policing and increased the understanding of police officers beliefs and increase knowledge about their training. |
| Consumer Assessment of Mental Health Service Providers' Cultural Competence |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amanda Gmyrek, University of Maryland, agmyrek@psych.umaryland.edu |
| Natasha Durant, University of Maryland Medical System, ndurant@kernan.umm.edu |
| Diana Seybolt, University of Maryland, dseybolt@psych.umaryland.edu |
| Iris Reeves, Maryland Mental Hygiene Administration, ireeves@dhmh.state.md.us |
| Abstract: According to the President's New Freedom Commission on Mental Health, one of the driving forces in transforming America's mental health system is to better address the mental health needs of our multicultural society by increasing access to culturally competent services. Currently, there are instruments that assess mental health service providers' cultural competence at the program- and individual practitioner-level, but few measures examine provider cultural competence from the consumers' perspective. This distinction is important given that the Commission also highlighted the need for consumer-driven care. This project was designed to examine the psychometric properties of a survey tool to be used in the assessment of provider cultural competence from the consumers' point of view. Data were collected from 406 adult mental health consumers who were actively engaged in treatment. Data analyses revealed that the survey tool had strong internal consistency reliability and content validity as well as adequate test-retest reliability. |
| HIV Prevention Among High School Students: Evaluation of a Peer Educator Project |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Manuel Voelkle, University of Mannheim, voelkle@rumms.uni-mannheim.de |
| David Dwertmann, University of Mannheim, ddwertma@rumms.uni-mannheim.de |
| Abstract: The poster provides an overview of the evaluation of a community based Peer Educator Program for HIV prevention among high school students. Approximately N = 500 teenagers from three different schools participated in the project. A quasi-experimental pre-post test design with a three month follow-up questionnaire was chosen to assess changes in knowledge, attitude and reported prevention behavior. An innovative combination of Ajzen's (1991) 'Theory of Planned Behavior' and the 'Information-Motivation-Behavioral-Skills' model, proposed by Fisher & Fisher (1992), constitutes the theoretical foundation of the study. The poster addresses the study-design, implementation of the program, assessment of data and results. A special emphasis is put on the psychometric quality of a newly developed questionnaire and the suitability of the underlying theoretical model. We conclude with some recommendations for model improvement. |
| Using Latent Growth Modeling to Test Effectiveness of Georgia Family Connection Collaboratives |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Adam Darnell, EMSTAR Research Inc, darnelladam@hotmail.com |
| James Emshoff, EMSTAR Research Inc, jemshoff@gsu.edu |
| Steve Erickson, EMSTAR Research Inc, ericksoneval@att.net |
| Abstract: The Family Connection initiative is a statewide network of community collaboratives that aims to address serious challenges facing Georgia's children and families. Currently 159 collaboratives, one in each county of the state, are implementing community-based support systems in accordance with a set of operating principles (a Theory of Change) designed to improve health-, education- and economic-related outcomes. Outcomes have been measured annually for each of the 159 counties over the eleven year period from 1994 to 2004. Longitudinal change in outcomes was modeled using latent growth curve modeling. Predictors, also measured longitudinally, include characteristics of collaborative structure and process. Alternative analytic approaches for demonstrating causal effects between collaborative characteristics and county-level outcomes will be presented. Discussion will address the strengths and weaknesses of the different approaches to associating change between multiple longitudinal variables. |
| Evaluator Learning Curve: Is it a Gamble for Stakeholders? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Caroline Lippy, Georgia State University, clippy1@student.gsu.edu |
| Leanne Valentine, Georiga State University, leannevalentine@comcast.net |
| Jim Emshoff, Georgia State University, psyjge@langate.gsu.edu |
| Beth Anthony, Georgia State University, eanthony1@student.gsu.edu |
| Dary Enkhtor, Georgia State University, denkhtor1@student.gsu.edu |
| Ayana Perkins, Georgia State University, ayanaperkins@msn.com |
| Lindsey Zimmerman, Georgia State University, lindsetzimmerman@gmail.com |
| Abstract: This poster examines a series of research projects funded by the Georgia Department of Human Resources, focusing on a comparison of two prevalence studies of problematic gambling behavior. Serving as needs assessments, the studies, conducted in 1999 and 2007, each completed approximately 1600 randomized telephone interviews of Georgia residents and helped to outline the need for problem gambling services in the state. This poster examines several modifications between the first and second study, indicating an evaluator learning curve. Specifically, evaluators utilized an alternative measure of pathological gambling and also became more aware of race/ethnicity interviewer effects on the sample. While such modifications improved the quality of the evaluation, they potentially induced unintended negative consequences for the state, including a decreased understanding of the effectiveness of previous efforts implementing evaluator recommendations from the first study. The implications and recommendations for future research are made for both evaluators and state officials. |
| Evaluator Learning Curve: Is it a Gamble for Stakeholders? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Caroline Lippy, Georgia State University, clippy1@student.gsu.edu |
| Leanne Valentine, Georiga State University, leannevalentine@comcast.net |
| Jim Emshoff, Georgia State University, psyjge@langate.gsu.edu |
| Beth Anthony, Georgia State University, eanthony1@student.gsu.edu |
| Dary Enkhtor, Georgia State University, denkhtor1@student.gsu.edu |
| Ayana Perkins, Georgia State University, ayanaperkins@msn.com |
| Lindsey Zimmerman, Georgia State University, lindsetzimmerman@gmail.com |
| Abstract: This poster examines a series of research projects funded by the Georgia Department of Human Resources, focusing on a comparison of two prevalence studies of problematic gambling behavior. Serving as needs assessments, the studies, conducted in 1999 and 2007, each completed approximately 1600 randomized telephone interviews of Georgia residents and helped to outline the need for problem gambling services in the state. This poster examines several modifications between the first and second study, indicating an evaluator learning curve. Specifically, evaluators utilized an alternative measure of pathological gambling and also became more aware of race/ethnicity interviewer effects on the sample. While such modifications improved the quality of the evaluation, they potentially induced unintended negative consequences for the state, including a decreased understanding of the effectiveness of previous efforts implementing evaluator recommendations from the first study. The implications and recommendations for future research are made for both evaluators and state officials. |
| Using Latent Growth Modeling to Test Effectiveness of Georgia Family Connection Collaboratives |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Adam Darnell, EMSTAR Research Inc, darnelladam@hotmail.com |
| James Emshoff, EMSTAR Research Inc, jemshoff@gsu.edu |
| Steve Erickson, EMSTAR Research Inc, ericksoneval@att.net |
| Abstract: The Family Connection initiative is a statewide network of community collaboratives that aims to address serious challenges facing Georgia's children and families. Currently 159 collaboratives, one in each county of the state, are implementing community-based support systems in accordance with a set of operating principles (a Theory of Change) designed to improve health-, education- and economic-related outcomes. Outcomes have been measured annually for each of the 159 counties over the eleven year period from 1994 to 2004. Longitudinal change in outcomes was modeled using latent growth curve modeling. Predictors, also measured longitudinally, include characteristics of collaborative structure and process. Alternative analytic approaches for demonstrating causal effects between collaborative characteristics and county-level outcomes will be presented. Discussion will address the strengths and weaknesses of the different approaches to associating change between multiple longitudinal variables. |
| Let Me Tell You What I Need! Using Multiple Perspectives to Explore the Needs of Adolescents After a Concussion |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Isabelle Gagnon, University of Montreal, isabelle.gagnon6@sympatico.ca |
| Swaine Bonnie, University of Montreal, bonnie.swaine@umontreal |
| François Champagne, University of Montreal, francois.champagne@umontreal.ca |
| Helene Lefebvre, University of Montreal, |
| Debbie Feldman, University of Montreal, |
| Jeff Atkinson, Montreal Children's Hospital, |
| Abstract: Practice guidelines have been published concerning the treatment of individuals after a concussion, but most target adults or children, with no particular emphasis on adolescents. Furthermore, most intervention programs put in place have been developed through expert consensus without the specific input of adolescents themselves. As an initial step in improving the care provided to adolescents after a concussion, their specific needs were assessed using different perspectives including those of the literature, of service providers and, especially, that of adolescents as users of the existing services. Data collection strategies included a systematic review of the literature, interviews with adolescents and with parents, focus groups of expert service providers and a postal survey. Our results demonstrate that, even after an event considered by many as trivial, adolescents have service needs that are unique and particularly important during adolescence. Clear benefits can be obtained by including adolescents themselves in evaluations addressing issues related to them and preconceived ideas about their needs should be discarded in favor of a more collaborative approach. |
| Impact of AgrAbility Program: Helping Farmers to Stay in Farming |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Connie Baggett, Pennsylvania State University, bbc@psu.edu |
| Rama Radhakrishna, Pennsylvania State University, brr100@psu.edu |
| Linda Fetzer, Pennsylvania State University, lmf8@psu,edu |
| Abstract: Agriculture work is ranked as one of the most dangerous occupations in the nation (RTC, 2000). AgrAbility is a four year grant project through the U.S. Department of Agriculture that provides direct services to farmers with a disability or injury. Services provided by AgrAbility include disabled farmer evaluation, work sites assessments, modification recommendations, suggesting equipment adaptation, task restructuring, mobilizing and coordinating community resources and services, and facilitating rural independent living, etc. The overall purpose of the study was to assess the impact of Agrability Program. Overall, several of the targets of the project were met. Assessments and site visits conducted by AgrAbility staff has helped farmers to perform farm related tasks and continue farming. During the implementation phase, we encountered several challenges and barriers. In order to address these challenges, several changes are underway in the new project cycle (2006-10) which was recently funded by USDA. |
| Collaboration and Evaluation Capacity Building: Lessons From a Learning Environment |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Randi K Nelson, University of Minnesota, nelso326@umn.edu |
| David Fischer, University of Minnesota, fisch413@umn.edu |
| Herbert Struss, University of Minnesota, strus0101@umn.edu |
| Abstract: Evaluation, data based decision making, and collaboration have tremendous potential for building the capacity of teachers and program evaluators to learn from evaluation. This poster highlights the views of teachers and program staff toward collegiality and data use to inform practice, during a three-year collaborative evaluation capacity building project between a university and two public schools. Program evaluation traditionally focuses on stakeholders who receive program services. In collaborations, the views of both the service providers and recipients must be considered. This poster illustrates how annual surveys of teachers and follow up interviews with teachers and program staff have been used to provide all stakeholders the opportunity to learn from their experiences. In this project, use of data from teachers and program staff demonstrates collaboration and data based decision making among all participants and models evaluation to promote collaborative capacity and learning in addition to its more traditional purposes. |
| Evaluating the Capacity and Quality of an Early Care and Education System |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rob Fischer, Case Western Reserve University, fischer@case.edu |
| Kristen Mikelbank, Case Western Reserve University, kristen.mikelbank@case.edu |
| Claudia Coulton, Case Western Reserve University, claudia.coulton@case.edu |
| Abstract: Since 1999, a public/private partnership in Cuyahoga County, Ohio, has been engaged in a comprehensive effort focused on the well-being of children under the age of six and their families. As part of its strategic planning process, the partnership identified goals around enhancing the early care and education system for the County's children. The present study provides a detailed assessment of the County's regulated early care system in regard to its capacity and quality, by combining administrative and observational data. Using geographic analyses, areas within the County exhibiting relative need for additional child care slots are identified. The study demonstrates the effective integration of a range of data sources in a geographic analysis designed to assist in County planning for the needs of young children and their families. |
| Using Qualitative Information to Develop Materials to Address Clinical and Psychosocial Needs of Lesbian, Gay, Bisexual, Transgendered, Questioning, Intersexed, and Two-Spirit (LGBTQI2S) Youth in Systems of Care |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sylvia Fisher, United States Department of Health and Human Services, sylvia.fisher@samhsa.hhs.gov |
| Jeffrey Poirier, American Institutes for Research, jpoirier@air.org |
| Abstract: Lesbian, gay, bisexual, transgendered, questioning, intersexed, and two-spirit (LGBTQI2-S) youth frequently do not receive appropriate and culturally-competent mental health interventions, partly due to the unavailability of useful materials and practice guides for providers. This presentation describes interview data results collected from GLBTQ youth and systems of care program directors with the purpose of developing useful and appropriate materials to address the needs of GLBTQ youth in systems of care. GLBTQ youth were interviewed using detailed protocols emphasizing issues associated with stigmatization, mental health service experiences, useful interventions, and related issues. Implications of this research activity for future research and practice interventions with sexual minority youth are also provided. |
| Conducting and Evaluating Effective Translations to Promote Linguistic Competence in a Federally Funded National Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sylvia Fisher, United States Department of Health and Human Services, sylvia.fisher@samhsa.hhs.gov |
| Michelle Schurig, Macro International Inc, michelle.l.schurig@orcmacro.com |
| Abstract: Translation is a complex process laden with numerous challenges, including (a) preserving the integrity and semantic equivalence between the original written or oral text and the translated text; and (b) maintaining a culturally sensitive translation that ensures that cultural traditions and typical language idioms are addressed in a respectful manner. In addition, successful oral and written translations reduce the potential for statistical and other forms of bias whenever possible, somewhat minimizing the potential for error in data collection activities (Fisher & Gerber, 2002). Several translation approaches are available, including back translation, expert groups (such as a translation advisory committee), cognitive interviews, focus groups, item response theory, respondent and interviewer debriefings, certified translators, use of the decentering method, and behavior coding. Marín and Marín (1991) especially favor translation-by-committee approaches for Spanish translations because Spanish is spoken in nearly 25 nations and, accordingly, is subject to many idioms and variations both in meaning and pronunciation. These variations and cultural idioms and differences can result in translations that have radically different meanings. Additionally, if a translation is not well-designed from its inception, formidable efforts often become necessary to disentangle the roots of the translation problem, which is difficult once a translation is completed. The literature on translation helped justify a decision for the national evaluation of systems of care for children with serious mental health challenges to develop an iterative process of review and feedback, through the establishment of a translation committee with strong links to the audience for translated evaluation products. This poster describes the process and the outcome of this approach to translation. |
| Robotics: Opportunities for Building Outstanding Talent in the Sciences (R.O.B.O.T.S.): The Collaborative Evaluation of a Middle School Science, Technology, Engineering, and Mathematics (STEM) Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tangie Fleming, University of North Carolina, Chapel Hill, tangie.fleming@gmail.com |
| Rita O'Sullivan, University of North Carolina, Chapel Hill, ritao@unc.edu |
| Abstract: The Robotics: Opportunities for Building Outstanding Talent in the Sciences (R.O.B.O.T.S.) middle school project seeks to provide a model for building a sustained community-wide awareness about the value of astronomy and physics, which are needed for success in virtually all science, technology, engineering, and mathematics (STEM) fields. An evaluation study was commissioned to investigate how middle school students form and change their attitudes about the world around them in the context of thinking about career options in STEM disciplines. The evaluation investigates the underlying reasons and relationships that influence outcome attainment. Further, it tests the model's assumptions about the experiential strategies and how they influence the desired outcomes. This session presents the methodology of the collaborative evaluation used to validate the theoretical framework upon which R.O.B.O.T.S. is based. |
| The Mississippi Delta Children's Partnership: A Child-focused Community Change Initiative |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bettye Fletcher, Professional Associates Inc, bwfletcher@paionline.org |
| LaTonya Lott, Professional Associates Inc, llott@paionline.org |
| Aisha Fletcher, Professional Associates Inc, afletcher@paionline.org |
| Abstract: Mississippi Delta Children's Partnership (MDCP) is a collaborative community response the grave need for after school programming for children in this impoverished Mississippi Delta region. The mechanism for the delivery of the child-focused component of the initiative is after school and summer enrichment programs called Children's Village. The Children's Village is a community-based site which is an inspirational and supportive learning environment for children to realize and develop their unique abilities. Each of the five sites has independently designed child- focused project that addresses the four determinants derived during the conceptual process (language development, cognitive and social development and self and ethnic group efficacy). It is a highly structured comprehensive enrichment program which includes critical thinking, cultural, and academic enrichment. This poster presents the evaluative findings from the first year of the Mississippi Delta Children's Partnership. |
| The Mississippi Delta Children's Partnership: A Child-focused Community Change Initiative |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bettye Fletcher, Professional Associates Inc, bwfletcher@paionline.org |
| LaTonya Lott, Professional Associates Inc, llott@paionline.org |
| Aisha Fletcher, Professional Associates Inc, afletcher@paionline.org |
| Abstract: Mississippi Delta Children's Partnership (MDCP) is a collaborative community response the grave need for after school programming for children in this impoverished Mississippi Delta region. The mechanism for the delivery of the child-focused component of the initiative is after school and summer enrichment programs called Children's Village. The Children's Village is a community-based site which is an inspirational and supportive learning environment for children to realize and develop their unique abilities. Each of the five sites has independently designed child- focused project that addresses the four determinants derived during the conceptual process (language development, cognitive and social development and self and ethnic group efficacy). It is a highly structured comprehensive enrichment program which includes critical thinking, cultural, and academic enrichment. This poster presents the evaluative findings from the first year of the Mississippi Delta Children's Partnership. |
| Institutionalizing Utilization-focused approaches to Evaluation in Non-Governmental Organizations (NGOs): Creating a Culture of Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carrie Willimann, Academy for Educational Development, cwillimann@aed.org |
| Winnehl Tubman, Academy for Educational Development, wtubman@aed.org |
| HyeJin Kim, Academy for Educational Development, hkim@aed.org |
| Arushi Terway, Academy for Educational Development, aterway@aed.org |
| Ana Florez, Academy for Educational Development, aflorez@aed.org |
| Audrey-Marie Moore, Academy for Educational Development, amoore@aed.org |
| Abstract: Evaluation can be a tool for ensuring that education projects provide data and information at the policy level to transfer technical innovation. However, evaluation, as currently structured under many donor funded projects, does not provide the necessary information to inform this process. Nor does it allow for a cohesive approach to draw lessons learned across projects to inform broader policy. Clearing space in the existing political economic landscape so that the substance of sound educational improvement can be implemented is the operational core of what a network of institutions and actors (-reform support infrastructure-) needs to do for reform to succeed. This panel examines how a small evaluation group within an international NGO attempted to clear and fill space for utilization-focused evaluation to flourish in order to create a more cohesive approach to evaluation, build the capacity of staff to implement UFE frameworks, and improve organizational learning across education projects. |
| How are They Faring? Using the South Carolina Data Warehouse to Track Outcomes for Young Adults After They Leave the Foster Care System |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Cynthia Flynn, University of South Carolina, cynthia.flynn@sc.edu |
| Abstract: Research shows that between 30 and 60% of the youth who age out of foster care each year will experience an array of negative consequences including economic instability, educational disadvantage, criminal activity, victimization, homelessness, health problems, substance abuse, and early parenting. To begin to address these unacceptable outcomes, Congress passed the Foster Care Independence Act of 1999. As a result, youth in foster care are now receiving help to better prepare them for life in the adult world. Politicians and child welfare administrators at the state and national level are asking whether these services are improving outcomes. In South Carolina, an interagency team is examining outcomes for adults across an array of indicators using the state's data warehouse. This poster session will describe an integrated database, how it is being used to assess outcomes, and how former foster youth in South Carolina are faring as adults. |
| Increasing Commitment to Organizational Change Through Evaluation and Learning |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rex Foster, University of Minnesota, rexfost@aol.com |
| Abstract: This paper explores evaluation utilization as a means for fostering employee commitment to organizational change. Many change efforts fail to fully achieve their intended goals, and a lack of employee commitment to these changes has been identified as one cause for such failures. In addition, there is little evidence surrounding what practices effectively cultivate commitment to change. Using the Cousins and Leithwood's (1986) conceptual framework for evaluation utilization, and Chin and Benne's (1961) normative re-educative change strategy, this paper proposes that evaluation utilization can contribute to higher commitment to change (as conceptualized by Herscovitch and Meyer, 2002). The paper includes a review of the literature on commitment to change and evaluation utilization, followed by a synthesis and a theoretical proposition of how the two concepts intersect. |
| Inclusion of Children With Disabilities in School Age Care Settings: Provider's Perspective |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Hamida Jinnah, University of Georgia, hamidajinnah@gmail.com |
| Zolinda Stoneman, University of Georgia, zo@ihdd.uga.edu |
| Christine Todd, University of Georgia, ctodd@fcs.uga.edu |
| Anika Francis, University of Georgia, anikakai@uga.edu |
| Abstract: This study assesses the needs of child care providers related to inclusion of children with disabilities in school age care settings. Focus groups and phone interviews were conducted with providers. Data was analyzed using Nudist (N6). Providers mentioned that they enjoy seeing children progress in their abilities and are constantly rewarded by the love and affection children with disabilities give to them. Providers however, also experience several challenges in including children with disabilities in their setting(s). These include inadequate staff-child ratio, staff attitudes, staff training and access to information and resources for inclusion. Additionally, providers face problems due to gaps in communication with parents, school teachers and therapists. Providers would like to have more training, more resources including additional staff, more funding, and more information about available resources. They would like to see greater collaboration and networking among different individuals working with school age children having disabilities. |
| Impact of a Financial Program on Knowledge and Practices of Adult Participants |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Cathy Bowen, Pennsylvania State University, cfb4@psu.edu |
| Marilyn Furry, Pennsylvania State University, mfurry@psu.edu |
| Rama Radhakrishna, Pennsylvania State University, brr100@psu.edu |
| Abstract: Reports continue to document the financial illiteracy of Americans. Lack of financial skills is illustrated by bankruptcy rates, increased credit card debt, and low savings rate. The purpose of this study was to assess the effectiveness of a financial education program on the knowledge and practices of participants. A pretest-posttest comparison group design was used to determine if participation in the program would lead to: 1) using a spending plans and recordkeeping system 2) starting a savings program or increase the amount contributed to an existing savings plan, 3) developing a plan to reduce the current debt load, and 4) using techniques to reduce their cost of using credit. Significant differences were found between the treatment and comparison groups' knowledge score on the posttest and delayed posttest. Treatment group participants not only made changes in behaviors relative to managing personal finances but also sustained those changes over a longer period. |
| Let Me Tell You What I Need! Using Multiple Perspectives to Explore the Needs of Adolescents After a Concussion |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Isabelle Gagnon, University of Montreal, isabelle.gagnon6@sympatico.ca |
| Swaine Bonnie, University of Montreal, bonnie.swaine@umontreal |
| François Champagne, University of Montreal, francois.champagne@umontreal.ca |
| Helene Lefebvre, University of Montreal, |
| Debbie Feldman, University of Montreal, |
| Jeff Atkinson, Montreal Children's Hospital, |
| Abstract: Practice guidelines have been published concerning the treatment of individuals after a concussion, but most target adults or children, with no particular emphasis on adolescents. Furthermore, most intervention programs put in place have been developed through expert consensus without the specific input of adolescents themselves. As an initial step in improving the care provided to adolescents after a concussion, their specific needs were assessed using different perspectives including those of the literature, of service providers and, especially, that of adolescents as users of the existing services. Data collection strategies included a systematic review of the literature, interviews with adolescents and with parents, focus groups of expert service providers and a postal survey. Our results demonstrate that, even after an event considered by many as trivial, adolescents have service needs that are unique and particularly important during adolescence. Clear benefits can be obtained by including adolescents themselves in evaluations addressing issues related to them and preconceived ideas about their needs should be discarded in favor of a more collaborative approach. |
| Consumer Assessment of Mental Health Service Providers' Cultural Competence |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amanda Gmyrek, University of Maryland, agmyrek@psych.umaryland.edu |
| Natasha Durant, University of Maryland Medical System, ndurant@kernan.umm.edu |
| Diana Seybolt, University of Maryland, dseybolt@psych.umaryland.edu |
| Iris Reeves, Maryland Mental Hygiene Administration, ireeves@dhmh.state.md.us |
| Abstract: According to the President's New Freedom Commission on Mental Health, one of the driving forces in transforming America's mental health system is to better address the mental health needs of our multicultural society by increasing access to culturally competent services. Currently, there are instruments that assess mental health service providers' cultural competence at the program- and individual practitioner-level, but few measures examine provider cultural competence from the consumers' perspective. This distinction is important given that the Commission also highlighted the need for consumer-driven care. This project was designed to examine the psychometric properties of a survey tool to be used in the assessment of provider cultural competence from the consumers' point of view. Data were collected from 406 adult mental health consumers who were actively engaged in treatment. Data analyses revealed that the survey tool had strong internal consistency reliability and content validity as well as adequate test-retest reliability. |
| Learning Through Focus Groups: Evaluation of the Treatment Planning Process by Clinicians Within a Community Health Center |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Erica Gosselin, Mental Health Center of Denver, erica.gosselin@mhcd.org |
| Steve Baumer, Mental Health Center of Denver, steve.baumer@mhcd.org |
| Kristi Helvig, Mental Health Center of Denver, kristi.helvig@mhcd.org |
| Antonio Olmos, Mental Health Center of Denver, antonio.olmos@mhcd.org |
| Abstract: At the 2006 AEA conference, the Mental Health Center of Denver (MHCD) presented a paper regarding mental health consumers' perceived participation in their treatment plans. Results from these focus groups yielded specific areas for follow-up regarding the perceived utility of the Individual Service Plan (ISP) and consumers' perception of involvement in the process. Following this study, we wanted to elicit additional information in order to create a comprehensive plan for change with the aim of improving quality. Based upon an identified area for follow-up, we conducted 12 focus groups with clinicians who work with the consumers at MHCD. These focus groups were conducted to explore 1) the clinician's perception of consumer's participation in treatment, 2) the clinician's perception of the ISP and ISP process, and 3) changes clinicians would like to see in this process. Results from the clinician focus groups were correlated with the prior data from consumer focus groups to formulate targeted changes in the ISP process. |
| Successful Organization Learning and Evaluation Capacity Building Among Nonprofits |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michele Graham, JVA Consulting LLC, michele@jvaconsulting.com |
| Rebecca Baggett, JVA Consulting LLC, rebecca@jvaconsulting.com |
| Sheridan Green, JVA Consulting LLC, sheridan@jvaconsulting.com |
| Abstract: JVA Consulting, LLC, developed a capacity-building program for community-and faith-based organizations with funding from the U.S. Department of Health and Human Services. The program was designed to increase organizational capacity in numerous areas, including evaluation. Capacity building services were provided for nine months and organizations took part in baseline and follow-up assessments of evaluation capacity. The purpose of the study was to examine factors related to capacity growth in evaluation and to subsequent organizational learning. Research questions include: 1) were there significant gains in evaluation capacity? 2) what organizational factors were associated with gains in evaluation capacity; and 3) how did organizational learning with regard to evaluation manifest at the individual or organizational level? A mixed method design was used for the study; a quantitative assessment tool measured capacity and organization characteristics and organizational learning was tapped through informal interview notes. Findings indicate significant gains in capacity and revealed important successes with regard to organizational learning. |
| Newly Revised Authentic Project and Rubric for Use in a Graduate Level Program Evaluation Course |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Zandra Gratz, Kean University, zgratz@aol.com |
| Abstract: Described is an authentic evaluation project developed as part of a graduate level program evaluation course. Central to the course, students were given an actual request for proposals and asked to develop an evaluation plan. An extensive, 16 quality rubric was delineated and recently revised in accordance with student feedback; rubric elements included program theory assessment, development of a logic model, identification of mixed models of assessment, and specification of process, outcome and impact evaluation efforts. The rubric not only created a performance contract, but clarified the criteria for grading and often served as the focus of discussion. Beyond the course applications, it is likely that the rubric will provide a format for reflection of project proposals and evaluation designs. |
| Assertive Community Treatment in Atlanta: Exploring Particular Outcomes of a Grady Healthcare System's Approach |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Angela Mooss, Georgia State University, amooss1@student.gsu.edu |
| John Barile, Georgia State University, jbarile1@student.gsu.edu |
| Joanna Weinberg, Georgia State University, jweinberg3@student.gsu.edu |
| Doyanne Darnell, Georgia State University, dhorst1@student.gsu.edu |
| Brandeis Green, Georgia State University, bgreen8@student.gsu.edu |
| Abstract: Extensive research indicates that the Assertive Community Treatment (ACT) model of mental health service delivery is effective in treating persons with severe mental illness. The Atlanta based ACT team (ECSEL) at the Grady Healthcare System employs a “housing first” philosophy which holds that effective service delivery cannot occur without stable housing. Supportive housing for homeless people with severe mental illness has been found to reduce overall service use, incarcerations and hospitalizations (Culhane, et. al. 2001). The ACT model is predicted to positively impact overall factors of wellbeing, such as housing satisfaction, familial relations, employment opportunities, prevalence of psychiatric symptoms, and reductions in substance abuse. This presentation examines the outcomes of the above factors among a homeless sample in Atlanta, and explores the unique challenges and successes of data collection in a variety of settings. Subsequent outcomes and issues of diversity among program participants and between the participants and members of the ACT team will also be addressed regarding fidelity to the original ACT model. |
| Successful Organization Learning and Evaluation Capacity Building Among Nonprofits |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michele Graham, JVA Consulting LLC, michele@jvaconsulting.com |
| Rebecca Baggett, JVA Consulting LLC, rebecca@jvaconsulting.com |
| Sheridan Green, JVA Consulting LLC, sheridan@jvaconsulting.com |
| Abstract: JVA Consulting, LLC, developed a capacity-building program for community-and faith-based organizations with funding from the U.S. Department of Health and Human Services. The program was designed to increase organizational capacity in numerous areas, including evaluation. Capacity building services were provided for nine months and organizations took part in baseline and follow-up assessments of evaluation capacity. The purpose of the study was to examine factors related to capacity growth in evaluation and to subsequent organizational learning. Research questions include: 1) were there significant gains in evaluation capacity? 2) what organizational factors were associated with gains in evaluation capacity; and 3) how did organizational learning with regard to evaluation manifest at the individual or organizational level? A mixed method design was used for the study; a quantitative assessment tool measured capacity and organization characteristics and organizational learning was tapped through informal interview notes. Findings indicate significant gains in capacity and revealed important successes with regard to organizational learning. |
| Classroom-level Influences on Literacy Growth Among Early Reading First Students |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sheridan Green, JVA Consulting LLC, sheridan@jvaconsulting.com |
| Susan Hutchinson, University of Northern Colorado, susan.hutchinson@unco.edu |
| Ana Leon, JVA Consulting LLC, ana@jvaconsulting.com |
| Jane Walsh, Denver Public Schools Early Reading First, jane_walsh@dpsk12.org |
| Abstract: The Denver Public Schools Early Reading First program utilizes a literacy-focused curriculum and a substantive professional development component for program teachers. Approximately 180 students are assessed three times annually to gauge improvements in early literacy. This paper will present the results of examining the effects of classroom characteristics on student literacy. Because these effects are nested within schools, hierarchical linear modeling will be used for analysis. Questions addressed: 1) what is the influence of classroom-level factors on literacy growth? And 2) do classroom-level factors differ in their effects on literacy scores? Study data comprise four categories: measures of early literacy skills, measures of classroom teacher qualifications and characteristics, measures of the classroom literacy environment, and measures of student, teacher, and school characteristics that we will employ as analytical controls. Dependent variables include receptive vocabulary and phonological awareness using the Peabody Picture Vocabulary Test-III and the Phonological Awareness and Literacy Screening. |
| Evaluating Outreach Strategies to Promote HIV Testing Among Young African-American Men who Have Sex With Men: The Case for Youth Empowerment |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| George Greene, Working For Togetherness, george.wft@sbcglobal.net |
| Clifford Armstead, Working For Togetherness, cliffordarmstead@sbcglobal.net |
| Gregory Norels, Working For Togetherness, gnorels@hotmail.com |
| Sean Morgan, Working For Togetherness, smorgan.wft@sbcglobal.net |
| Abstract: Increasing HIV rates among young, African-American men who have sex with men (YAAMSM) suggest this population is not being effectively served by traditional prevention programs. This poster articulates our organizational response to the epidemic and presents an evaluation of our multi-pronged outreach approach. Incorporating urban marketing strategies into HIV prevention for YAAMSM, our agency provides both mobile community-wide and fixed-site outreach in HIV-impacted neighborhoods and high-risk settings by offering HIV/STD education, counseling, and testing. Additionally, our agency opened a youth empowerment center to equip participants with resources to manage their healthcare needs. Preliminary evaluation data indicate that mobile- and fixed-site outreach located large numbers of YAAMSM for information and condom dissemination; however, youth-centered services in a GLBTQ safe space better promoted HIV testing among youth. Taken together, this service delivery approach minimizes cultural barriers to health-seeking behaviors and delivers HIV prevention services to groups disproportionately affected by the HIV epidemic. |
| Three Decades of Program Evaluation in Brazil: Lessons Learned and the Case of Roberto Marinho Foundation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Thereza Penna Firme, Cesgranrio Foundation, therezapf@uol.com.br |
| Vilma Guimarães, Roberto Marinho Foundation, vilma@frm.org.br |
| Angela Dannemann, D&S Engenheiras Associadas Ltda, angcdann@terra.com.br |
| Ana Carolina Letichevsky, Cesgranrio Foundation, anacarolina@cesgranrio.org.br |
| Abstract: This paper sheds light on lessons learned over thirty years of evaluation in Brazil. A crucial aspect of this journey was learning the do's and the don'ts of evaluation. Some findings: respect stakeholders values, take community knowledge into account, report outcomes with appropriate language including potential and limitations, maintain an attitude of social responsibility. Also, never impose evaluations, hurt values in the name of accuracy or breach agreements without consent. The permanent concern with program evaluation standards while meeting the Brazilian challenges of institutional diversity in the face of social vulnerability has demanded creativity and courage from all evaluators. This apparent dilemma requires the courage to blend and use multiple approaches tuned to the changing reality in order to promote inclusion, dialogue, and appreciation. Hence, this paper focuses on the evaluation case of one of the most renowned Brazilian foundations committed to adult education and cultural patrimony – Roberto Marinho Foundation. |
| In-situ® Simulation: An Evaluative Mechanism for Health Care Systems |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amy Gullickson, Western Michigan University, amy.m.gullickson@wmich.edu |
| Abstract: More than 90,000 accidental deaths or injuries are suffered by patients of the US healthcare system every year. One of the more difficult issues underlying crisis is the fear of litigation, which impedes open discussion of problem areas within health care systems. The TriCorridor Center of Excellence for Simulation Research at Western Michigan University is attempting to address this issue through performing team skill oriented simulations In-situ® (on site in the wards of hospitals). Those simulation experiences have become an evaluative mechanism that clearly identifies problems within the hospital system. Additionally, the post-simulation debrief brings health care professionals together from a variety of departments to discuss and implement plans to address those issues. This paper describes the In-situ® process and offers case study examples of how that process has resulted in systemic changes within hospital systems in Michigan and Minnesota. |
| Evaluating the Virginia Community College System Professional Development Program: Lessons in Conducting a Formal Assessment of a Large-scale, Centralized Professional Development Initiative |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kelle Basta, Caliber an ICF International Company, kbasta@icfcaliber.com |
| Thomas J Horwood, Caliber an ICF International Company, thorwood@icfcaliber.com |
| Mary Ann Hanson, Caliber an ICF International Company, mhanson@icfcaliber.com |
| Nan Ottenritter, Virginia Community College System, nottenritter@vccs.edu |
| Abstract: This presentation will showcase the methodology and findings related to the evaluation of the Virginia Community College System (VCCS) Professional Development (PD) program. The evaluation was conducted to provide the VCCS with information on participation, needs, satisfaction, preferred format for delivery, and outcomes related to the professional development of faculty and staff. A mixed-methods approach was used, both qualitative and quantitative data were collected, and each project phase was designed to build on the previous steps. Evaluators conducted interviews and focus groups, designed a logic model, and administered a survey to VCCS employees. The evaluators integrated the results to develop recommendations for the system-wide PD program and for future program assessment activities. Overall, the conclusions of this assessment demonstrate that the VCCS PD program is highly effective in meeting many of the constituents' professional development needs. |
| Kinder, Gentler Pre-tests for Math-Science Partnership Teachers |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Linda Heath, Loyola University, Chicago, lheath@luc.edu |
| Aisha Leverett, Loyola University, Chicago, jlevere@luc.edu |
| David Slavsky, Loyola University, Chicago, dslavsk@luc.edu |
| Abstract: A recurring theme at the regional meetings of evaluators for the NSF-funded Math Science Partnerships (MSP's) was the difficulty of obtaining valid pre-test measures of teachers' knowledge and skills. The problem is not that valid, standardized measures do not exist. The problem is that completing the pre-test assessment and recognizing they have done poorly frustrates, embarrasses, and demoralizes program participants. Evaluators at the regional MSP meetings reported needing to spend a whole day (often out of only five allotted to the program) to build cohesiveness and comfort in the group prior to the test or to repair damage to morale after the test. In this paper, we examine correlates and subscales within one standardized test (The Force Concept Inventory) to see if a less-threatening but equally valid and reliable pre-test instrument is possible, using data from the Loyola University Chicago/ Chicago Public Schools (LUC/CPS) Physics Endorsement Project. |
| Learning Through Focus Groups: Evaluation of the Treatment Planning Process by Clinicians Within a Community Health Center |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Erica Gosselin, Mental Health Center of Denver, erica.gosselin@mhcd.org |
| Steve Baumer, Mental Health Center of Denver, steve.baumer@mhcd.org |
| Kristi Helvig, Mental Health Center of Denver, kristi.helvig@mhcd.org |
| Antonio Olmos, Mental Health Center of Denver, antonio.olmos@mhcd.org |
| Abstract: At the 2006 AEA conference, the Mental Health Center of Denver (MHCD) presented a paper regarding mental health consumers' perceived participation in their treatment plans. Results from these focus groups yielded specific areas for follow-up regarding the perceived utility of the Individual Service Plan (ISP) and consumers' perception of involvement in the process. Following this study, we wanted to elicit additional information in order to create a comprehensive plan for change with the aim of improving quality. Based upon an identified area for follow-up, we conducted 12 focus groups with clinicians who work with the consumers at MHCD. These focus groups were conducted to explore 1) the clinician's perception of consumer's participation in treatment, 2) the clinician's perception of the ISP and ISP process, and 3) changes clinicians would like to see in this process. Results from the clinician focus groups were correlated with the prior data from consumer focus groups to formulate targeted changes in the ISP process. |
| Surviving Establishment of an Evaluation Center: Lessons Learned |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Melinda Hess, University of South Florida, mhess@tempest.coedu.usf.edu |
| Jeffrey Kromrey, University of South Florida, kromrey@tempest.coedu.usf.edu |
| Abstract: Similar to other fields, the call for accountability in education has been increasing over recent years. As such, the need for rigorous and skilled evaluation has also increased. In order to provide evaluation services in a centralized and consistent manner, an evaluation and research center in the College of Education at a large, urban university was initiated four years ago. The center is staffed primarily with doctoral students specializing in Educational Research, Measurement, and Evaluation. There is a full-time director for the center and department faculty are actively involved in various projects. The center serves the university (e.g., evaluation of state and federal grants), the local school districts (e.g., contracting to evaluate programs), and the local community (e.g., evaluation of education or training programs). This paper provides an overview of how the center came to be as well as lessons learned. |
| Identifying Challenges to Evaluating Abstinence Education Programs |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sharon McGroder, The Lewin Group, sharon.mcgroder@lewin.com |
| Jane Koppelman, The Lewin Group, jane.koppelman@lewin.com |
| Mark Thomas, The Lewin Group, mark.thomas@lewin.com |
| Shanise Hill, The Lewin Group, shanise.hill@lewin.com |
| Abstract: The federally funded Center for Research and Evaluation on Abstinence Education was launched in 2006 to build capacity for conducting rigorous evaluations on abstinence education (AE) programs. The Center's first task was to conduct an assessment of the evaluation needs of the field—the first formal assessment of its kind. Evaluation challenges were identified through the following mechanisms: (1) discussions with practitioners, policymakers and evaluators in the field; (2) discussions with national evaluation experts and contractors assisting federal grantees in conducting evaluations; and (3) analysis of an array of published and unpublished AE evaluation studies. Evaluation challenges were documented from the above sources and listed by informant into four categories: (1) design (process and impact) challenges, (2) measurement issues, (3) analyzing, interpreting and reporting findings, and (4) implementation challenges. This needs assessment will help pinpoint the types of assistance evaluators require to conduct quality research on abstinence education. |
| Evaluating the Virginia Community College System Professional Development Program: Lessons in Conducting a Formal Assessment of a Large-scale, Centralized Professional Development Initiative |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kelle Basta, Caliber an ICF International Company, kbasta@icfcaliber.com |
| Thomas J Horwood, Caliber an ICF International Company, thorwood@icfcaliber.com |
| Mary Ann Hanson, Caliber an ICF International Company, mhanson@icfcaliber.com |
| Nan Ottenritter, Virginia Community College System, nottenritter@vccs.edu |
| Abstract: This presentation will showcase the methodology and findings related to the evaluation of the Virginia Community College System (VCCS) Professional Development (PD) program. The evaluation was conducted to provide the VCCS with information on participation, needs, satisfaction, preferred format for delivery, and outcomes related to the professional development of faculty and staff. A mixed-methods approach was used, both qualitative and quantitative data were collected, and each project phase was designed to build on the previous steps. Evaluators conducted interviews and focus groups, designed a logic model, and administered a survey to VCCS employees. The evaluators integrated the results to develop recommendations for the system-wide PD program and for future program assessment activities. Overall, the conclusions of this assessment demonstrate that the VCCS PD program is highly effective in meeting many of the constituents' professional development needs. |
| Out of the Cocoon and Into the Real World: Challenges Faced and Lessons Learned by Students in Their First Endeavor as Professional Evaluators |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rabia Hos, University of Rochester, rabiahos@yahoo.com |
| Kankana Mukhopadhyay, University of Rochester, kankana.m@gmail.com |
| Abstract: This poster is an attempt to systematically document the challenges faced and lessons learned by student evaluators, when they break out of the cocoon of the classroom environment with constant guidance and constructive feedback from instructor and peers, and wear the first hats of professional evaluators. The evaluation project undertaken was to render a summary judgment, based on quantitative data analysis, on certain critical aspects of an awards and scholarships program of a university. The university program's purpose was to recognize high school talents in various disciplines and offer them further opportunities for their future success. The summative evaluation activities primarily aimed to determine whether the specific goals and objectives of the program were being met. In addition to the planning and implementation of the evaluation, the poster will also highlight the ethical dilemmas faced in negotiating with the demands of the stakeholders and the perspectives of the program participants. |
| Naming and Framing: Are Logic Models Used to its Full Potential by Program Stakeholders and Evaluators Alike? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chidinma Ibe, Johns Hopkins University, cibe@jhsph.edu |
| Helen Hsu, University of British Columbia, helenhsu@interchange.ubc.ca |
| Abstract: Logic models are widely used in evaluation practice. Their value lies in their utility as a mechanism for guiding various aspects of the program lifecycle, such as the design, implementation and evaluation of a program. However, logic models are very often framed for evaluation purposes and sometimes can be used exclusively by the evaluation team. This practice may come to be conceptualized amongst program staff/stakeholders as a tool being exclusively related to evaluation, thereby limiting the use of the logic model to its full potential by various program stakeholders, such as guiding program implementation or planning by the program staff. This paper will draw from the literature to examine uses of logic model for different stages of a program lifecycle from the standpoint of the program staff and the evaluator(s). |
| Learning About Use and Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jeanne Hubelbank, Independent Consultant, jhubel@evalconsult.com |
| Lauren Chapman, Boston College, chapmala@bc.edu |
| Maureen Kavanaugh, Boston College, maureen.kavanaugh@marist.edu |
| Abstract: This paper will share the experiences of teaching and learning about evaluation in a graduate-level course on the practical aspects of evaluation. Specifically, the paper will concentrate on evaluation use and how understanding of it was translated into practical evaluation activities. An assignment to interview someone who had been involved in an evaluation about the use of its results was used also to teach the evaluation process and to provide experience in gathering and presenting results. Students summarized their results, created a poster, and analyzed the summaries to practice content analysis. Student input was sought throughout the course. This paper will discuss the instructor's approaches and students' reactions and learning from the assignment. The materials, successes and the areas in need of improvement will be shared. |
| Immersion in School-based Evaluation Communities |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Douglas Huffman, University of Kansas, huffman@ku.edu |
| Kelli Thomas, University of Kansas, kthomas@ku.edu |
| Karen Lombardi, University of Kansas, lombaka6@yahoo.com |
| Carrie Hohl, University of Kansas, cmhohl@salvajor.com |
| Abstract: This poster session will focus on the results of the Collaborative Evaluation Communities in Urban Schools project (CEC Project) at the University of Kansas. The project was designed to enhance the evaluation capacity of K-8 schools through collaborative evaluation communities comprised of teachers, instructional coaches, graduate students, and university faculty. The goals of the project were to improve the evaluation capacity of urban schools, develop graduate level educational leaders with the knowledge and skills to evaluate science and mathematics education programs, and develop the evaluation capacity of K-8 teachers. The poster will focus on the impact of using evaluation to develop elementary mathematics and science instruction in schools. The poster will also address the broader question: In what ways can collaborative evaluation create learning communities of practice? |
| Classroom-level Influences on Literacy Growth Among Early Reading First Students |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sheridan Green, JVA Consulting LLC, sheridan@jvaconsulting.com |
| Susan Hutchinson, University of Northern Colorado, susan.hutchinson@unco.edu |
| Ana Leon, JVA Consulting LLC, ana@jvaconsulting.com |
| Jane Walsh, Denver Public Schools Early Reading First, jane_walsh@dpsk12.org |
| Abstract: The Denver Public Schools Early Reading First program utilizes a literacy-focused curriculum and a substantive professional development component for program teachers. Approximately 180 students are assessed three times annually to gauge improvements in early literacy. This paper will present the results of examining the effects of classroom characteristics on student literacy. Because these effects are nested within schools, hierarchical linear modeling will be used for analysis. Questions addressed: 1) what is the influence of classroom-level factors on literacy growth? And 2) do classroom-level factors differ in their effects on literacy scores? Study data comprise four categories: measures of early literacy skills, measures of classroom teacher qualifications and characteristics, measures of the classroom literacy environment, and measures of student, teacher, and school characteristics that we will employ as analytical controls. Dependent variables include receptive vocabulary and phonological awareness using the Peabody Picture Vocabulary Test-III and the Phonological Awareness and Literacy Screening. |
| Naming and Framing: Are Logic Models Used to its Full Potential by Program Stakeholders and Evaluators Alike? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chidinma Ibe, Johns Hopkins University, cibe@jhsph.edu |
| Helen Hsu, University of British Columbia, helenhsu@interchange.ubc.ca |
| Abstract: Logic models are widely used in evaluation practice. Their value lies in their utility as a mechanism for guiding various aspects of the program lifecycle, such as the design, implementation and evaluation of a program. However, logic models are very often framed for evaluation purposes and sometimes can be used exclusively by the evaluation team. This practice may come to be conceptualized amongst program staff/stakeholders as a tool being exclusively related to evaluation, thereby limiting the use of the logic model to its full potential by various program stakeholders, such as guiding program implementation or planning by the program staff. This paper will draw from the literature to examine uses of logic model for different stages of a program lifecycle from the standpoint of the program staff and the evaluator(s). |
| Progress on Establishing Project-evaluation System in a Government-led Fund for Environmental Protection in Japan |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kenichi Ishida, Ocean Research Institute, ishiken@cd6.so-net.ne.jp |
| Abstract: Japan Fund for Global Environmental (JGEF), established with an initial endowment from Japanese government, together with contributions from the private sector, has been operating its grants program for environmental NGOs since 1993. Grants program, which extends financial support for environmental conservation projects conducted by NGOs accounts for 200 in each year. With the change of evaluation policy in Japanese government several years ago, JGEF started to establish an evaluation system for the grant projects in order to obtain information on the improvement of the projects; to obtain lessons-learned for the both operational and policy level of JGEF. At the poster, both the achievement and future issues will be presented. |
| Using Exact and Propensity Score Matching in the Integration and Matching of Evaluation Data |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Mukaria Itang'ata, Western Michigan University, mukaria.itangata@wmich.edu |
| Abstract: Often program, project or policy evaluators using non-experimental evaluation designs are faced with situations where they have a large population in which there is a much smaller subpopulation of foci. In such situations determining what is the best strategy for defining or selecting a comparison sample that would match and integrate the evaluation data becomes critically important. Occasionally either exact or propensity score matching methods have been used to match and integrate data. These matching methods are increasingly becoming important tools for data collection and analysis in evaluation. This presentation will compare and contrast the use of exact and propensity score matching approaches/methods to determine which between the two approaches/methods is the best strategy to define or select an evaluation comparison sample. |
| Inclusion of Children With Disabilities in School Age Care Settings: Provider's Perspective |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Hamida Jinnah, University of Georgia, hamidajinnah@gmail.com |
| Zolinda Stoneman, University of Georgia, zo@ihdd.uga.edu |
| Christine Todd, University of Georgia, ctodd@fcs.uga.edu |
| Anika Francis, University of Georgia, anikakai@uga.edu |
| Abstract: This study assesses the needs of child care providers related to inclusion of children with disabilities in school age care settings. Focus groups and phone interviews were conducted with providers. Data was analyzed using Nudist (N6). Providers mentioned that they enjoy seeing children progress in their abilities and are constantly rewarded by the love and affection children with disabilities give to them. Providers however, also experience several challenges in including children with disabilities in their setting(s). These include inadequate staff-child ratio, staff attitudes, staff training and access to information and resources for inclusion. Additionally, providers face problems due to gaps in communication with parents, school teachers and therapists. Providers would like to have more training, more resources including additional staff, more funding, and more information about available resources. They would like to see greater collaboration and networking among different individuals working with school age children having disabilities. |
| Global Diffusion of Evaluation Theories and Practices. Driving Forces and Implications |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rahel Kahlert, University of Texas, Austin, kahlert@mail.utexas.edu |
| Abstract: In the last half century, evaluation has developed from a sporadic exercise in a few Western governments into an indispensable public policy tool in modern societies across the globe. The paper examines the global diffusion processes of evaluation as a public policy tool and a professional practice. Using interpretive and social network analysis, the paper traces the driving forces behind these international diffusion processes, in the context of political accountability and transparency. The paper identifies some of the main actors' theoretical and methodological frameworks and addresses intended and unintended implications of these diffusion processes, for example a predilection for certain evidence-based evaluation designs to increase legitimacy of government action. |
| Evaluation and Learning: Project ECHO, a Training and Capacity Building Model for Hepatitis C |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Denise Dion, University of New Mexico, dmdion@salud.unm.edu |
| Jan Mines, University of New Mexico, jmines@salud.unm.edu |
| Karla Thornton, University of New Mexico, kthornton@salud.unm.edu |
| Sanjeev Arora, University of New Mexico, sarora@salud.unm.edu |
| Abstract: Using telemedicine infrastructure for clinical learning, Project ECHO (Extension for Community Healthcare Outcomes) trains primary care providers who serve rural, underserved and prison populations in providing Hepatitis C (HCV) treatment for their patients with the guidance of specialists. This model is relevant to populations with chronic, complex diseases or with limited access to specialists (Arora et al, 2007). Using cased-based and iterative learning, primary care providers present HCV patients during weekly telemedicine clinics. Specialist physicians co-manage patients with primary care providers utilizing evidence-based treatment protocols. Learning loops are created through iterative and collaborative case-based discussion during each clinic. The evaluation views the provider as learning a complex set of behaviors with extensive support in a community of similar practitioners. The evaluation draws from self efficacy, situated learning and social cognitive theory. Providers endorse their knowledge growth and self-efficacy about Hepatitis C screening, symptom management, and treatment. |
| Learning About Use and Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jeanne Hubelbank, Independent Consultant, jhubel@evalconsult.com |
| Lauren Chapman, Boston College, chapmala@bc.edu |
| Maureen Kavanaugh, Boston College, maureen.kavanaugh@marist.edu |
| Abstract: This paper will share the experiences of teaching and learning about evaluation in a graduate-level course on the practical aspects of evaluation. Specifically, the paper will concentrate on evaluation use and how understanding of it was translated into practical evaluation activities. An assignment to interview someone who had been involved in an evaluation about the use of its results was used also to teach the evaluation process and to provide experience in gathering and presenting results. Students summarized their results, created a poster, and analyzed the summaries to practice content analysis. Student input was sought throughout the course. This paper will discuss the instructor's approaches and students' reactions and learning from the assignment. The materials, successes and the areas in need of improvement will be shared. |
| Increasing the Response Rate for Satisfaction Surveys |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michaela Kennedy, Family Services of Western Pennsylvania, kennedym@fswp.org |
| Abstract: Family Services of Western Pennsylvania is committed to high quality services and customer satisfaction. Satisfied customers are more likely to remain in treatment and satisfaction is considered a vital part of service success. Family Services has implemented several interventions to reach a high response rate of customer surveys, to meet agency requirements, departmental needs, and licensing regulations. A driving force is emphasizing the importance of obtaining customer feedback for every service provided (including internal customers), getting a high and representative response rate (70%), and to use the feedback by making service improvements accordingly. Interventions include getting buy-in from all supervisors and line staff, standardizing surveys for better inter-departmental and national comparison, analyzing the results using statistical procedures and top box responses, communicating the results back to both staff and individuals served in a variety of creative ways (meetings, posters, newsletters, etc), implementing a raffle as an incentive to complete surveys, and providing small tokens/gifts and prizes for programs reaching the response rate goal and above. Results are distributed to all staff and supervisors write an improvement plan based on program specific results. Results are used for program improvement, licensing bodies, and payors. The poster for the conference will provide all relevant information regarding agency goals, interventions for increasing the response rate, tools used, analysis methods, distribution of information, and how to make satisfaction surveys more meaningful and useful for staff and individuals served. |
| Institutionalizing Utilization-focused approaches to Evaluation in Non-Governmental Organizations (NGOs): Creating a Culture of Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carrie Willimann, Academy for Educational Development, cwillimann@aed.org |
| Winnehl Tubman, Academy for Educational Development, wtubman@aed.org |
| HyeJin Kim, Academy for Educational Development, hkim@aed.org |
| Arushi Terway, Academy for Educational Development, aterway@aed.org |
| Ana Florez, Academy for Educational Development, aflorez@aed.org |
| Audrey-Marie Moore, Academy for Educational Development, amoore@aed.org |
| Abstract: Evaluation can be a tool for ensuring that education projects provide data and information at the policy level to transfer technical innovation. However, evaluation, as currently structured under many donor funded projects, does not provide the necessary information to inform this process. Nor does it allow for a cohesive approach to draw lessons learned across projects to inform broader policy. Clearing space in the existing political economic landscape so that the substance of sound educational improvement can be implemented is the operational core of what a network of institutions and actors (-reform support infrastructure-) needs to do for reform to succeed. This panel examines how a small evaluation group within an international NGO attempted to clear and fill space for utilization-focused evaluation to flourish in order to create a more cohesive approach to evaluation, build the capacity of staff to implement UFE frameworks, and improve organizational learning across education projects. |
| Using Formative Evaluation to Culturally Adapt a Parenting Program for Black Fathers of Children With Special Needs |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| James C Bridgers Jr, University of Maryland, College Park, jbridgers@mayatech.com |
| Suzanne M Randolph, University of Maryland, College Park, suzanner@umd.edu |
| Cecil H Doggette, Health Services for Children with Special Needs Inc, cdoggette@hscsn.org |
| Sally Koblinsky, University of Maryland, koblinsk@umd.edu |
| Abstract: Ecological systems theory was used in a formative evaluation to assess the feasibility of a model program, developed for Black mothers' normative parenting, for use with Black fathers of children with special needs. There is scant literature on Black fathering; and parenting programs often focus on parents' challenges rather than strengths. This evaluation examined the rewards, strengths, challenges, and support needs of African American fathers parenting children with special needs. The theoretical framework guided development of focus group guides and analyses. Data were used to culturally adapt the model parenting program and select evaluation measures for the locally implemented intervention. |
| Identifying Challenges to Evaluating Abstinence Education Programs |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sharon McGroder, The Lewin Group, sharon.mcgroder@lewin.com |
| Jane Koppelman, The Lewin Group, jane.koppelman@lewin.com |
| Mark Thomas, The Lewin Group, mark.thomas@lewin.com |
| Shanise Hill, The Lewin Group, shanise.hill@lewin.com |
| Abstract: The federally funded Center for Research and Evaluation on Abstinence Education was launched in 2006 to build capacity for conducting rigorous evaluations on abstinence education (AE) programs. The Center's first task was to conduct an assessment of the evaluation needs of the field—the first formal assessment of its kind. Evaluation challenges were identified through the following mechanisms: (1) discussions with practitioners, policymakers and evaluators in the field; (2) discussions with national evaluation experts and contractors assisting federal grantees in conducting evaluations; and (3) analysis of an array of published and unpublished AE evaluation studies. Evaluation challenges were documented from the above sources and listed by informant into four categories: (1) design (process and impact) challenges, (2) measurement issues, (3) analyzing, interpreting and reporting findings, and (4) implementation challenges. This needs assessment will help pinpoint the types of assistance evaluators require to conduct quality research on abstinence education. |
| Mentoring Adult Ex-offenders in Ready4Work: Assessing the Effects of Mentoring |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shawn Bauldry, University of North Carolina, sbauldry@email.unc.edu |
| Igor Holas, Public/Private Ventures, iholas@ppv.org |
| Danijela Korom-Djakovic, Public/Private ventures, dkoromdjakovic@ppv.org |
| Abstract: Ready4Work, a three-year national demonstration, was designed to address barriers people coming out of prison face as they transition back into their communities. In addition to typical components of a reentry program (case management and employment), Ready4Work sites also sought to provide program participants with mentors. An analysis of numerous sources of data suggests meeting with mentors may improve program retention and the employment prospects of participants. As a demonstration, however, this research lacked a comparison group. Particular attention is given to acknowledging the limitations of non-comparison group research, while making the most of the available data. |
| Mentoring Adult Ex-offenders in Ready4Work: Mentors' Perspectives |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Danijela Korom-Djakovic, Public/Private Ventures, dkoromdjakovic@ppv.org |
| Igor Holas, Public/Private Ventures, iholas@ppv.org |
| Shawn Bauldry, University of North Carolina, sbauldry@email.unc.edu |
| Abstract: Ready4Work, a three-year national demonstration, was designed to address barriers ex-offenders face as they transition back into their communities. In addition to typical components of a reentry program (case management and employment), Ready4Work sites also sought to provide program participants with mentors. This paper explores the experiences of volunteer mentors who mentored adult ex-offenders. We analyzed interviews of 31 mentors from seven cities, focusing on their motivations for mentoring ex-offenders and their perceived roles as mentors. Preliminary data analyses indicate that mentors' ex-offender status (i.e., whether a mentor was ever incarcerated or not) as well as religious orientation shape their construction of what it means and what it takes to mentor ex-offenders. Mentoring programs for adult ex-offenders are sparse and so is data about them. This study informs future reentry program evaluations and provides insights about evaluation designs that take into account mentors' perspectives and a range of their experiences. |
| Using Evaluation of Community Service School Programme to Enhance Participants' Learning From Their Experience |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Natalia Kosheleva, Ideas for Radiostations and Communities, nkochele@yandex.ru |
| Abstract: Final evaluation of the Community Service School Programme (implemented in 6 regions of Russia in 2004-2006) was designed as reflection and conceptualization stages of the experiential learning model. Mid-term evaluation revealed that teachers and students of participating schools concentrated only on performance. Final evaluation was to find if participating schools developed ownership of community service technology and to enhance participants' learning from programme experience. All evaluation activities were designed following “learn how – do it yourself” model. Evaluator visited a sample of programme locations to meet with teachers and students. Each meeting started with explaning what evaluation was, than participants evaluated attainment of programme goals and programme design. Final programme conference was turned into an evaluation exercise. It started with presentation of what evaluation was and findings and conclusions of the first stage of the evaluation, than participants evaluated the programme impact on individual, school, community and city levels. |
| Validity of Patient-reported Health-related Quality of Life Global Ratings of Change Using Structural Equation Modeling |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-generated single-item global ratings of change (GRCs) are often used as anchors of health-related quality of life (HRQoL) since they are easy for clinicians to interpret and provide clinicians with change score interpretations that may guide clinical decision-making and assist in determining intervention efficacy. Although this approach may be preferred, the validity of the anchor-based approach is currently under scrutiny. We used structural equation modeling to estimate the explained variation in domain-specific GRCs that is accounted for by Time 1 (T1) and Time 2 (T2) domain-specific summary scores from the Short-Form 36, V2 (SF-36) Health Survey in 356 asthma outpatients. Correlations revealed that domain-specific GRCs were more strongly related to T2 than T1 domain summary scores, indicating that patients were not equally relying on T1 and T2 to generate the GRCs. Furthermore, T1-domain summary scores were not of equal magnitude and opposite sign as compared to T2 scores. There is insufficient evidence to establish SF-36 domain-specific GRC validity in asthma outpatients. |
| A Comparison of Traditional and Rasch Cut Points for Assessing Clinically Important Change in Health-Related Quality of Life Among Patients with Asthma |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-perceived change in health-related quality of life (HRQoL) domains has often been classified using a 15-point patient transition rating scale with the arbitrarily-derived change levels of minimal (2, 3 or -2, -3), moderate (4, 5 or -4, -5) and large (6, 7 or -6, -7). Therefore, in order to determine the adequacy of these traditional change levels, item response theory methodology was employed to derive empirically-based cut points for the change levels. Our sample included 396 asthmatic outpatients who completed bimonthly telephone interviews on the Asthma Quality of Life Questionnaire and transition rating items over one year. After computing Rasch partial credit model transition rating cut points, we compared the mean domain-specific HRQoL changes from Time 1 to Time 2 under each traditional transition rating change level with those under Rasch-derived change levels. Although traditional and Rasch categorizations for small, moderate, and large changes slightly differed, nearly all mean changes between classification approaches were comparable. Traditional transition rating cut points remain suitable to assess HRQoL clinical significance in outpatients with asthma. |
| Surviving Establishment of an Evaluation Center: Lessons Learned |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Melinda Hess, University of South Florida, mhess@tempest.coedu.usf.edu |
| Jeffrey Kromrey, University of South Florida, kromrey@tempest.coedu.usf.edu |
| Abstract: Similar to other fields, the call for accountability in education has been increasing over recent years. As such, the need for rigorous and skilled evaluation has also increased. In order to provide evaluation services in a centralized and consistent manner, an evaluation and research center in the College of Education at a large, urban university was initiated four years ago. The center is staffed primarily with doctoral students specializing in Educational Research, Measurement, and Evaluation. There is a full-time director for the center and department faculty are actively involved in various projects. The center serves the university (e.g., evaluation of state and federal grants), the local school districts (e.g., contracting to evaluate programs), and the local community (e.g., evaluation of education or training programs). This paper provides an overview of how the center came to be as well as lessons learned. |
| An Evaluation of Techniques for Recruiting Parents into HIV Prevention Research in Rural Communities: Teen Recruitment of Adults Versus Traditional Methods: What Works Best? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Josephine Allen, Cornell University, jaa7@cornell.edu |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Jennifer Tiffany, Cornell University, jst5@cornell.edu |
| Deladem Kusi-Appouh, Cornell University, dnk7@cornell.edu |
| Wendy Vonhof, Cornell University, wmv2@cornell.edu |
| Abstract: Reducing adolescent sexual risk-taking behaviors is one of the leading national priorities under the “Healthy People 2010” initiative, and adolescents in rural areas are particularly at high risk of acquiring STIs, including HIV. Adolescents reporting positive communication with parents are less likely to engage in sexual risk-taking behaviors. Yet, engaging teenagers and their parents in HIV prevention interventions remains a challenge. We report our experience with recruiting parents or guardians into an HIV prevention research project through teenage participants who had been engaged using a Participant-Driven Recruitment approach. Our results indicate that recruitment of adults into HIV preventive interventions may work better when done using traditional techniques, rather than by asking adolescents to recruit adults. The reasons and applications to other interventions and evaluations will be discussed. |
| HIV/AIDS/STD-related Worry and Sexual Activity Within Adolescent Social Networks |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deladem Kusi-Appouh, Cornell University, dnk7@cornell.edu |
| Josephine Allen, Cornell University, jaa7@cornell.edu |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Jennifer Tiffany, Cornell University, jst5@cornell.edu |
| Lindy Williams, Cornell University, lbw@cornell.edu |
| Abstract: This paper examines the relationship of four personal worry factors to sexual activity among 15-19 year-olds. Adolescents were recruited by their peers through Participant-Driven Recruitment (PDR), a methodology that combines participatory research and Respondent-Driven Sampling (RDS). The social networks illustrated by the PDR/RDS recruitment data enabled us to evaluate the tendency for adolescents to form social ties based on similarity (homophily). The recruiter/recruitee social network was our unit of analysis. The survey instrument included measures of worry about 1) getting HIV/AIDS; 2) becoming pregnant/getting someone pregnant; 3) having sex; and 4) getting a sexually transmitted disease. Worry was measured as a continuous variable: never worried, sometimes worried, often worried and constantly worried. Results suggest that those adolescents who worried about these factors had increased odds of engaging in sexual activity during the month preceding the survey. Color graphics displaying the social networks according to each worry factor complement the analysis. |
| How Good is Our Advice? An Evaluation of an Undergraduate Advising Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jessica Ladage, Old Dominion University, jladage@odu.edu |
| Catherine Butler, Old Dominion University, cbbutler@odu.edu |
| Kelly Carpenter, Old Dominion University, kcarp005@odu.edu |
| Karen Otero-Fisher, Old Dominion University, kfish003@odu.edu |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Abstract: Advising programs at the university level allow students to gain advice on class offerings, school activities, future career plans, and more. The purpose of this evaluation is to determine the effectiveness of our university's undergraduate Psychology advising program. This evaluation is interested in students' awareness of the advising process, how well students are using the available resources, and student satisfaction towards the advising program. Quality of faculty advisors and advising sessions will also be examined. Undergraduate declared Psychology majors will complete an on-line questionnaire regarding the advising program. Both quantitative and qualitative data will be used to assess students' views on the advising program. Correlational analyses will be conducted to examine relationships, as well as analyses of variance to see if any gender or class year differences are present. |
| Evaluation of Web-based Applications: The eXtension Experience |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michael Lambur, Virginia Tech, lamburmt@vt.edu |
| Abstract: This paper presentation will focus on a review of web metrics and online surveys as they are being used in evaluating the eXtension initiative. eXtension is a website that is being developed for the national Extension System. This presentation will include an overview of these two evaluation methods as they are being implemented in eXtension, with implications for how they might be useful for evaluating other web based applications. |
| Where Do We Go From Here? Lessons on Organizational Change and Development From an Assessment of a School Resource Officer Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kathleen Crowley, Circle Solutions, kcrowley@circlesolutions.com |
| Anna Laszlo, Circle Solutions, alaszlo@circlesolutions.com |
| Deanna Breslin, Circle Solutions, dbreslin@circlesolutions.com |
| Abstract: The purpose of this paper is to present findings from an evaluation of the Montgomery County Police Department's (MCPD) Educational Facilities Officers (EFO) program. The evaluation is based on a mixed method approach that primarily focused on EFO activities at 26 Montgomery County high schools. Multiple sources of data were collected including(1) official program documentation (e.g., grant application, progress reports, etc.); (2) interviews with key stakeholders; (3) observation of EFO activities; (4) surveys with school staff; and (5) secondary school data. We will discuss policy-relevant recommendations around effective management structures and program operations and present key points from a compendium of best practice intended for other SRO programs nationally. |
| A Network Map of Stakeholder Partnerships |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anthony Lee, University of South Carolina, Columbia, abslee@gwm.sc.edu |
| Abstract: The findings I present in this paper suggest that self reported accounts of overall satisfaction with stakeholder engagement are limiting in several aspects. First, they do not provide the structure of how stakeholders are engaged. Second, these subjective accounts do not provide the opportunity to extend our understanding of how to further grow stakeholder engagement. Third, subjective accounts can hide social desirability bias common in survey responses. I present a network map of stakeholders to provide a two dimensional orientation of stakeholder engagement by portraying the relationship between frequency of communication and role familiarity with stakeholder partners. These findings are particularly relevant to the field of public health practice, where the gold standard is partner collaboration. This paper contributes to the literature on stage models of collaboration by pinpointing with a network map, what stage of collaboration, stakeholders are engaged in, notwithstanding their subjective indications of overall satisfaction. |
| Let Me Tell You What I Need! Using Multiple Perspectives to Explore the Needs of Adolescents After a Concussion |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Isabelle Gagnon, University of Montreal, isabelle.gagnon6@sympatico.ca |
| Swaine Bonnie, University of Montreal, bonnie.swaine@umontreal |
| François Champagne, University of Montreal, francois.champagne@umontreal.ca |
| Helene Lefebvre, University of Montreal, |
| Debbie Feldman, University of Montreal, |
| Jeff Atkinson, Montreal Children's Hospital, |
| Abstract: Practice guidelines have been published concerning the treatment of individuals after a concussion, but most target adults or children, with no particular emphasis on adolescents. Furthermore, most intervention programs put in place have been developed through expert consensus without the specific input of adolescents themselves. As an initial step in improving the care provided to adolescents after a concussion, their specific needs were assessed using different perspectives including those of the literature, of service providers and, especially, that of adolescents as users of the existing services. Data collection strategies included a systematic review of the literature, interviews with adolescents and with parents, focus groups of expert service providers and a postal survey. Our results demonstrate that, even after an event considered by many as trivial, adolescents have service needs that are unique and particularly important during adolescence. Clear benefits can be obtained by including adolescents themselves in evaluations addressing issues related to them and preconceived ideas about their needs should be discarded in favor of a more collaborative approach. |
| A Short Primer on Structural Equation Modeling: The Essential Concepts |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristi Lekies, The Ohio State University, lekies.1@osu.edu |
| Abstract: Structural equation modeling is an analytical procedure that can determine the degree to which a hypothesized model fits sample data. Its advantages include the ability to allow for the inclusion of latent and observed variables, multiple dependent variables, and measurement error, as well as opportunities to compare and identify best fitting models. This presentation will provide an overview of structural equation modeling for those who are new to this type of procedure, or for those who have had limited experience working with it. An explanation of structural equation modeling, its uses and benefits, and terminology will be discussed, along with the overall process of model specification, testing, and modification. Software programs and examples of helpful resources also will be covered. |
| Classroom-level Influences on Literacy Growth Among Early Reading First Students |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sheridan Green, JVA Consulting LLC, sheridan@jvaconsulting.com |
| Susan Hutchinson, University of Northern Colorado, susan.hutchinson@unco.edu |
| Ana Leon, JVA Consulting LLC, ana@jvaconsulting.com |
| Jane Walsh, Denver Public Schools Early Reading First, jane_walsh@dpsk12.org |
| Abstract: The Denver Public Schools Early Reading First program utilizes a literacy-focused curriculum and a substantive professional development component for program teachers. Approximately 180 students are assessed three times annually to gauge improvements in early literacy. This paper will present the results of examining the effects of classroom characteristics on student literacy. Because these effects are nested within schools, hierarchical linear modeling will be used for analysis. Questions addressed: 1) what is the influence of classroom-level factors on literacy growth? And 2) do classroom-level factors differ in their effects on literacy scores? Study data comprise four categories: measures of early literacy skills, measures of classroom teacher qualifications and characteristics, measures of the classroom literacy environment, and measures of student, teacher, and school characteristics that we will employ as analytical controls. Dependent variables include receptive vocabulary and phonological awareness using the Peabody Picture Vocabulary Test-III and the Phonological Awareness and Literacy Screening. |
| Three Decades of Program Evaluation in Brazil: Lessons Learned and the Case of Roberto Marinho Foundation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Thereza Penna Firme, Cesgranrio Foundation, therezapf@uol.com.br |
| Vilma Guimarães, Roberto Marinho Foundation, vilma@frm.org.br |
| Angela Dannemann, D&S Engenheiras Associadas Ltda, angcdann@terra.com.br |
| Ana Carolina Letichevsky, Cesgranrio Foundation, anacarolina@cesgranrio.org.br |
| Abstract: This paper sheds light on lessons learned over thirty years of evaluation in Brazil. A crucial aspect of this journey was learning the do's and the don'ts of evaluation. Some findings: respect stakeholders values, take community knowledge into account, report outcomes with appropriate language including potential and limitations, maintain an attitude of social responsibility. Also, never impose evaluations, hurt values in the name of accuracy or breach agreements without consent. The permanent concern with program evaluation standards while meeting the Brazilian challenges of institutional diversity in the face of social vulnerability has demanded creativity and courage from all evaluators. This apparent dilemma requires the courage to blend and use multiple approaches tuned to the changing reality in order to promote inclusion, dialogue, and appreciation. Hence, this paper focuses on the evaluation case of one of the most renowned Brazilian foundations committed to adult education and cultural patrimony – Roberto Marinho Foundation. |
| Kinder, Gentler Pre-tests for Math-Science Partnership Teachers |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Linda Heath, Loyola University, Chicago, lheath@luc.edu |
| Aisha Leverett, Loyola University, Chicago, jlevere@luc.edu |
| David Slavsky, Loyola University, Chicago, dslavsk@luc.edu |
| Abstract: A recurring theme at the regional meetings of evaluators for the NSF-funded Math Science Partnerships (MSP's) was the difficulty of obtaining valid pre-test measures of teachers' knowledge and skills. The problem is not that valid, standardized measures do not exist. The problem is that completing the pre-test assessment and recognizing they have done poorly frustrates, embarrasses, and demoralizes program participants. Evaluators at the regional MSP meetings reported needing to spend a whole day (often out of only five allotted to the program) to build cohesiveness and comfort in the group prior to the test or to repair damage to morale after the test. In this paper, we examine correlates and subscales within one standardized test (The Force Concept Inventory) to see if a less-threatening but equally valid and reliable pre-test instrument is possible, using data from the Loyola University Chicago/ Chicago Public Schools (LUC/CPS) Physics Endorsement Project. |
| Evaluating an Organizational Capacity Building Program Designed to Increase Diversity Among Health Services Researchers |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Andrea S Burling, American Institutes for Research, aburling@air.org |
| Roger E Levine, American Institutes for Research, rlevine@air.org |
| Darlene Russ-Eft, Oregon State University, darlene.russeft@oregonstate.edu |
| Jennnifer Stephens, American Institutes for Research, jstephens@air.org |
| Abstract: This paper will describe a large-scale organizational capacity building evaluation, discuss challenges faced and strategies for overcoming them, describe a unique methodology being used, and may present initial findings. The American Institutes for Research is evaluating two programs funded by the Agency for Healthcare Research and Quality (AHRQ) that are designed to improve the organizational capacities to conduct health services research of academic institutions that predominantly serve racial/ethnic minorities, and institutions located in geographic areas that have had historically low levels of AHRQ funding. The evaluation is determining how, and to what extent, grantees have strengthened their institutional infrastructures and enhanced the capabilities of individual faculty in relation to their ability to undertake health services research. Multiple methods are being used including a unique document data abstraction approach, as well as telephone interviews and site visits. Key challenges include defining and measuring success for organizational capacity building programs and efficiently accessing organizational data. |
| Evaluator Learning Curve: Is it a Gamble for Stakeholders? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Caroline Lippy, Georgia State University, clippy1@student.gsu.edu |
| Leanne Valentine, Georiga State University, leannevalentine@comcast.net |
| Jim Emshoff, Georgia State University, psyjge@langate.gsu.edu |
| Beth Anthony, Georgia State University, eanthony1@student.gsu.edu |
| Dary Enkhtor, Georgia State University, denkhtor1@student.gsu.edu |
| Ayana Perkins, Georgia State University, ayanaperkins@msn.com |
| Lindsey Zimmerman, Georgia State University, lindsetzimmerman@gmail.com |
| Abstract: This poster examines a series of research projects funded by the Georgia Department of Human Resources, focusing on a comparison of two prevalence studies of problematic gambling behavior. Serving as needs assessments, the studies, conducted in 1999 and 2007, each completed approximately 1600 randomized telephone interviews of Georgia residents and helped to outline the need for problem gambling services in the state. This poster examines several modifications between the first and second study, indicating an evaluator learning curve. Specifically, evaluators utilized an alternative measure of pathological gambling and also became more aware of race/ethnicity interviewer effects on the sample. While such modifications improved the quality of the evaluation, they potentially induced unintended negative consequences for the state, including a decreased understanding of the effectiveness of previous efforts implementing evaluator recommendations from the first study. The implications and recommendations for future research are made for both evaluators and state officials. |
| Evaluating a Grass-Roots Non-traditional Educational Program: Challenges for Non-academic Mid-career New Evaluators |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sean Little, Independent Consultant, lughsf@aol.com |
| Abstract: Designed as a formative, sequential (Quan-Qual) mixed method evaluation of a grass-roots non-traditional educational conference, this study encountered numerous challenges: subject attrition , role confusion, invisible chains of authority, lack of stakeholder buy-in, lack of mentoring, and the dynamic, but chaotic nature of many grass-roots organizations. Creative responses to those challenges and a proposed models and designs for new evaluator participation in evaluation of grass –roots projects are discussed. As the new evaluators in these situations may be without a formal mentor, the ethical issue of competency is discussed. The advantages and disadvantages of a sequential Mixed Method Quan-Qual are presented as a way to obtain breath and depth of information. Discussed are potential models for new evaluator participation in grass-root evaluation. |
| Immersion in School-based Evaluation Communities |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Douglas Huffman, University of Kansas, huffman@ku.edu |
| Kelli Thomas, University of Kansas, kthomas@ku.edu |
| Karen Lombardi, University of Kansas, lombaka6@yahoo.com |
| Carrie Hohl, University of Kansas, cmhohl@salvajor.com |
| Abstract: This poster session will focus on the results of the Collaborative Evaluation Communities in Urban Schools project (CEC Project) at the University of Kansas. The project was designed to enhance the evaluation capacity of K-8 schools through collaborative evaluation communities comprised of teachers, instructional coaches, graduate students, and university faculty. The goals of the project were to improve the evaluation capacity of urban schools, develop graduate level educational leaders with the knowledge and skills to evaluate science and mathematics education programs, and develop the evaluation capacity of K-8 teachers. The poster will focus on the impact of using evaluation to develop elementary mathematics and science instruction in schools. The poster will also address the broader question: In what ways can collaborative evaluation create learning communities of practice? |
| The Mississippi Delta Children's Partnership: A Child-focused Community Change Initiative |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bettye Fletcher, Professional Associates Inc, bwfletcher@paionline.org |
| LaTonya Lott, Professional Associates Inc, llott@paionline.org |
| Aisha Fletcher, Professional Associates Inc, afletcher@paionline.org |
| Abstract: Mississippi Delta Children's Partnership (MDCP) is a collaborative community response the grave need for after school programming for children in this impoverished Mississippi Delta region. The mechanism for the delivery of the child-focused component of the initiative is after school and summer enrichment programs called Children's Village. The Children's Village is a community-based site which is an inspirational and supportive learning environment for children to realize and develop their unique abilities. Each of the five sites has independently designed child- focused project that addresses the four determinants derived during the conceptual process (language development, cognitive and social development and self and ethnic group efficacy). It is a highly structured comprehensive enrichment program which includes critical thinking, cultural, and academic enrichment. This poster presents the evaluative findings from the first year of the Mississippi Delta Children's Partnership. |
| Evaluation of Research Program Structures Using Logic Models and Value Stream Management: A Pilot Study |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Delia J Valles-Rosales, New Mexico State University, dvalles@nmsu.edu |
| Elizabeth Schott, New Mexico State University, libby.schott@gmail.com |
| Barbara Lugo, New Mexico State University, blugo@nmsu.edu |
| Abstract: This paper presents an innovative approach to evaluate research program structures at college level. The Alliance for Minority Participation Undergraduate Research Assistantship funded by the National Science Foundation at New Mexico State University has been selected to be the pilot program for this study. The purpose of the study was to evaluate the structure of the program to locate areas for improvement that have not been identified yet by the current evaluation practices in the direction of supporting the development of students that increases the quality of educational experience and motivates them towards graduate studies. The methodology consists of employing logic models combined with value stream management techniques. Our paper discusses the overall process used to conduct the evaluation as well as our initial results and recommendations. |
| Benefits and Challenges Using Participatory Evaluation for a National Multi-site Participatory-Action-Research, Business Education, and Policy Development Initiative |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Becky A Melzer, Independent Consultant, becky@evaluationedge.com |
| Gwen Martin, Center for Women's Business Research, gmartin@womensbusinessresearch.org |
| Abstract: This poster demonstrates the use of participatory evaluation (PE) for the Center for Women's Business Research's national participatory-action-research (PAR), business education, and policy building initiative-Accelerating the Growth of Businesses Owned by Women of Color (WOC). The initiative was developed to address the fact that the number of businesses owned by WOC is growing while significant disparities in firm revenues by gender, race, and ethnicity remain. The Center gathers meaningful data at regional forums from WOC who are tackling challenges related to elements required for business growth including human, social, and financial capital, as well as environmental support. The PAR approach informs the field of business research while encouraging participants to develop growth plans that will work for them. This poster illustrates the project and evaluation design, benefits and challenges of PE, and lessons learned from the pilot forum evaluation that are being used to immediately improve future forums. |
| Promoting Science, Technology, Engineering, Mathematics, and Geography (STEM-G) Through Professional Development: Learning From Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Karen Chen, Wheeling Jesuit University, kchen@cet.edu |
| Laurie Ruberg, Wheeling Jesuit University, lruberg@cet.edu |
| Judy Martin, Wheeling Jesuit University, jmartin@cet.edu |
| Abstract: This poster presents an evaluation project that was conducted to assess the effectiveness of a nationwide science, technology, engineering, mathematics, and geography (STEM-G) educational program established in 2003. The project highlights the need for sound evaluation to document the quality and impact of the program. An interpretive, multiple case study approach was employed to review the program's impact on students. Multiple data sources were collected and analyzed, including teachers' observations of students, e-Portfolios, and student interest surveys. The evaluation team analyzed the data for emerging themes from a set of cluster-based, randomly selected case study schools. The poster presentation will discuss evaluation themes and challenges encountered. |
| The Questions Parking Lot Project: Data Source for Training Curriculum and Service Enhancements |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Norma Martinez-Rubin, Evaluation Focused Consulting, norma@evaluationfocused.com |
| Abstract: The “Questions Parking Lot” is a training technique used to defer training participants' questions for follow-up. Such deferral permits a trainer and training participants to minimize a response to questions that may be tangential to the content being presented. Questions generated from question “parking lots,” that are ordinarily captured on easel chart paper or similar note taking training tools, are potentially valuable qualitative data. Subsequent data analysis can inform future training and/or curriculum development and support organizational knowledge transfer. This paper presentation highlights findings from the analysis of questions posed by women enrolled in a 10-week small-business training program. The presenter will describe a process of involving a disparate trainer group in a simple “evaluation project” that supports organizational interest in training and service delivery enhancement with minimum disruption to existing training processes. Evaluators who occasionally do training will come away with a replicable low-cost data collection method. |
| Insights on Learning From Using Qualitative Methods to Evaluate Safety Training in Mining: Gaining Deeper Understanding |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lori Matthews, Spokane Research Laboratory, lmatthews@cdc.gov |
| Theodore Teske, Spokane Research Laboratory, tteske@cdc.gov |
| Abstract: The Health Communications team at the Spokane Research Laboratory (SRL), part of the National Institute for Occupational Safety and Health (NIOSH), uses qualitative research methods to develop and evaluate effective safety training for the mining industry. This poster presentation highlights various qualitative methods used and insights gained regarding the use of such methods. General qualitative themes will be presented and further illustrated using narrative data collected from multiple evaluation studies. Some of themes include: quality, credibility, accurate content, engaging while educational, and the extended value or transferability of safety training. This presentation will focus specifically on the learning insights gained from using qualitative methods, such as storytelling and ethnography, to evaluate safety training in mining. Ultimately, the ability to gain “deeper” understanding from the use of qualitative methods in evaluation research will be highlighted and promoted in this presentation. |
| Developmental Evaluation as a Learning Tool for Novice Evaluators |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chris De Luca, Queen's University, 2cd16@qlink.queensu.ca |
| Laura McEwen, Queen's University, laura@education.concordia.ca |
| Hana Saab, Queen's University, saabh@educ.queensu.ca |
| Abstract: Organizations are frequently required to respond on short notice to emergent needs and policy decisions. This presentation examines the contribution of evaluation to the expansion and implementation of a new course in student assessment at a Faculty of Education, initiated in response to Ministry recommendations. Patton's developmental evaluation (1994) provides a framework for the evaluation conducted by a team of graduate students despite recommendations that such an approach may not be feasible for novice evaluators. The paper will discuss how the evaluation contributed to the professional learning of the novice evaluation team, to the positioning of evaluation as an ongoing iterative and interpretive process that contributed to the emergent nature of the course, and to creating a level of sustained interactivity among the stakeholders. Suggested accommodations and adjustments to the developmental evaluation model will be discussed in anticipation of its continued use as a learning tool for students in evaluation. |
| Identifying Challenges to Evaluating Abstinence Education Programs |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sharon McGroder, The Lewin Group, sharon.mcgroder@lewin.com |
| Jane Koppelman, The Lewin Group, jane.koppelman@lewin.com |
| Mark Thomas, The Lewin Group, mark.thomas@lewin.com |
| Shanise Hill, The Lewin Group, shanise.hill@lewin.com |
| Abstract: The federally funded Center for Research and Evaluation on Abstinence Education was launched in 2006 to build capacity for conducting rigorous evaluations on abstinence education (AE) programs. The Center's first task was to conduct an assessment of the evaluation needs of the field—the first formal assessment of its kind. Evaluation challenges were identified through the following mechanisms: (1) discussions with practitioners, policymakers and evaluators in the field; (2) discussions with national evaluation experts and contractors assisting federal grantees in conducting evaluations; and (3) analysis of an array of published and unpublished AE evaluation studies. Evaluation challenges were documented from the above sources and listed by informant into four categories: (1) design (process and impact) challenges, (2) measurement issues, (3) analyzing, interpreting and reporting findings, and (4) implementation challenges. This needs assessment will help pinpoint the types of assistance evaluators require to conduct quality research on abstinence education. |
| Benefits and Challenges Using Participatory Evaluation for a National Multi-site Participatory-Action-Research, Business Education, and Policy Development Initiative |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Becky A Melzer, Independent Consultant, becky@evaluationedge.com |
| Gwen Martin, Center for Women's Business Research, gmartin@womensbusinessresearch.org |
| Abstract: This poster demonstrates the use of participatory evaluation (PE) for the Center for Women's Business Research's national participatory-action-research (PAR), business education, and policy building initiative-Accelerating the Growth of Businesses Owned by Women of Color (WOC). The initiative was developed to address the fact that the number of businesses owned by WOC is growing while significant disparities in firm revenues by gender, race, and ethnicity remain. The Center gathers meaningful data at regional forums from WOC who are tackling challenges related to elements required for business growth including human, social, and financial capital, as well as environmental support. The PAR approach informs the field of business research while encouraging participants to develop growth plans that will work for them. This poster illustrates the project and evaluation design, benefits and challenges of PE, and lessons learned from the pilot forum evaluation that are being used to immediately improve future forums. |
| Validity of Patient-reported Health-related Quality of Life Global Ratings of Change Using Structural Equation Modeling |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-generated single-item global ratings of change (GRCs) are often used as anchors of health-related quality of life (HRQoL) since they are easy for clinicians to interpret and provide clinicians with change score interpretations that may guide clinical decision-making and assist in determining intervention efficacy. Although this approach may be preferred, the validity of the anchor-based approach is currently under scrutiny. We used structural equation modeling to estimate the explained variation in domain-specific GRCs that is accounted for by Time 1 (T1) and Time 2 (T2) domain-specific summary scores from the Short-Form 36, V2 (SF-36) Health Survey in 356 asthma outpatients. Correlations revealed that domain-specific GRCs were more strongly related to T2 than T1 domain summary scores, indicating that patients were not equally relying on T1 and T2 to generate the GRCs. Furthermore, T1-domain summary scores were not of equal magnitude and opposite sign as compared to T2 scores. There is insufficient evidence to establish SF-36 domain-specific GRC validity in asthma outpatients. |
| A Comparison of Traditional and Rasch Cut Points for Assessing Clinically Important Change in Health-Related Quality of Life Among Patients with Asthma |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-perceived change in health-related quality of life (HRQoL) domains has often been classified using a 15-point patient transition rating scale with the arbitrarily-derived change levels of minimal (2, 3 or -2, -3), moderate (4, 5 or -4, -5) and large (6, 7 or -6, -7). Therefore, in order to determine the adequacy of these traditional change levels, item response theory methodology was employed to derive empirically-based cut points for the change levels. Our sample included 396 asthmatic outpatients who completed bimonthly telephone interviews on the Asthma Quality of Life Questionnaire and transition rating items over one year. After computing Rasch partial credit model transition rating cut points, we compared the mean domain-specific HRQoL changes from Time 1 to Time 2 under each traditional transition rating change level with those under Rasch-derived change levels. Although traditional and Rasch categorizations for small, moderate, and large changes slightly differed, nearly all mean changes between classification approaches were comparable. Traditional transition rating cut points remain suitable to assess HRQoL clinical significance in outpatients with asthma. |
| Evaluating the Capacity and Quality of an Early Care and Education System |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rob Fischer, Case Western Reserve University, fischer@case.edu |
| Kristen Mikelbank, Case Western Reserve University, kristen.mikelbank@case.edu |
| Claudia Coulton, Case Western Reserve University, claudia.coulton@case.edu |
| Abstract: Since 1999, a public/private partnership in Cuyahoga County, Ohio, has been engaged in a comprehensive effort focused on the well-being of children under the age of six and their families. As part of its strategic planning process, the partnership identified goals around enhancing the early care and education system for the County's children. The present study provides a detailed assessment of the County's regulated early care system in regard to its capacity and quality, by combining administrative and observational data. Using geographic analyses, areas within the County exhibiting relative need for additional child care slots are identified. The study demonstrates the effective integration of a range of data sources in a geographic analysis designed to assist in County planning for the needs of young children and their families. |
| "Apples to Apples": The Experience of Developing an Evaluation Tool for the Workforce Development Performance Benchmarking Project |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacy Woodruff-Bolte, Public/Private Ventures, swoodruff-bolte@ppv.org |
| Martha Miles, Training Inc, mmiles@traininginc.org |
| Carol Clymer, Public/Private Ventures, cclymer@ppv.org |
| Abstract: The Workforce Development Performance Benchmarks Project represents an attempt to determine the feasibility of identifying standards of performance for workforce development organizations and use the benchmarks to improve performance. Following the second wave of survey data collection from workforce organizations across the country serving TANF recipients, lessons related to creating a performance measurement evaluation tool are emerging. Using methods to account for differences among employment and training organizations, we will discuss the development of a much needed evaluation tool in the field for workforce development. |
| Evaluation and Learning: Project ECHO, a Training and Capacity Building Model for Hepatitis C |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Denise Dion, University of New Mexico, dmdion@salud.unm.edu |
| Jan Mines, University of New Mexico, jmines@salud.unm.edu |
| Karla Thornton, University of New Mexico, kthornton@salud.unm.edu |
| Sanjeev Arora, University of New Mexico, sarora@salud.unm.edu |
| Abstract: Using telemedicine infrastructure for clinical learning, Project ECHO (Extension for Community Healthcare Outcomes) trains primary care providers who serve rural, underserved and prison populations in providing Hepatitis C (HCV) treatment for their patients with the guidance of specialists. This model is relevant to populations with chronic, complex diseases or with limited access to specialists (Arora et al, 2007). Using cased-based and iterative learning, primary care providers present HCV patients during weekly telemedicine clinics. Specialist physicians co-manage patients with primary care providers utilizing evidence-based treatment protocols. Learning loops are created through iterative and collaborative case-based discussion during each clinic. The evaluation views the provider as learning a complex set of behaviors with extensive support in a community of similar practitioners. The evaluation draws from self efficacy, situated learning and social cognitive theory. Providers endorse their knowledge growth and self-efficacy about Hepatitis C screening, symptom management, and treatment. |
| Challenges in Program Evaluation: Lessons Learned From Evaluating Child Maltreatment Prevention Interventions |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lei Zhang, Georgia State University, alhlzx@langate.gsu.edu |
| Angela Snyder, Georgia State University, alhabs@langate.gsu.edu |
| Karen Minyard, Georgia State University, alhkjm@langate.gsu.edu |
| Abstract: Child maltreatment program evaluation raises several ethical/technical issues that have yet to be addressed by evaluation research. A real-world example is Georgia Health Policy Center's evaluation of the Intensive Home Visiting (IHV) program funded by Georgia Division of Public Health. The IHV program features trained, supervised public health nurses that will provide frequent and intense home visits with families of young children identified as at-risk for abuse and neglect. The nurses will use standardized tools and curricula to teach parents about child development, health, and safety. The major ethical/technical issues involved in this evaluation include (1) obtaining sample sizes large enough to detect statistically significant impacts; (2) maintaining victim safety; and (3) tracking participants in a way that minimizes privacy violation. Our feedback from the evaluation indicates (1) evaluation needs to be sensitive to its contexts; and (2) ethical guidance provided by the Joint Committee needs to be more specific. |
| Evaluation of an Intensive Rehabilitation Services Project for Adolescents With Behavior Problems |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Gilles Mireault, Centre Jeunesse de Québec-Institut Universitaire, gilles_mireault@ssss.gouv.qc.ca |
| Abstract: In 2003, le Centre jeunesse de Québec-Institut universitaire, a university-based youth protection agency, launched an initiative to provide within the community intensive rehabilitation services for adolescents with behavior problems. The project's staff included three youth careworkers and one coordinator. The intervention is based on frequent contacts, reinforcement of parental skills and discipline consistency. At the end of the project, 79% of the project group youth (n=82) were maintained in their family. A 6 month follow-up of closed cases of the project revealed that they do not use more services from the agency than a comparison group. Finally, parents score between 90 and 95% and youth between 75 and 80% on a 20 items satisfaction questionnaire. Considering that these youth were heading for an almost inevitable out of home placement when they were referred to the project, these results suggest that intensive services for these youths can have beneficial effects on parent's capacities to deal with their youth. |
| Institutionalizing Utilization-focused approaches to Evaluation in Non-Governmental Organizations (NGOs): Creating a Culture of Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carrie Willimann, Academy for Educational Development, cwillimann@aed.org |
| Winnehl Tubman, Academy for Educational Development, wtubman@aed.org |
| HyeJin Kim, Academy for Educational Development, hkim@aed.org |
| Arushi Terway, Academy for Educational Development, aterway@aed.org |
| Ana Florez, Academy for Educational Development, aflorez@aed.org |
| Audrey-Marie Moore, Academy for Educational Development, amoore@aed.org |
| Abstract: Evaluation can be a tool for ensuring that education projects provide data and information at the policy level to transfer technical innovation. However, evaluation, as currently structured under many donor funded projects, does not provide the necessary information to inform this process. Nor does it allow for a cohesive approach to draw lessons learned across projects to inform broader policy. Clearing space in the existing political economic landscape so that the substance of sound educational improvement can be implemented is the operational core of what a network of institutions and actors (-reform support infrastructure-) needs to do for reform to succeed. This panel examines how a small evaluation group within an international NGO attempted to clear and fill space for utilization-focused evaluation to flourish in order to create a more cohesive approach to evaluation, build the capacity of staff to implement UFE frameworks, and improve organizational learning across education projects. |
| Assertive Community Treatment in Atlanta: Exploring Particular Outcomes of a Grady Healthcare System's Approach |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Angela Mooss, Georgia State University, amooss1@student.gsu.edu |
| John Barile, Georgia State University, jbarile1@student.gsu.edu |
| Joanna Weinberg, Georgia State University, jweinberg3@student.gsu.edu |
| Doyanne Darnell, Georgia State University, dhorst1@student.gsu.edu |
| Brandeis Green, Georgia State University, bgreen8@student.gsu.edu |
| Abstract: Extensive research indicates that the Assertive Community Treatment (ACT) model of mental health service delivery is effective in treating persons with severe mental illness. The Atlanta based ACT team (ECSEL) at the Grady Healthcare System employs a “housing first” philosophy which holds that effective service delivery cannot occur without stable housing. Supportive housing for homeless people with severe mental illness has been found to reduce overall service use, incarcerations and hospitalizations (Culhane, et. al. 2001). The ACT model is predicted to positively impact overall factors of wellbeing, such as housing satisfaction, familial relations, employment opportunities, prevalence of psychiatric symptoms, and reductions in substance abuse. This presentation examines the outcomes of the above factors among a homeless sample in Atlanta, and explores the unique challenges and successes of data collection in a variety of settings. Subsequent outcomes and issues of diversity among program participants and between the participants and members of the ACT team will also be addressed regarding fidelity to the original ACT model. |
| Evaluating Outreach Strategies to Promote HIV Testing Among Young African-American Men who Have Sex With Men: The Case for Youth Empowerment |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| George Greene, Working For Togetherness, george.wft@sbcglobal.net |
| Clifford Armstead, Working For Togetherness, cliffordarmstead@sbcglobal.net |
| Gregory Norels, Working For Togetherness, gnorels@hotmail.com |
| Sean Morgan, Working For Togetherness, smorgan.wft@sbcglobal.net |
| Abstract: Increasing HIV rates among young, African-American men who have sex with men (YAAMSM) suggest this population is not being effectively served by traditional prevention programs. This poster articulates our organizational response to the epidemic and presents an evaluation of our multi-pronged outreach approach. Incorporating urban marketing strategies into HIV prevention for YAAMSM, our agency provides both mobile community-wide and fixed-site outreach in HIV-impacted neighborhoods and high-risk settings by offering HIV/STD education, counseling, and testing. Additionally, our agency opened a youth empowerment center to equip participants with resources to manage their healthcare needs. Preliminary evaluation data indicate that mobile- and fixed-site outreach located large numbers of YAAMSM for information and condom dissemination; however, youth-centered services in a GLBTQ safe space better promoted HIV testing among youth. Taken together, this service delivery approach minimizes cultural barriers to health-seeking behaviors and delivers HIV prevention services to groups disproportionately affected by the HIV epidemic. |
| Navigating to Discovery: Personal Development of a Culturally Competent Evaluator While Coordinating a Community-wide Health & Wellness Fair |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Vernita Morgan, University of Iowa, vernita-morgan@uiowa.edu |
| Abstract: The evaluation of the Cedar Rapids Community-Wide Health & Wellness Fair will offer insight on the health status of participants who attended the health fair. Cedar Rapids is a diverse community located in an area known as the “Cultural Corridor.” Thus, an evaluator not only must consider the dimensions of quality when working on this project, but also the cultural and ethnic disposition of its stakeholders so that the product is openly accepted across the board. Insight on best practices and strategies for becoming a culturally competent evaluator while coordinating a community-wide health & wellness fair will be explored. |
| How Graduate Students Gain Evaluation Skills: What Worked for Us |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Margot Ackermann, Old Dominion University, margot.ackermann@gmail.com |
| Abstract: There are few universities in the United States that offer graduate programs either directly in evaluation or with a concentration in evaluation. Therefore, students looking for practical experience are often left on their own to create opportunities for themselves. The focus of this presentation will be a description of what we have done at our institution to train our students in program evaluation and a discussion of what has worked for us. A description of how students can find opportunities to conduct their own evaluation research at their institution, strategies for fitting evaluation into their typical graduate work load, as well as how they can obtain education on evaluation practices outside of their educational institution will be presented. |
| How Good is Our Advice? An Evaluation of an Undergraduate Advising Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jessica Ladage, Old Dominion University, jladage@odu.edu |
| Catherine Butler, Old Dominion University, cbbutler@odu.edu |
| Kelly Carpenter, Old Dominion University, kcarp005@odu.edu |
| Karen Otero-Fisher, Old Dominion University, kfish003@odu.edu |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Abstract: Advising programs at the university level allow students to gain advice on class offerings, school activities, future career plans, and more. The purpose of this evaluation is to determine the effectiveness of our university's undergraduate Psychology advising program. This evaluation is interested in students' awareness of the advising process, how well students are using the available resources, and student satisfaction towards the advising program. Quality of faculty advisors and advising sessions will also be examined. Undergraduate declared Psychology majors will complete an on-line questionnaire regarding the advising program. Both quantitative and qualitative data will be used to assess students' views on the advising program. Correlational analyses will be conducted to examine relationships, as well as analyses of variance to see if any gender or class year differences are present. |
| Having Graduate Students as Evaluators: A Description of a Community Based Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shawn Dickerson, Old Dominion University, sdick003@odu.edu |
| Abby Braitman, Old Dominion University, abraitma@odu.edu |
| Krystall Dunaway, Old Dominion University, kdunaway@odu.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Abstract: The current evaluation was a team evaluation conducted by graduate students which utilized an online survey to collect information from the stakeholders of interest. Evaluators gathered and examined information concerning different aspects of the job roles for Uniform Patrol Officers (UPOs) and Neighborhood Impact Officers (NIOs) from a local community's Police Department. The researchers assessed attitudes and behavior in regards to the officers' training and responsibilities, mainly pertaining to the Weed & Seed program. The Weed & Seed program was established to assist the city in reducing violent crime, the trafficking of illegal drugs and reducing juvenile crime rates. This outcome-focused evaluation gained vital information for community policing and increased the understanding of police officers beliefs and increase knowledge about their training. |
| How Learning From Today’s “Generations” Will Help Make Evaluation More Relevant and Useful for Future Generations |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Vanessa Moss-Summers, Xerox Corporation, vanessa.moss-summers@xerox.com |
| Abstract: Are you a “baby boomer” gathering data from a “Gen-X’er?” Are you a “Gen-X’er” presenting evaluation results to a “Millennial”? Are you finding the conversation and work practice discussion a potential “generation gap” and impediment to effective evaluation? Do you account for generational differences in your evaluation plan, data collection, and reporting? This session will describe what we know so far about “generational” differences and begin a dialogue about how learning from each other can lead to better evaluation. |
| The Use of a Standardized Exercise to Evaluate Emergency Preparedness Training Effectiveness for Health Care Workers |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Eileen Blake, Yale New Haven Health, eileen.blake@ynhh.org |
| Michael Mozzer, Yale New Haven Health, michael.mozzer@ynhh.org |
| Abstract: The Yale New Haven Center for Emergency Preparedness and Disaster Response National Alliance for Training HealthCare for Emergency Response (NATHCER) is a nationwide education and training program for the healthcare delivery workforce. The project provides healthcare emergency preparedness training to ten partner organizations throughout the nation. The effectiveness of the NATHCER project will be determined in part by a standardized process where each regional partner hosts a Tabletop Exercise facilitated by YNH-CEPDR staff. The exercise will consist of a subset of staff that has completed either Introduction to Emergency Management with NIMS or Best Practices for the Protection of Healthcare Based First Receivers. The exercise scenario and objectives will be consistent at all locations. The outcomes of this exercise will inform the ability of these courses to increase the emergency preparedness competency of the participants and test a strategy for use of exercises to validate training content. |
| Why They Do It - Or Not: Evaluation of the Implementation of a Web-based Educational Innovation for Teachers |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michael P Mueller, The Hospital for Sick Children, michael.mueller@sickkids.ca |
| Abstract: Evaluating educational programs often focuses just on outcomes. However, the success of educational innovations crucially depends on implementation. This presentation will report the results from an evaluation that addressed the questions of how and how often an educational innovation was used by teachers and the factors that affected the innovation diffusion process. Teachers across Ontario had been introduced to the Web- Based Teaching Tool, an innovative intervention for kindergarten and grade 1. The theoretical model that guided the evaluation assumes that teachers continuously evaluate an innovation using criteria such as the perceived need for change, benefits, costs, and complexity of the innovation, and compatibility with previous practices. The results confirmed the theoretical model. Teachers were willing to get involved if they in return received benefits from it. Considering the innovation as too time-consuming only reduced its usage if benefits were perceived as low. Implementation levels also affected outcomes. |
| Out of the Cocoon and Into the Real World: Challenges Faced and Lessons Learned by Students in Their First Endeavor as Professional Evaluators |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rabia Hos, University of Rochester, rabiahos@yahoo.com |
| Kankana Mukhopadhyay, University of Rochester, kankana.m@gmail.com |
| Abstract: This poster is an attempt to systematically document the challenges faced and lessons learned by student evaluators, when they break out of the cocoon of the classroom environment with constant guidance and constructive feedback from instructor and peers, and wear the first hats of professional evaluators. The evaluation project undertaken was to render a summary judgment, based on quantitative data analysis, on certain critical aspects of an awards and scholarships program of a university. The university program's purpose was to recognize high school talents in various disciplines and offer them further opportunities for their future success. The summative evaluation activities primarily aimed to determine whether the specific goals and objectives of the program were being met. In addition to the planning and implementation of the evaluation, the poster will also highlight the ethical dilemmas faced in negotiating with the demands of the stakeholders and the perspectives of the program participants. |
| Lesson Study: Capturing Collegial Professional Development for Educators |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Robert Nelson, ASPEN Associates Inc, rnelson@aspenassociates.org |
| Abstract: Lesson study provides educators ongoing opportunities to collaborate on lesson development for broader and deeper student learning. This is accomplished by focusing more on how individual students learn than on how teachers teach. With this method students are engaged in ways that are meaningful to each individual. The resulting classroom practices promote both independent student thinking and cooperation among students. Lesson study is relatively new in America, giving us an opportunity to educate people outside the classroom as well as inside; making it a good fit for this year's “Evaluation and Learning” conference theme. Our evaluation, like others, examined the impact of lesson study on teaching and learning through a comparison of test scores and teaching practices at treatment and control schools. It also included mini–case studies of teachers new to and more skilled in using lesson study. The result was a rich and rigorous look at lesson study. |
| Evaluating Outreach Strategies to Promote HIV Testing Among Young African-American Men who Have Sex With Men: The Case for Youth Empowerment |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| George Greene, Working For Togetherness, george.wft@sbcglobal.net |
| Clifford Armstead, Working For Togetherness, cliffordarmstead@sbcglobal.net |
| Gregory Norels, Working For Togetherness, gnorels@hotmail.com |
| Sean Morgan, Working For Togetherness, smorgan.wft@sbcglobal.net |
| Abstract: Increasing HIV rates among young, African-American men who have sex with men (YAAMSM) suggest this population is not being effectively served by traditional prevention programs. This poster articulates our organizational response to the epidemic and presents an evaluation of our multi-pronged outreach approach. Incorporating urban marketing strategies into HIV prevention for YAAMSM, our agency provides both mobile community-wide and fixed-site outreach in HIV-impacted neighborhoods and high-risk settings by offering HIV/STD education, counseling, and testing. Additionally, our agency opened a youth empowerment center to equip participants with resources to manage their healthcare needs. Preliminary evaluation data indicate that mobile- and fixed-site outreach located large numbers of YAAMSM for information and condom dissemination; however, youth-centered services in a GLBTQ safe space better promoted HIV testing among youth. Taken together, this service delivery approach minimizes cultural barriers to health-seeking behaviors and delivers HIV prevention services to groups disproportionately affected by the HIV epidemic. |
| Evaluation: A Tool for Teaching and Learning High School Mathematics |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anane Olatunji, George Washington University, dr_o@gwu.edu |
| Paige Allison, Alachua County Public Schools, plado@cox.net |
| Abstract: Baseline survey results from this pilot project in spring 2006 indicated that at a regular high school in Florida most students reported mathematics not only was less interesting than other subjects but also more difficult to learn. Our aim, therefore, was to create an effective method of teaching mathematics. Efficacy was based on two criteria: student engagement and achievement. The evaluators used students' weekly test scores to illustrate a unit on statistical concepts. Graphs were generated and displayed using SPSS and PowerPoint slides. Student incentives also were provided. Final survey results indicated that the “Data Analysis Project” was highly effective. Ninety-four percent of students reported that they enjoyed the project, 70 percent said that it helped them to learn mathematical concepts, and 85 percent agreed that the project helped them to understand the utility of mathematics. In addition, student comments bolstered survey results. |
| Learning Through Focus Groups: Evaluation of the Treatment Planning Process by Clinicians Within a Community Health Center |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Erica Gosselin, Mental Health Center of Denver, erica.gosselin@mhcd.org |
| Steve Baumer, Mental Health Center of Denver, steve.baumer@mhcd.org |
| Kristi Helvig, Mental Health Center of Denver, kristi.helvig@mhcd.org |
| Antonio Olmos, Mental Health Center of Denver, antonio.olmos@mhcd.org |
| Abstract: At the 2006 AEA conference, the Mental Health Center of Denver (MHCD) presented a paper regarding mental health consumers' perceived participation in their treatment plans. Results from these focus groups yielded specific areas for follow-up regarding the perceived utility of the Individual Service Plan (ISP) and consumers' perception of involvement in the process. Following this study, we wanted to elicit additional information in order to create a comprehensive plan for change with the aim of improving quality. Based upon an identified area for follow-up, we conducted 12 focus groups with clinicians who work with the consumers at MHCD. These focus groups were conducted to explore 1) the clinician's perception of consumer's participation in treatment, 2) the clinician's perception of the ISP and ISP process, and 3) changes clinicians would like to see in this process. Results from the clinician focus groups were correlated with the prior data from consumer focus groups to formulate targeted changes in the ISP process. |
| Robotics: Opportunities for Building Outstanding Talent in the Sciences (R.O.B.O.T.S.): The Collaborative Evaluation of a Middle School Science, Technology, Engineering, and Mathematics (STEM) Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tangie Fleming, University of North Carolina, Chapel Hill, tangie.fleming@gmail.com |
| Rita O'Sullivan, University of North Carolina, Chapel Hill, ritao@unc.edu |
| Abstract: The Robotics: Opportunities for Building Outstanding Talent in the Sciences (R.O.B.O.T.S.) middle school project seeks to provide a model for building a sustained community-wide awareness about the value of astronomy and physics, which are needed for success in virtually all science, technology, engineering, and mathematics (STEM) fields. An evaluation study was commissioned to investigate how middle school students form and change their attitudes about the world around them in the context of thinking about career options in STEM disciplines. The evaluation investigates the underlying reasons and relationships that influence outcome attainment. Further, it tests the model's assumptions about the experiential strategies and how they influence the desired outcomes. This session presents the methodology of the collaborative evaluation used to validate the theoretical framework upon which R.O.B.O.T.S. is based. |
| Collaborative Evaluation of Project Quest: A Comprehensive Program Preparing Middle and High School Students for Academic Success |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Monifa Beverly, University of North Carolina, Chapel Hill, mngreen@email.unc.edu |
| Rita O'Sullivan, University of North Carolina, Chapel Hill, ritao@unc.edu |
| Abstract: This session presents the results of a two-year summative evaluation of a three-year grant from the U.S. Department of Education's Advanced Placement Incentive Program. Project Quest is a three-school partnership designed to increase the successful participation of low-income students in academically rigorous, higher–level academic courses. To address this goal, Project Quest has implemented a complex and integrated strategy comprising four major programs: Advanced Placement (AP), Advancement via Individual Determination (AVID), International Baccalaureate (IB), and Paideia. A collaborative evaluation was undertaken as a means of building the evaluation capacity of program staff leading to increased utility of evaluation findings. The results of this evaluation, guided by a collaborative evaluation approach, support the notion that programs with limited and ample resources alike can secure sound, effective evaluations. Moreover, the approach's focus on acknowledging and valuing a staff's own program expertise leads to better program management and execution of future evaluation responsibilities. |
| How Good is Our Advice? An Evaluation of an Undergraduate Advising Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jessica Ladage, Old Dominion University, jladage@odu.edu |
| Catherine Butler, Old Dominion University, cbbutler@odu.edu |
| Kelly Carpenter, Old Dominion University, kcarp005@odu.edu |
| Karen Otero-Fisher, Old Dominion University, kfish003@odu.edu |
| Erin Burr, University of Tennessee, eburr@utk.edu |
| Jennifer Morrow, University of Tennessee, jamorrow@utk.edu |
| Abstract: Advising programs at the university level allow students to gain advice on class offerings, school activities, future career plans, and more. The purpose of this evaluation is to determine the effectiveness of our university's undergraduate Psychology advising program. This evaluation is interested in students' awareness of the advising process, how well students are using the available resources, and student satisfaction towards the advising program. Quality of faculty advisors and advising sessions will also be examined. Undergraduate declared Psychology majors will complete an on-line questionnaire regarding the advising program. Both quantitative and qualitative data will be used to assess students' views on the advising program. Correlational analyses will be conducted to examine relationships, as well as analyses of variance to see if any gender or class year differences are present. |
| Evaluating the Virginia Community College System Professional Development Program: Lessons in Conducting a Formal Assessment of a Large-scale, Centralized Professional Development Initiative |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kelle Basta, Caliber an ICF International Company, kbasta@icfcaliber.com |
| Thomas J Horwood, Caliber an ICF International Company, thorwood@icfcaliber.com |
| Mary Ann Hanson, Caliber an ICF International Company, mhanson@icfcaliber.com |
| Nan Ottenritter, Virginia Community College System, nottenritter@vccs.edu |
| Abstract: This presentation will showcase the methodology and findings related to the evaluation of the Virginia Community College System (VCCS) Professional Development (PD) program. The evaluation was conducted to provide the VCCS with information on participation, needs, satisfaction, preferred format for delivery, and outcomes related to the professional development of faculty and staff. A mixed-methods approach was used, both qualitative and quantitative data were collected, and each project phase was designed to build on the previous steps. Evaluators conducted interviews and focus groups, designed a logic model, and administered a survey to VCCS employees. The evaluators integrated the results to develop recommendations for the system-wide PD program and for future program assessment activities. Overall, the conclusions of this assessment demonstrate that the VCCS PD program is highly effective in meeting many of the constituents' professional development needs. |
| Three Decades of Program Evaluation in Brazil: Lessons Learned and the Case of Roberto Marinho Foundation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Thereza Penna Firme, Cesgranrio Foundation, therezapf@uol.com.br |
| Vilma Guimarães, Roberto Marinho Foundation, vilma@frm.org.br |
| Angela Dannemann, D&S Engenheiras Associadas Ltda, angcdann@terra.com.br |
| Ana Carolina Letichevsky, Cesgranrio Foundation, anacarolina@cesgranrio.org.br |
| Abstract: This paper sheds light on lessons learned over thirty years of evaluation in Brazil. A crucial aspect of this journey was learning the do's and the don'ts of evaluation. Some findings: respect stakeholders values, take community knowledge into account, report outcomes with appropriate language including potential and limitations, maintain an attitude of social responsibility. Also, never impose evaluations, hurt values in the name of accuracy or breach agreements without consent. The permanent concern with program evaluation standards while meeting the Brazilian challenges of institutional diversity in the face of social vulnerability has demanded creativity and courage from all evaluators. This apparent dilemma requires the courage to blend and use multiple approaches tuned to the changing reality in order to promote inclusion, dialogue, and appreciation. Hence, this paper focuses on the evaluation case of one of the most renowned Brazilian foundations committed to adult education and cultural patrimony – Roberto Marinho Foundation. |
| Evaluator Learning Curve: Is it a Gamble for Stakeholders? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Caroline Lippy, Georgia State University, clippy1@student.gsu.edu |
| Leanne Valentine, Georiga State University, leannevalentine@comcast.net |
| Jim Emshoff, Georgia State University, psyjge@langate.gsu.edu |
| Beth Anthony, Georgia State University, eanthony1@student.gsu.edu |
| Dary Enkhtor, Georgia State University, denkhtor1@student.gsu.edu |
| Ayana Perkins, Georgia State University, ayanaperkins@msn.com |
| Lindsey Zimmerman, Georgia State University, lindsetzimmerman@gmail.com |
| Abstract: This poster examines a series of research projects funded by the Georgia Department of Human Resources, focusing on a comparison of two prevalence studies of problematic gambling behavior. Serving as needs assessments, the studies, conducted in 1999 and 2007, each completed approximately 1600 randomized telephone interviews of Georgia residents and helped to outline the need for problem gambling services in the state. This poster examines several modifications between the first and second study, indicating an evaluator learning curve. Specifically, evaluators utilized an alternative measure of pathological gambling and also became more aware of race/ethnicity interviewer effects on the sample. While such modifications improved the quality of the evaluation, they potentially induced unintended negative consequences for the state, including a decreased understanding of the effectiveness of previous efforts implementing evaluator recommendations from the first study. The implications and recommendations for future research are made for both evaluators and state officials. |
| Case Studies in Transformative Learning: Interviews, Assessments, and Evidence Used to Evaluate “Ready Schools” |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paula Plonski, Praxis Research Inc, pmplonski@carolina.rr.com |
| Bruce Yelton, Praxis Research Inc, praxis1@att.net |
| Abstract: The concept of “ready schools” grew out of the national concern for children's school readiness. However, helping children to be ready for school is only part of the battle; schools also need to be ready to receive, nurture, and support these “ready children”. As part of the evaluation of the implementation of a W.K. Kellogg SPARK grant in North Carolina, a case study methodology was utilized to assess the readiness of two exemplary schools in the region. The evaluators conducted interviews with teachers, administrators, and parents; observed the administration of High/Scope's Ready School Assessment; and considered documents and other evidence gathered by school personnel. As a result of this process, the evaluators and the Ready School Team simultaneously uncovered the school's level of readiness. |
| Using Qualitative Information to Develop Materials to Address Clinical and Psychosocial Needs of Lesbian, Gay, Bisexual, Transgendered, Questioning, Intersexed, and Two-Spirit (LGBTQI2S) Youth in Systems of Care |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sylvia Fisher, United States Department of Health and Human Services, sylvia.fisher@samhsa.hhs.gov |
| Jeffrey Poirier, American Institutes for Research, jpoirier@air.org |
| Abstract: Lesbian, gay, bisexual, transgendered, questioning, intersexed, and two-spirit (LGBTQI2-S) youth frequently do not receive appropriate and culturally-competent mental health interventions, partly due to the unavailability of useful materials and practice guides for providers. This presentation describes interview data results collected from GLBTQ youth and systems of care program directors with the purpose of developing useful and appropriate materials to address the needs of GLBTQ youth in systems of care. GLBTQ youth were interviewed using detailed protocols emphasizing issues associated with stigmatization, mental health service experiences, useful interventions, and related issues. Implications of this research activity for future research and practice interventions with sexual minority youth are also provided. |
| A Data Quality Rating System to Improve Organizational Use of Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paul Pope, Texas Cooperative Extension, ppope@tamu.edu |
| Abstract: A data quality rating system was developed and implemented at a large non-profit organization with three interrelated goals: to educate users of evaluation on factors that contribute to data quality, to improve executive decision-making, and for program staff to incorporate sound methodological practices to improve data quality. The rating system, designed for use with survey data, initially places results in one of six broad quality levels based on response rate. Then other factors such as steps to address non-response, accuracy (margin of error), sources of non-sampling error, etc. are considered and the rating adjusted accordingly. A final quality rating is assigned to the survey results along with a very broad statement of recommended use and appropriate cautions which can be included in the executive summary or methodological section of an evaluation report. The rating system is flexible, easily adaptable to other workplace settings, and an effective learning tool. |
| Impact of a Financial Program on Knowledge and Practices of Adult Participants |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Cathy Bowen, Pennsylvania State University, cfb4@psu.edu |
| Marilyn Furry, Pennsylvania State University, mfurry@psu.edu |
| Rama Radhakrishna, Pennsylvania State University, brr100@psu.edu |
| Abstract: Reports continue to document the financial illiteracy of Americans. Lack of financial skills is illustrated by bankruptcy rates, increased credit card debt, and low savings rate. The purpose of this study was to assess the effectiveness of a financial education program on the knowledge and practices of participants. A pretest-posttest comparison group design was used to determine if participation in the program would lead to: 1) using a spending plans and recordkeeping system 2) starting a savings program or increase the amount contributed to an existing savings plan, 3) developing a plan to reduce the current debt load, and 4) using techniques to reduce their cost of using credit. Significant differences were found between the treatment and comparison groups' knowledge score on the posttest and delayed posttest. Treatment group participants not only made changes in behaviors relative to managing personal finances but also sustained those changes over a longer period. |
| Impact of AgrAbility Program: Helping Farmers to Stay in Farming |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Connie Baggett, Pennsylvania State University, bbc@psu.edu |
| Rama Radhakrishna, Pennsylvania State University, brr100@psu.edu |
| Linda Fetzer, Pennsylvania State University, lmf8@psu,edu |
| Abstract: Agriculture work is ranked as one of the most dangerous occupations in the nation (RTC, 2000). AgrAbility is a four year grant project through the U.S. Department of Agriculture that provides direct services to farmers with a disability or injury. Services provided by AgrAbility include disabled farmer evaluation, work sites assessments, modification recommendations, suggesting equipment adaptation, task restructuring, mobilizing and coordinating community resources and services, and facilitating rural independent living, etc. The overall purpose of the study was to assess the impact of Agrability Program. Overall, several of the targets of the project were met. Assessments and site visits conducted by AgrAbility staff has helped farmers to perform farm related tasks and continue farming. During the implementation phase, we encountered several challenges and barriers. In order to address these challenges, several changes are underway in the new project cycle (2006-10) which was recently funded by USDA. |
| Using Formative Evaluation to Culturally Adapt a Parenting Program for Black Fathers of Children With Special Needs |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| James C Bridgers Jr, University of Maryland, College Park, jbridgers@mayatech.com |
| Suzanne M Randolph, University of Maryland, College Park, suzanner@umd.edu |
| Cecil H Doggette, Health Services for Children with Special Needs Inc, cdoggette@hscsn.org |
| Sally Koblinsky, University of Maryland, koblinsk@umd.edu |
| Abstract: Ecological systems theory was used in a formative evaluation to assess the feasibility of a model program, developed for Black mothers' normative parenting, for use with Black fathers of children with special needs. There is scant literature on Black fathering; and parenting programs often focus on parents' challenges rather than strengths. This evaluation examined the rewards, strengths, challenges, and support needs of African American fathers parenting children with special needs. The theoretical framework guided development of focus group guides and analyses. Data were used to culturally adapt the model parenting program and select evaluation measures for the locally implemented intervention. |
| Consumer Assessment of Mental Health Service Providers' Cultural Competence |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amanda Gmyrek, University of Maryland, agmyrek@psych.umaryland.edu |
| Natasha Durant, University of Maryland Medical System, ndurant@kernan.umm.edu |
| Diana Seybolt, University of Maryland, dseybolt@psych.umaryland.edu |
| Iris Reeves, Maryland Mental Hygiene Administration, ireeves@dhmh.state.md.us |
| Abstract: According to the President's New Freedom Commission on Mental Health, one of the driving forces in transforming America's mental health system is to better address the mental health needs of our multicultural society by increasing access to culturally competent services. Currently, there are instruments that assess mental health service providers' cultural competence at the program- and individual practitioner-level, but few measures examine provider cultural competence from the consumers' perspective. This distinction is important given that the Commission also highlighted the need for consumer-driven care. This project was designed to examine the psychometric properties of a survey tool to be used in the assessment of provider cultural competence from the consumers' point of view. Data were collected from 406 adult mental health consumers who were actively engaged in treatment. Data analyses revealed that the survey tool had strong internal consistency reliability and content validity as well as adequate test-retest reliability. |
| Comparison of On-line and On-site Training for Public Health Emergency Preparedness |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Thomas Reischl, University of Michigan, reischl@umich.edu |
| Amy Sarigiannis, University of Michigan, abuss@umich.edu |
| Abstract: To expand the availability of public health emergency preparedness training opportunities, the Michigan Center for Public Health Preparedness has offered simultaneous web casts of several training events. This study examines the training evaluation data for participants who attended one of three training events in person (on-site) or viewed the same training event as a live web cast (on-line). The training events included full-day trainings preparing participants for radiological (n=49 on-line; n=87 on-site) and for chemical (n=40 on-line; n=32 on-site) emergencies and a "grand rounds" seminar on mental health implications of public health disasters (n=43 on-line; n=30 on-site). Using pretest-posttest designs, we noted similar gains in self-rated competencies and in content knowledge test scores for both training modalities. However, we noted that on-line training participants reported higher course satisfaction ratings than the on-site training participants. The discussion of the results will focus on costs and benefits of each training modality. |
| HIV/AIDS/STD-related Worry and Sexual Activity Within Adolescent Social Networks |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deladem Kusi-Appouh, Cornell University, dnk7@cornell.edu |
| Josephine Allen, Cornell University, jaa7@cornell.edu |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Jennifer Tiffany, Cornell University, jst5@cornell.edu |
| Lindy Williams, Cornell University, lbw@cornell.edu |
| Abstract: This paper examines the relationship of four personal worry factors to sexual activity among 15-19 year-olds. Adolescents were recruited by their peers through Participant-Driven Recruitment (PDR), a methodology that combines participatory research and Respondent-Driven Sampling (RDS). The social networks illustrated by the PDR/RDS recruitment data enabled us to evaluate the tendency for adolescents to form social ties based on similarity (homophily). The recruiter/recruitee social network was our unit of analysis. The survey instrument included measures of worry about 1) getting HIV/AIDS; 2) becoming pregnant/getting someone pregnant; 3) having sex; and 4) getting a sexually transmitted disease. Worry was measured as a continuous variable: never worried, sometimes worried, often worried and constantly worried. Results suggest that those adolescents who worried about these factors had increased odds of engaging in sexual activity during the month preceding the survey. Color graphics displaying the social networks according to each worry factor complement the analysis. |
| An Evaluation of Techniques for Recruiting Parents into HIV Prevention Research in Rural Communities: Teen Recruitment of Adults Versus Traditional Methods: What Works Best? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Josephine Allen, Cornell University, jaa7@cornell.edu |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Jennifer Tiffany, Cornell University, jst5@cornell.edu |
| Deladem Kusi-Appouh, Cornell University, dnk7@cornell.edu |
| Wendy Vonhof, Cornell University, wmv2@cornell.edu |
| Abstract: Reducing adolescent sexual risk-taking behaviors is one of the leading national priorities under the “Healthy People 2010” initiative, and adolescents in rural areas are particularly at high risk of acquiring STIs, including HIV. Adolescents reporting positive communication with parents are less likely to engage in sexual risk-taking behaviors. Yet, engaging teenagers and their parents in HIV prevention interventions remains a challenge. We report our experience with recruiting parents or guardians into an HIV prevention research project through teenage participants who had been engaged using a Participant-Driven Recruitment approach. Our results indicate that recruitment of adults into HIV preventive interventions may work better when done using traditional techniques, rather than by asking adolescents to recruit adults. The reasons and applications to other interventions and evaluations will be discussed. |
| Evaluating an Online New Teacher Mentorship Program: Science, Technology, Engineering, and Mathematics Mentoring Program (STEMMP) |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Herbert Struss, University of Minnesota, strus010@umn.edu |
| Joel Donna, University of Minnesota, donna010@umn.edu |
| Gillian Roehrig, University of Minnesota, roehr0013@umn.edu |
| Abstract: The subject of the evaluation was a virtual learning community – Science, Technology, Engineering, and Mathematics Mentoring Program (STEMMP). The goals of STEMMP were to improve teacher job satisfaction and to increase teachers' use of reform-based teaching strategies, with the ultimate goal being to improve student achievement. Besides outlining the evaluation, the poster highlights the challenges of evaluating online learning communities. Approximately sixty beginning (untenured) STEM teachers throughout Minnesota participated in the STEMMP program, along with sixty experienced STEM teachers who served as mentors to the untenured teachers. Teachers represented a wide range of settings from urban to rural, and public to charter. Electronic communication features of STEMMP included private chat rooms and larger forums. For example, the inquiry discussion area was developed as a learning cycle where the teacher planned, implemented, and reflected on a lesson or unit with their peers and mentor. |
| Evaluator Certification: More than 30 Years Without Resolution |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Arisara Roengsumran, University of Minnesota, roen0013@umn.edu |
| Abstract: The need to establish a certification system for evaluators has been a controversial topic for more than 30 years. This poster will present key issues, arguments on both sides of the debate, and results from the only survey about certification for evaluators that has been conducted by the AEA. Examples of certification practices in other professions will be illustrated to provide a basis for the potential design of a system for evaluation. An informal short survey of AEA conference attendees will be conducted to collect attendee's opinions toward certification and to increase interaction with the audience. This poster will help members recognize the benefits and issues associated with establishing and maintaining a certification system. It will also inform members about recent changes and development in the profession that may influence their decisions regarding certification. This poster should increase dialogue about certification, which may generate further actions towards resolving this issue. |
| Tools for a Successful Randomized Control Trial (RCT) in an International Arena |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kathlyn Rohrbaugh, Macro International Inc, kathlyn.g.rohrbaugh@orcmacro.com |
| Abstract: Randomized controlled trials (RCTs) are difficult to design and implement, but are being used increasingly to measure effectiveness and impact. This difficulty multiplies when they are conducted in other countries. The key components of any RCT are high quality design and data collection—without it the analysis becomes more complicated if not entirely useless. Conducting an RCT in an international setting poses challenges beyond those that are encountered here in the United States. To address these challenges, it is important to have an understanding of the culture, to provide training, and to design the appropriate administrative tools for data collection. This poster session will provide an overview of the design, approaches, and tools used by Macro International Inc. for an RCT conducted in 2006 of a teacher training program on economic education in three Russian regions. |
| An Innovative Methodology for Measuring the Results of Peacebuilding Media Interventions |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Alice Rowley, Search for Common Ground, arowley@sfcg.org |
| Ratiba Taouti-Cherif, Search for Common Ground, rtcherif@sfcg.org |
| Abstract: Measuring the results of peacebuilding programs and attributing them to their interventions is one of the main challenges facing peacebuilding organizations. For the last two decades, Search for Common Ground has been using media to transform the way people deal with conflict. While external evaluations of its programs report positive listener/viewership results and an increase in knowledge of the programs' issues, they fail to capture and relate the latter to behavioral and attitudinal change in a systematic way. This presentation describes an innovative formative research methodology, which adapts the health sector cohort study approach and participatory research techniques to evaluating peacebuilding media programs. It will seek to measure knowledge and behavior changes in the short to medium term, but also attribute those changes to a specific media intervention. The presentation also outlines the tools used and the initial findings from a pilot tested in Nepal. |
| Promoting Science, Technology, Engineering, Mathematics, and Geography (STEM-G) Through Professional Development: Learning From Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Karen Chen, Wheeling Jesuit University, kchen@cet.edu |
| Laurie Ruberg, Wheeling Jesuit University, lruberg@cet.edu |
| Judy Martin, Wheeling Jesuit University, jmartin@cet.edu |
| Abstract: This poster presents an evaluation project that was conducted to assess the effectiveness of a nationwide science, technology, engineering, mathematics, and geography (STEM-G) educational program established in 2003. The project highlights the need for sound evaluation to document the quality and impact of the program. An interpretive, multiple case study approach was employed to review the program's impact on students. Multiple data sources were collected and analyzed, including teachers' observations of students, e-Portfolios, and student interest surveys. The evaluation team analyzed the data for emerging themes from a set of cluster-based, randomly selected case study schools. The poster presentation will discuss evaluation themes and challenges encountered. |
| Satisfaction Surveys and Their Relevance in Service Delivery |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Maureen Rubin, Program Development and Evaluation Associates, maureen.rubin@pdea.org |
| Abstract: This paper is based on findings from a SAMHSA funded system of care site in a southeastern state in the United States of America. As part of the funding protocol all funded sites are to participate in a national evaluation which measures various socio-demographic and psychosocial developmental aspects of the youth and families along with their level of satisfaction with services received. For purposes of this paper, data on the satisfaction levels of youth and families as measured by Youth Services Survey (YSS) and Youth Services Survey for Families (YSS-F) will be highlighted. It will highlight the relevance and importance of sharing what was learned through engaging stakeholders in the evaluation process and infusing knowledge gained through the process with the appropriate personnel to make required changes, if any. |
| Evaluating an Organizational Capacity Building Program Designed to Increase Diversity Among Health Services Researchers |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Andrea S Burling, American Institutes for Research, aburling@air.org |
| Roger E Levine, American Institutes for Research, rlevine@air.org |
| Darlene Russ-Eft, Oregon State University, darlene.russeft@oregonstate.edu |
| Jennnifer Stephens, American Institutes for Research, jstephens@air.org |
| Abstract: This paper will describe a large-scale organizational capacity building evaluation, discuss challenges faced and strategies for overcoming them, describe a unique methodology being used, and may present initial findings. The American Institutes for Research is evaluating two programs funded by the Agency for Healthcare Research and Quality (AHRQ) that are designed to improve the organizational capacities to conduct health services research of academic institutions that predominantly serve racial/ethnic minorities, and institutions located in geographic areas that have had historically low levels of AHRQ funding. The evaluation is determining how, and to what extent, grantees have strengthened their institutional infrastructures and enhanced the capabilities of individual faculty in relation to their ability to undertake health services research. Multiple methods are being used including a unique document data abstraction approach, as well as telephone interviews and site visits. Key challenges include defining and measuring success for organizational capacity building programs and efficiently accessing organizational data. |
| Steps to Developing Evaluation Capacity: Enhancing the Learning Potential of Nonprofit Organizations |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Karen Russon, Evaluation Capacity Development Group, krusson@gmail.com |
| Abstract: For a nonprofit organization to not only survive but thrive, it must become a learning organization, or better stated, an organization of perpetual learners. It needs informal feedback mechanisms as well as more formal monitoring and measurement systems in order to achieve optimal learning. Evaluation can be viewed as the cornerstone of a learning organization. It drives learning. How does an organization move from a desire to learn more about what and how it is doing to incorporating evaluation strategies which support learning? It is accomplished through developing its evaluation capacity. Evaluation capacity development can means different things to different people. The strategy presented in this proposal focuses on organizational as distinct from individual evaluation capacity. It presents the steps necessary for creating an infrastructure to support evaluation prior to training specific individuals to do evaluation. These thoughtful steps will enhance the development of a learning organization. |
| Developmental Evaluation as a Learning Tool for Novice Evaluators |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chris De Luca, Queen's University, 2cd16@qlink.queensu.ca |
| Laura McEwen, Queen's University, laura@education.concordia.ca |
| Hana Saab, Queen's University, saabh@educ.queensu.ca |
| Abstract: Organizations are frequently required to respond on short notice to emergent needs and policy decisions. This presentation examines the contribution of evaluation to the expansion and implementation of a new course in student assessment at a Faculty of Education, initiated in response to Ministry recommendations. Patton's developmental evaluation (1994) provides a framework for the evaluation conducted by a team of graduate students despite recommendations that such an approach may not be feasible for novice evaluators. The paper will discuss how the evaluation contributed to the professional learning of the novice evaluation team, to the positioning of evaluation as an ongoing iterative and interpretive process that contributed to the emergent nature of the course, and to creating a level of sustained interactivity among the stakeholders. Suggested accommodations and adjustments to the developmental evaluation model will be discussed in anticipation of its continued use as a learning tool for students in evaluation. |
| Comparison of On-line and On-site Training for Public Health Emergency Preparedness |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Thomas Reischl, University of Michigan, reischl@umich.edu |
| Amy Sarigiannis, University of Michigan, abuss@umich.edu |
| Abstract: To expand the availability of public health emergency preparedness training opportunities, the Michigan Center for Public Health Preparedness has offered simultaneous web casts of several training events. This study examines the training evaluation data for participants who attended one of three training events in person (on-site) or viewed the same training event as a live web cast (on-line). The training events included full-day trainings preparing participants for radiological (n=49 on-line; n=87 on-site) and for chemical (n=40 on-line; n=32 on-site) emergencies and a "grand rounds" seminar on mental health implications of public health disasters (n=43 on-line; n=30 on-site). Using pretest-posttest designs, we noted similar gains in self-rated competencies and in content knowledge test scores for both training modalities. However, we noted that on-line training participants reported higher course satisfaction ratings than the on-site training participants. The discussion of the results will focus on costs and benefits of each training modality. |
| Evaluation of Research Program Structures Using Logic Models and Value Stream Management: A Pilot Study |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Delia J Valles-Rosales, New Mexico State University, dvalles@nmsu.edu |
| Elizabeth Schott, New Mexico State University, libby.schott@gmail.com |
| Barbara Lugo, New Mexico State University, blugo@nmsu.edu |
| Abstract: This paper presents an innovative approach to evaluate research program structures at college level. The Alliance for Minority Participation Undergraduate Research Assistantship funded by the National Science Foundation at New Mexico State University has been selected to be the pilot program for this study. The purpose of the study was to evaluate the structure of the program to locate areas for improvement that have not been identified yet by the current evaluation practices in the direction of supporting the development of students that increases the quality of educational experience and motivates them towards graduate studies. The methodology consists of employing logic models combined with value stream management techniques. Our paper discusses the overall process used to conduct the evaluation as well as our initial results and recommendations. |
| Conducting and Evaluating Effective Translations to Promote Linguistic Competence in a Federally Funded National Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sylvia Fisher, United States Department of Health and Human Services, sylvia.fisher@samhsa.hhs.gov |
| Michelle Schurig, Macro International Inc, michelle.l.schurig@orcmacro.com |
| Abstract: Translation is a complex process laden with numerous challenges, including (a) preserving the integrity and semantic equivalence between the original written or oral text and the translated text; and (b) maintaining a culturally sensitive translation that ensures that cultural traditions and typical language idioms are addressed in a respectful manner. In addition, successful oral and written translations reduce the potential for statistical and other forms of bias whenever possible, somewhat minimizing the potential for error in data collection activities (Fisher & Gerber, 2002). Several translation approaches are available, including back translation, expert groups (such as a translation advisory committee), cognitive interviews, focus groups, item response theory, respondent and interviewer debriefings, certified translators, use of the decentering method, and behavior coding. Marín and Marín (1991) especially favor translation-by-committee approaches for Spanish translations because Spanish is spoken in nearly 25 nations and, accordingly, is subject to many idioms and variations both in meaning and pronunciation. These variations and cultural idioms and differences can result in translations that have radically different meanings. Additionally, if a translation is not well-designed from its inception, formidable efforts often become necessary to disentangle the roots of the translation problem, which is difficult once a translation is completed. The literature on translation helped justify a decision for the national evaluation of systems of care for children with serious mental health challenges to develop an iterative process of review and feedback, through the establishment of a translation committee with strong links to the audience for translated evaluation products. This poster describes the process and the outcome of this approach to translation. |
| Consumer Assessment of Mental Health Service Providers' Cultural Competence |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amanda Gmyrek, University of Maryland, agmyrek@psych.umaryland.edu |
| Natasha Durant, University of Maryland Medical System, ndurant@kernan.umm.edu |
| Diana Seybolt, University of Maryland, dseybolt@psych.umaryland.edu |
| Iris Reeves, Maryland Mental Hygiene Administration, ireeves@dhmh.state.md.us |
| Abstract: According to the President's New Freedom Commission on Mental Health, one of the driving forces in transforming America's mental health system is to better address the mental health needs of our multicultural society by increasing access to culturally competent services. Currently, there are instruments that assess mental health service providers' cultural competence at the program- and individual practitioner-level, but few measures examine provider cultural competence from the consumers' perspective. This distinction is important given that the Commission also highlighted the need for consumer-driven care. This project was designed to examine the psychometric properties of a survey tool to be used in the assessment of provider cultural competence from the consumers' point of view. Data were collected from 406 adult mental health consumers who were actively engaged in treatment. Data analyses revealed that the survey tool had strong internal consistency reliability and content validity as well as adequate test-retest reliability. |
| Educating Evaluations: Designing and Implementing Impact Evaluations for Non-formal and Vocational Education Development Projects |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Marc Shapiro, Independent Consultant, shapiro@urgrad.rochester.edu |
| Abstract: Not all international educational projects are alike, and neither should be their evaluations. Projects involving non-formal education or vocational/technical education often exhibit substantial differences from projects focused on basic education. Differences include variety of types of interventions involved, proximity with measurable outcomes outside of the academic context, potential linkages between causal agent and receptor population, and more. These differences affect how evaluations should be designed to measure outcomes and impacts of these types of interventions as well as affecting procedures to conduct the evaluations. This presentation will discuss learnings across countries and across education projects conducted by a variety of organizations and donors. It examines more traditional basic education projects, vocational/technical education projects, and the growing field of non-formal education focusing on key differences that affect the design and implementation of evaluations. |
| Kinder, Gentler Pre-tests for Math-Science Partnership Teachers |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Linda Heath, Loyola University, Chicago, lheath@luc.edu |
| Aisha Leverett, Loyola University, Chicago, jlevere@luc.edu |
| David Slavsky, Loyola University, Chicago, dslavsk@luc.edu |
| Abstract: A recurring theme at the regional meetings of evaluators for the NSF-funded Math Science Partnerships (MSP's) was the difficulty of obtaining valid pre-test measures of teachers' knowledge and skills. The problem is not that valid, standardized measures do not exist. The problem is that completing the pre-test assessment and recognizing they have done poorly frustrates, embarrasses, and demoralizes program participants. Evaluators at the regional MSP meetings reported needing to spend a whole day (often out of only five allotted to the program) to build cohesiveness and comfort in the group prior to the test or to repair damage to morale after the test. In this paper, we examine correlates and subscales within one standardized test (The Force Concept Inventory) to see if a less-threatening but equally valid and reliable pre-test instrument is possible, using data from the Loyola University Chicago/ Chicago Public Schools (LUC/CPS) Physics Endorsement Project. |
| Challenges in Program Evaluation: Lessons Learned From Evaluating Child Maltreatment Prevention Interventions |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lei Zhang, Georgia State University, alhlzx@langate.gsu.edu |
| Angela Snyder, Georgia State University, alhabs@langate.gsu.edu |
| Karen Minyard, Georgia State University, alhkjm@langate.gsu.edu |
| Abstract: Child maltreatment program evaluation raises several ethical/technical issues that have yet to be addressed by evaluation research. A real-world example is Georgia Health Policy Center's evaluation of the Intensive Home Visiting (IHV) program funded by Georgia Division of Public Health. The IHV program features trained, supervised public health nurses that will provide frequent and intense home visits with families of young children identified as at-risk for abuse and neglect. The nurses will use standardized tools and curricula to teach parents about child development, health, and safety. The major ethical/technical issues involved in this evaluation include (1) obtaining sample sizes large enough to detect statistically significant impacts; (2) maintaining victim safety; and (3) tracking participants in a way that minimizes privacy violation. Our feedback from the evaluation indicates (1) evaluation needs to be sensitive to its contexts; and (2) ethical guidance provided by the Joint Committee needs to be more specific. |
| Evaluating an Organizational Capacity Building Program Designed to Increase Diversity Among Health Services Researchers |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Andrea S Burling, American Institutes for Research, aburling@air.org |
| Roger E Levine, American Institutes for Research, rlevine@air.org |
| Darlene Russ-Eft, Oregon State University, darlene.russeft@oregonstate.edu |
| Jennnifer Stephens, American Institutes for Research, jstephens@air.org |
| Abstract: This paper will describe a large-scale organizational capacity building evaluation, discuss challenges faced and strategies for overcoming them, describe a unique methodology being used, and may present initial findings. The American Institutes for Research is evaluating two programs funded by the Agency for Healthcare Research and Quality (AHRQ) that are designed to improve the organizational capacities to conduct health services research of academic institutions that predominantly serve racial/ethnic minorities, and institutions located in geographic areas that have had historically low levels of AHRQ funding. The evaluation is determining how, and to what extent, grantees have strengthened their institutional infrastructures and enhanced the capabilities of individual faculty in relation to their ability to undertake health services research. Multiple methods are being used including a unique document data abstraction approach, as well as telephone interviews and site visits. Key challenges include defining and measuring success for organizational capacity building programs and efficiently accessing organizational data. |
| Inclusion of Children With Disabilities in School Age Care Settings: Provider's Perspective |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Hamida Jinnah, University of Georgia, hamidajinnah@gmail.com |
| Zolinda Stoneman, University of Georgia, zo@ihdd.uga.edu |
| Christine Todd, University of Georgia, ctodd@fcs.uga.edu |
| Anika Francis, University of Georgia, anikakai@uga.edu |
| Abstract: This study assesses the needs of child care providers related to inclusion of children with disabilities in school age care settings. Focus groups and phone interviews were conducted with providers. Data was analyzed using Nudist (N6). Providers mentioned that they enjoy seeing children progress in their abilities and are constantly rewarded by the love and affection children with disabilities give to them. Providers however, also experience several challenges in including children with disabilities in their setting(s). These include inadequate staff-child ratio, staff attitudes, staff training and access to information and resources for inclusion. Additionally, providers face problems due to gaps in communication with parents, school teachers and therapists. Providers would like to have more training, more resources including additional staff, more funding, and more information about available resources. They would like to see greater collaboration and networking among different individuals working with school age children having disabilities. |
| Collaboration and Evaluation Capacity Building: Lessons From a Learning Environment |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Randi K Nelson, University of Minnesota, nelso326@umn.edu |
| David Fischer, University of Minnesota, fisch413@umn.edu |
| Herbert Struss, University of Minnesota, strus0101@umn.edu |
| Abstract: Evaluation, data based decision making, and collaboration have tremendous potential for building the capacity of teachers and program evaluators to learn from evaluation. This poster highlights the views of teachers and program staff toward collegiality and data use to inform practice, during a three-year collaborative evaluation capacity building project between a university and two public schools. Program evaluation traditionally focuses on stakeholders who receive program services. In collaborations, the views of both the service providers and recipients must be considered. This poster illustrates how annual surveys of teachers and follow up interviews with teachers and program staff have been used to provide all stakeholders the opportunity to learn from their experiences. In this project, use of data from teachers and program staff demonstrates collaboration and data based decision making among all participants and models evaluation to promote collaborative capacity and learning in addition to its more traditional purposes. |
| Evaluating an Online New Teacher Mentorship Program: Science, Technology, Engineering, and Mathematics Mentoring Program (STEMMP) |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Herbert Struss, University of Minnesota, strus010@umn.edu |
| Joel Donna, University of Minnesota, donna010@umn.edu |
| Gillian Roehrig, University of Minnesota, roehr0013@umn.edu |
| Abstract: The subject of the evaluation was a virtual learning community – Science, Technology, Engineering, and Mathematics Mentoring Program (STEMMP). The goals of STEMMP were to improve teacher job satisfaction and to increase teachers' use of reform-based teaching strategies, with the ultimate goal being to improve student achievement. Besides outlining the evaluation, the poster highlights the challenges of evaluating online learning communities. Approximately sixty beginning (untenured) STEM teachers throughout Minnesota participated in the STEMMP program, along with sixty experienced STEM teachers who served as mentors to the untenured teachers. Teachers represented a wide range of settings from urban to rural, and public to charter. Electronic communication features of STEMMP included private chat rooms and larger forums. For example, the inquiry discussion area was developed as a learning cycle where the teacher planned, implemented, and reflected on a lesson or unit with their peers and mentor. |
| Implementation of Evidence-based Interventions in Community Mental Health Settings: Organizational Factors and Provider Perspectives Impacting Implementation Processes and Child and Family Outcomes |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Elizabeth Douglas, Macro International Inc, elizabeth.b.douglas@orcmacro.com |
| Christine Walrath, Macro International Inc, christine.m.walrath-greene@orcmacro.com |
| Bhuvana Sukumar, Macro International Inc, bhuvana.sukumar@orcmacro.com |
| Abstract: A growing body of knowledge exists about evidence-based practices (EBPs) that work to improve outcomes for children with mental health needs. However, as much of this research is conducted in tightly controlled settings (Burns & Hoagwood, 2002), it often reflects intervention efficacy while leaving questions about effectiveness in diverse "real-world" settings. Moreover, reports indicate that even in the case of interventions found to be effective through evaluation in clinical settings, the protocols are not widely understood by clinicians, and are not being translated into practice nearly often enough (NIMH, 2000). As part of an effort to contribute to the knowledge base regarding EBP implementation and effectiveness, this symposium will present findings related to organizational factors and provider perspectives impacting implementation as well as the results of a study examining the effectiveness of Parent-Child Interaction Therapy in community mental health settings. Implications for future implementation and effectiveness research will be discussed. |
| The Evaluation of a Listening-While-Reading Program: Challenges in and Recommendations From Conducting an In-school Evaluation Experiment |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Aubrie Swan, Rutgers University, aswan@eden.rutgers.edu |
| Abstract: Recently, the federal Department of Education has been emphasizing the use of scientifically-based educational program evaluations, while simultaneously deemphasizing various other evaluation methods. Researchers were hired to evaluate a program that intends to help students with classified reading difficulties improve their reading skills, namely by using listening-while-reading technology. They employed an experimental design using control groups and random assignment in order to validate the program's effectiveness, but no significant differences between groups were found. This poster describes some of the challenges that confronted the evaluators while doing scientifically-based evaluation research in classroom settings, and will give recommendations about useful and non-constructive methods and practices in conducting scientifically-based in-school evaluation research. It will also discuss the complexities of carrying out evaluations involving students with learning disabilities and in urban schools. |
| Influences on the Career Paths of Underrepresented Minority Students in a Biology Enrichment Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jennifer Sweeney, University of California, Davis, jksweeney@ucdavis.edu |
| Merna Villarejo, University of California, Davis, mrvillarejo@ucdavis.edu |
| Amy Barlow, University of California, Davis, aebarlow@ucdavis.edu |
| Abstract: A two-stage survey/interview evaluation design was used to test the premise that science enrichment activities, particularly participation in undergraduate research, encourage underrepresented minority students to pursue biology-based professions. Alumni who participated in an educational enrichment program at the University of California, Davis responded to a survey and a follow-up telephone interview regarding influences on their career choices. The survey and interview complement a statistical study associating undergraduate research participation and positive graduation outcomes including graduation in a biology major. Survey respondents (n=202) detailed undergraduate experiences and career choices. Over 100 interviewees reflected on the contribution of pre-college, undergraduate, and post-graduate experiences to their subsequent career choices in the CATI follow-up interview. Preliminary analysis suggests that undergraduate research experience affects career choice by providing an opportunity for students to discover, refine, or confirm their interests. The value of this experience appears to persist regardless of whether students pursue a research career. |
| An Innovative Methodology for Measuring the Results of Peacebuilding Media Interventions |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Alice Rowley, Search for Common Ground, arowley@sfcg.org |
| Ratiba Taouti-Cherif, Search for Common Ground, rtcherif@sfcg.org |
| Abstract: Measuring the results of peacebuilding programs and attributing them to their interventions is one of the main challenges facing peacebuilding organizations. For the last two decades, Search for Common Ground has been using media to transform the way people deal with conflict. While external evaluations of its programs report positive listener/viewership results and an increase in knowledge of the programs' issues, they fail to capture and relate the latter to behavioral and attitudinal change in a systematic way. This presentation describes an innovative formative research methodology, which adapts the health sector cohort study approach and participatory research techniques to evaluating peacebuilding media programs. It will seek to measure knowledge and behavior changes in the short to medium term, but also attribute those changes to a specific media intervention. The presentation also outlines the tools used and the initial findings from a pilot tested in Nepal. |
| Amplifying Community Voices: Enhancing the Needs Assessment Process |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Catherine Carey, Practical Research, catherine@practical-research.com |
| Ede Taylor, Belair-Edison Healthy Community Coalition, etaylor@becoalition.org |
| Abstract: Needs assessments are commonly conducted during the program development process. Needs assessments generally include a review publicly available data sets and / or a survey. Seeking a better understanding of survey results, a community activist and researcher added a focus group where community leaders reflected on survey results and added their opinions about priorities for services. Focus group participants were invited due to their leadership positions in the community. They received a summary of the survey results prior to the meeting and the survey results were presented at the meeting. The purpose of this poster is to present the focus group methodology and to engage in discussion about its usefulness: Would needs assessment practice be improved by the addition of a focus group process? When? Where there unique features that facilitated a useful experience with lengthening the needs assessment process? |
| Institutionalizing Utilization-focused approaches to Evaluation in Non-Governmental Organizations (NGOs): Creating a Culture of Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carrie Willimann, Academy for Educational Development, cwillimann@aed.org |
| Winnehl Tubman, Academy for Educational Development, wtubman@aed.org |
| HyeJin Kim, Academy for Educational Development, hkim@aed.org |
| Arushi Terway, Academy for Educational Development, aterway@aed.org |
| Ana Florez, Academy for Educational Development, aflorez@aed.org |
| Audrey-Marie Moore, Academy for Educational Development, amoore@aed.org |
| Abstract: Evaluation can be a tool for ensuring that education projects provide data and information at the policy level to transfer technical innovation. However, evaluation, as currently structured under many donor funded projects, does not provide the necessary information to inform this process. Nor does it allow for a cohesive approach to draw lessons learned across projects to inform broader policy. Clearing space in the existing political economic landscape so that the substance of sound educational improvement can be implemented is the operational core of what a network of institutions and actors (-reform support infrastructure-) needs to do for reform to succeed. This panel examines how a small evaluation group within an international NGO attempted to clear and fill space for utilization-focused evaluation to flourish in order to create a more cohesive approach to evaluation, build the capacity of staff to implement UFE frameworks, and improve organizational learning across education projects. |
| Insights on Learning From Using Qualitative Methods to Evaluate Safety Training in Mining: Gaining Deeper Understanding |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lori Matthews, Spokane Research Laboratory, lmatthews@cdc.gov |
| Theodore Teske, Spokane Research Laboratory, tteske@cdc.gov |
| Abstract: The Health Communications team at the Spokane Research Laboratory (SRL), part of the National Institute for Occupational Safety and Health (NIOSH), uses qualitative research methods to develop and evaluate effective safety training for the mining industry. This poster presentation highlights various qualitative methods used and insights gained regarding the use of such methods. General qualitative themes will be presented and further illustrated using narrative data collected from multiple evaluation studies. Some of themes include: quality, credibility, accurate content, engaging while educational, and the extended value or transferability of safety training. This presentation will focus specifically on the learning insights gained from using qualitative methods, such as storytelling and ethnography, to evaluate safety training in mining. Ultimately, the ability to gain “deeper” understanding from the use of qualitative methods in evaluation research will be highlighted and promoted in this presentation. |
| Immersion in School-based Evaluation Communities |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Douglas Huffman, University of Kansas, huffman@ku.edu |
| Kelli Thomas, University of Kansas, kthomas@ku.edu |
| Karen Lombardi, University of Kansas, lombaka6@yahoo.com |
| Carrie Hohl, University of Kansas, cmhohl@salvajor.com |
| Abstract: This poster session will focus on the results of the Collaborative Evaluation Communities in Urban Schools project (CEC Project) at the University of Kansas. The project was designed to enhance the evaluation capacity of K-8 schools through collaborative evaluation communities comprised of teachers, instructional coaches, graduate students, and university faculty. The goals of the project were to improve the evaluation capacity of urban schools, develop graduate level educational leaders with the knowledge and skills to evaluate science and mathematics education programs, and develop the evaluation capacity of K-8 teachers. The poster will focus on the impact of using evaluation to develop elementary mathematics and science instruction in schools. The poster will also address the broader question: In what ways can collaborative evaluation create learning communities of practice? |
| Identifying Challenges to Evaluating Abstinence Education Programs |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sharon McGroder, The Lewin Group, sharon.mcgroder@lewin.com |
| Jane Koppelman, The Lewin Group, jane.koppelman@lewin.com |
| Mark Thomas, The Lewin Group, mark.thomas@lewin.com |
| Shanise Hill, The Lewin Group, shanise.hill@lewin.com |
| Abstract: The federally funded Center for Research and Evaluation on Abstinence Education was launched in 2006 to build capacity for conducting rigorous evaluations on abstinence education (AE) programs. The Center's first task was to conduct an assessment of the evaluation needs of the field—the first formal assessment of its kind. Evaluation challenges were identified through the following mechanisms: (1) discussions with practitioners, policymakers and evaluators in the field; (2) discussions with national evaluation experts and contractors assisting federal grantees in conducting evaluations; and (3) analysis of an array of published and unpublished AE evaluation studies. Evaluation challenges were documented from the above sources and listed by informant into four categories: (1) design (process and impact) challenges, (2) measurement issues, (3) analyzing, interpreting and reporting findings, and (4) implementation challenges. This needs assessment will help pinpoint the types of assistance evaluators require to conduct quality research on abstinence education. |
| Evaluation and Learning: Project ECHO, a Training and Capacity Building Model for Hepatitis C |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Summers Kalishman, University of New Mexico, skalish@salud.unm.edu |
| Denise Dion, University of New Mexico, dmdion@salud.unm.edu |
| Jan Mines, University of New Mexico, jmines@salud.unm.edu |
| Karla Thornton, University of New Mexico, kthornton@salud.unm.edu |
| Sanjeev Arora, University of New Mexico, sarora@salud.unm.edu |
| Abstract: Using telemedicine infrastructure for clinical learning, Project ECHO (Extension for Community Healthcare Outcomes) trains primary care providers who serve rural, underserved and prison populations in providing Hepatitis C (HCV) treatment for their patients with the guidance of specialists. This model is relevant to populations with chronic, complex diseases or with limited access to specialists (Arora et al, 2007). Using cased-based and iterative learning, primary care providers present HCV patients during weekly telemedicine clinics. Specialist physicians co-manage patients with primary care providers utilizing evidence-based treatment protocols. Learning loops are created through iterative and collaborative case-based discussion during each clinic. The evaluation views the provider as learning a complex set of behaviors with extensive support in a community of similar practitioners. The evaluation draws from self efficacy, situated learning and social cognitive theory. Providers endorse their knowledge growth and self-efficacy about Hepatitis C screening, symptom management, and treatment. |
| Validity of Patient-reported Health-related Quality of Life Global Ratings of Change Using Structural Equation Modeling |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-generated single-item global ratings of change (GRCs) are often used as anchors of health-related quality of life (HRQoL) since they are easy for clinicians to interpret and provide clinicians with change score interpretations that may guide clinical decision-making and assist in determining intervention efficacy. Although this approach may be preferred, the validity of the anchor-based approach is currently under scrutiny. We used structural equation modeling to estimate the explained variation in domain-specific GRCs that is accounted for by Time 1 (T1) and Time 2 (T2) domain-specific summary scores from the Short-Form 36, V2 (SF-36) Health Survey in 356 asthma outpatients. Correlations revealed that domain-specific GRCs were more strongly related to T2 than T1 domain summary scores, indicating that patients were not equally relying on T1 and T2 to generate the GRCs. Furthermore, T1-domain summary scores were not of equal magnitude and opposite sign as compared to T2 scores. There is insufficient evidence to establish SF-36 domain-specific GRC validity in asthma outpatients. |
| A Comparison of Traditional and Rasch Cut Points for Assessing Clinically Important Change in Health-Related Quality of Life Among Patients with Asthma |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-perceived change in health-related quality of life (HRQoL) domains has often been classified using a 15-point patient transition rating scale with the arbitrarily-derived change levels of minimal (2, 3 or -2, -3), moderate (4, 5 or -4, -5) and large (6, 7 or -6, -7). Therefore, in order to determine the adequacy of these traditional change levels, item response theory methodology was employed to derive empirically-based cut points for the change levels. Our sample included 396 asthmatic outpatients who completed bimonthly telephone interviews on the Asthma Quality of Life Questionnaire and transition rating items over one year. After computing Rasch partial credit model transition rating cut points, we compared the mean domain-specific HRQoL changes from Time 1 to Time 2 under each traditional transition rating change level with those under Rasch-derived change levels. Although traditional and Rasch categorizations for small, moderate, and large changes slightly differed, nearly all mean changes between classification approaches were comparable. Traditional transition rating cut points remain suitable to assess HRQoL clinical significance in outpatients with asthma. |
| An Evaluation of Techniques for Recruiting Parents into HIV Prevention Research in Rural Communities: Teen Recruitment of Adults Versus Traditional Methods: What Works Best? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Josephine Allen, Cornell University, jaa7@cornell.edu |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Jennifer Tiffany, Cornell University, jst5@cornell.edu |
| Deladem Kusi-Appouh, Cornell University, dnk7@cornell.edu |
| Wendy Vonhof, Cornell University, wmv2@cornell.edu |
| Abstract: Reducing adolescent sexual risk-taking behaviors is one of the leading national priorities under the “Healthy People 2010” initiative, and adolescents in rural areas are particularly at high risk of acquiring STIs, including HIV. Adolescents reporting positive communication with parents are less likely to engage in sexual risk-taking behaviors. Yet, engaging teenagers and their parents in HIV prevention interventions remains a challenge. We report our experience with recruiting parents or guardians into an HIV prevention research project through teenage participants who had been engaged using a Participant-Driven Recruitment approach. Our results indicate that recruitment of adults into HIV preventive interventions may work better when done using traditional techniques, rather than by asking adolescents to recruit adults. The reasons and applications to other interventions and evaluations will be discussed. |
| HIV/AIDS/STD-related Worry and Sexual Activity Within Adolescent Social Networks |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deladem Kusi-Appouh, Cornell University, dnk7@cornell.edu |
| Josephine Allen, Cornell University, jaa7@cornell.edu |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Jennifer Tiffany, Cornell University, jst5@cornell.edu |
| Lindy Williams, Cornell University, lbw@cornell.edu |
| Abstract: This paper examines the relationship of four personal worry factors to sexual activity among 15-19 year-olds. Adolescents were recruited by their peers through Participant-Driven Recruitment (PDR), a methodology that combines participatory research and Respondent-Driven Sampling (RDS). The social networks illustrated by the PDR/RDS recruitment data enabled us to evaluate the tendency for adolescents to form social ties based on similarity (homophily). The recruiter/recruitee social network was our unit of analysis. The survey instrument included measures of worry about 1) getting HIV/AIDS; 2) becoming pregnant/getting someone pregnant; 3) having sex; and 4) getting a sexually transmitted disease. Worry was measured as a continuous variable: never worried, sometimes worried, often worried and constantly worried. Results suggest that those adolescents who worried about these factors had increased odds of engaging in sexual activity during the month preceding the survey. Color graphics displaying the social networks according to each worry factor complement the analysis. |
| The Development and Validation of the Evaluation Involvement Scale |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Toal, University of Minnesota, toal0002@umn.edu |
| Abstract: Evaluation researchers and practitioners acknowledge that involving stakeholders in an evaluation increases buy-in, understanding, and use. Although there is a growing amount of qualitative research on participatory approaches that provides strong support for the positive impact of participation on increased use, to date there are no valid quantitative measures available to measure stakeholder perceptions of participation in evaluations. Researchers and evaluators describe the ways in which stakeholders participate, but rarely are stakeholders asked to reflect or report on the depth of participation. The purpose of this research is to validate a measure of project involvement in multi-site evaluations called the Evaluation Involvement Scale. In Fall 2006, data were collected from an electronic survey sent to project-level evaluators and principal investigators of four National Science Foundation program evaluations (N=346), along with follow-up telephone interviews to survey respondents in Spring 2007. Validity will be demonstrated using both theoretical and empirical evidence. |
| Inclusion of Children With Disabilities in School Age Care Settings: Provider's Perspective |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Hamida Jinnah, University of Georgia, hamidajinnah@gmail.com |
| Zolinda Stoneman, University of Georgia, zo@ihdd.uga.edu |
| Christine Todd, University of Georgia, ctodd@fcs.uga.edu |
| Anika Francis, University of Georgia, anikakai@uga.edu |
| Abstract: This study assesses the needs of child care providers related to inclusion of children with disabilities in school age care settings. Focus groups and phone interviews were conducted with providers. Data was analyzed using Nudist (N6). Providers mentioned that they enjoy seeing children progress in their abilities and are constantly rewarded by the love and affection children with disabilities give to them. Providers however, also experience several challenges in including children with disabilities in their setting(s). These include inadequate staff-child ratio, staff attitudes, staff training and access to information and resources for inclusion. Additionally, providers face problems due to gaps in communication with parents, school teachers and therapists. Providers would like to have more training, more resources including additional staff, more funding, and more information about available resources. They would like to see greater collaboration and networking among different individuals working with school age children having disabilities. |
| Institutionalizing Utilization-focused approaches to Evaluation in Non-Governmental Organizations (NGOs): Creating a Culture of Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carrie Willimann, Academy for Educational Development, cwillimann@aed.org |
| Winnehl Tubman, Academy for Educational Development, wtubman@aed.org |
| HyeJin Kim, Academy for Educational Development, hkim@aed.org |
| Arushi Terway, Academy for Educational Development, aterway@aed.org |
| Ana Florez, Academy for Educational Development, aflorez@aed.org |
| Audrey-Marie Moore, Academy for Educational Development, amoore@aed.org |
| Abstract: Evaluation can be a tool for ensuring that education projects provide data and information at the policy level to transfer technical innovation. However, evaluation, as currently structured under many donor funded projects, does not provide the necessary information to inform this process. Nor does it allow for a cohesive approach to draw lessons learned across projects to inform broader policy. Clearing space in the existing political economic landscape so that the substance of sound educational improvement can be implemented is the operational core of what a network of institutions and actors (-reform support infrastructure-) needs to do for reform to succeed. This panel examines how a small evaluation group within an international NGO attempted to clear and fill space for utilization-focused evaluation to flourish in order to create a more cohesive approach to evaluation, build the capacity of staff to implement UFE frameworks, and improve organizational learning across education projects. |
| Stress Influences to Age Crossover in Heavy Alcohol Consumption |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Diana Urbanas, National Institutes of Health, durbanas@mail.nih.gov |
| Abstract: Evaluating racial differences in patterns of alcohol consumption aids the detection and prevention of alcohol abuse and alcoholism. Research comparing alcohol consumption patterns across racial groups reveals that when compared to Whites, rates of heavy alcohol consumption is lower for African Americans during adolescence but in adulthood rates of heavy alcohol use exceeds that of Whites. The factors underlying the shift in alcohol consumption remains a subject of debate. This paper discusses the reciprocal interactions of social and environmental stress, age crossover, and alcohol consumption. |
| Evaluator Learning Curve: Is it a Gamble for Stakeholders? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Caroline Lippy, Georgia State University, clippy1@student.gsu.edu |
| Leanne Valentine, Georiga State University, leannevalentine@comcast.net |
| Jim Emshoff, Georgia State University, psyjge@langate.gsu.edu |
| Beth Anthony, Georgia State University, eanthony1@student.gsu.edu |
| Dary Enkhtor, Georgia State University, denkhtor1@student.gsu.edu |
| Ayana Perkins, Georgia State University, ayanaperkins@msn.com |
| Lindsey Zimmerman, Georgia State University, lindsetzimmerman@gmail.com |
| Abstract: This poster examines a series of research projects funded by the Georgia Department of Human Resources, focusing on a comparison of two prevalence studies of problematic gambling behavior. Serving as needs assessments, the studies, conducted in 1999 and 2007, each completed approximately 1600 randomized telephone interviews of Georgia residents and helped to outline the need for problem gambling services in the state. This poster examines several modifications between the first and second study, indicating an evaluator learning curve. Specifically, evaluators utilized an alternative measure of pathological gambling and also became more aware of race/ethnicity interviewer effects on the sample. While such modifications improved the quality of the evaluation, they potentially induced unintended negative consequences for the state, including a decreased understanding of the effectiveness of previous efforts implementing evaluator recommendations from the first study. The implications and recommendations for future research are made for both evaluators and state officials. |
| Evaluation of Research Program Structures Using Logic Models and Value Stream Management: A Pilot Study |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Delia J Valles-Rosales, New Mexico State University, dvalles@nmsu.edu |
| Elizabeth Schott, New Mexico State University, libby.schott@gmail.com |
| Barbara Lugo, New Mexico State University, blugo@nmsu.edu |
| Abstract: This paper presents an innovative approach to evaluate research program structures at college level. The Alliance for Minority Participation Undergraduate Research Assistantship funded by the National Science Foundation at New Mexico State University has been selected to be the pilot program for this study. The purpose of the study was to evaluate the structure of the program to locate areas for improvement that have not been identified yet by the current evaluation practices in the direction of supporting the development of students that increases the quality of educational experience and motivates them towards graduate studies. The methodology consists of employing logic models combined with value stream management techniques. Our paper discusses the overall process used to conduct the evaluation as well as our initial results and recommendations. |
| What is Success? Considerations From an Evaluation of a First Nations Treatment Centre |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kim Van der Woerd, Simon Fraser University, kvanderwoerd@shaw.ca |
| Abstract: The purpose of this paper is to discuss a participatory evaluation of program outcomes with a First Nations residential (six-week) substance abuse treatment program. This project included a follow-up interview with clients who participated in 17 different six-week sessions over a two and a half year time period (n = 218). All of the program staff was interviewed to develop a follow-up interview that focused on rating the program components and staff, abstinence and relapses, support after treatment, and overall wellbeing. A total 91 clients were interviewed by telephone, representing 42% of the potential clients for interview. The interviews took an average of thirty minutes. This paper will discuss methodology and follow-up interview findings, including a discussion of what constitutes a successful outcome in addictions treatment, and what factors predict outcomes. The long term goal is to develop an effective information management system as well as a permanent follow-up program. |
| How Learning About Evaluation Can Better Support Teaching and Learning in a Postsecondary Environment |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stanley Varnhagen, University of Alberta, stanley.varnhagen@ualberta.ca |
| Brad Arkison, University of Alberta, bradley.arkison@ualberta.ca |
| Jason Daniels, University of Alberta, jason.daniels@ualberta.ca |
| Abstract: Although evaluation is common in post-secondary environments, it is often limited to being summative and judgmental in nature, which restricts its effectiveness as a tool to enhance teaching and learning. In the post-secondary environment, and in a broader scholarship context, it is necessary for academics and researchers to better understand evaluation and applied research methods. In order to enhance the effectiveness of evaluation at the post-secondary level, academics and researchers need to understand the discipline of formative evaluation, its tools, and potential benefits. In addition, the evaluative process needs to be safe from external judgment and comparisons. This presentation examines some current evaluative projects that allow various users of innovative instructional approaches to learn evaluative methods, to receive formative feedback in a safe environment, and to participate actively in the evaluation process. |
| Influences on the Career Paths of Underrepresented Minority Students in a Biology Enrichment Program |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jennifer Sweeney, University of California, Davis, jksweeney@ucdavis.edu |
| Merna Villarejo, University of California, Davis, mrvillarejo@ucdavis.edu |
| Amy Barlow, University of California, Davis, aebarlow@ucdavis.edu |
| Abstract: A two-stage survey/interview evaluation design was used to test the premise that science enrichment activities, particularly participation in undergraduate research, encourage underrepresented minority students to pursue biology-based professions. Alumni who participated in an educational enrichment program at the University of California, Davis responded to a survey and a follow-up telephone interview regarding influences on their career choices. The survey and interview complement a statistical study associating undergraduate research participation and positive graduation outcomes including graduation in a biology major. Survey respondents (n=202) detailed undergraduate experiences and career choices. Over 100 interviewees reflected on the contribution of pre-college, undergraduate, and post-graduate experiences to their subsequent career choices in the CATI follow-up interview. Preliminary analysis suggests that undergraduate research experience affects career choice by providing an opportunity for students to discover, refine, or confirm their interests. The value of this experience appears to persist regardless of whether students pursue a research career. |
| HIV Prevention Among High School Students: Evaluation of a Peer Educator Project |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Manuel Voelkle, University of Mannheim, voelkle@rumms.uni-mannheim.de |
| David Dwertmann, University of Mannheim, ddwertma@rumms.uni-mannheim.de |
| Abstract: The poster provides an overview of the evaluation of a community based Peer Educator Program for HIV prevention among high school students. Approximately N = 500 teenagers from three different schools participated in the project. A quasi-experimental pre-post test design with a three month follow-up questionnaire was chosen to assess changes in knowledge, attitude and reported prevention behavior. An innovative combination of Ajzen's (1991) 'Theory of Planned Behavior' and the 'Information-Motivation-Behavioral-Skills' model, proposed by Fisher & Fisher (1992), constitutes the theoretical foundation of the study. The poster addresses the study-design, implementation of the program, assessment of data and results. A special emphasis is put on the psychometric quality of a newly developed questionnaire and the suitability of the underlying theoretical model. We conclude with some recommendations for model improvement. |
| An Evaluation of Techniques for Recruiting Parents into HIV Prevention Research in Rural Communities: Teen Recruitment of Adults Versus Traditional Methods: What Works Best? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Josephine Allen, Cornell University, jaa7@cornell.edu |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Jennifer Tiffany, Cornell University, jst5@cornell.edu |
| Deladem Kusi-Appouh, Cornell University, dnk7@cornell.edu |
| Wendy Vonhof, Cornell University, wmv2@cornell.edu |
| Abstract: Reducing adolescent sexual risk-taking behaviors is one of the leading national priorities under the “Healthy People 2010” initiative, and adolescents in rural areas are particularly at high risk of acquiring STIs, including HIV. Adolescents reporting positive communication with parents are less likely to engage in sexual risk-taking behaviors. Yet, engaging teenagers and their parents in HIV prevention interventions remains a challenge. We report our experience with recruiting parents or guardians into an HIV prevention research project through teenage participants who had been engaged using a Participant-Driven Recruitment approach. Our results indicate that recruitment of adults into HIV preventive interventions may work better when done using traditional techniques, rather than by asking adolescents to recruit adults. The reasons and applications to other interventions and evaluations will be discussed. |
| Implementation of Evidence-based Interventions in Community Mental Health Settings: Organizational Factors and Provider Perspectives Impacting Implementation Processes and Child and Family Outcomes |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Elizabeth Douglas, Macro International Inc, elizabeth.b.douglas@orcmacro.com |
| Christine Walrath, Macro International Inc, christine.m.walrath-greene@orcmacro.com |
| Bhuvana Sukumar, Macro International Inc, bhuvana.sukumar@orcmacro.com |
| Abstract: A growing body of knowledge exists about evidence-based practices (EBPs) that work to improve outcomes for children with mental health needs. However, as much of this research is conducted in tightly controlled settings (Burns & Hoagwood, 2002), it often reflects intervention efficacy while leaving questions about effectiveness in diverse "real-world" settings. Moreover, reports indicate that even in the case of interventions found to be effective through evaluation in clinical settings, the protocols are not widely understood by clinicians, and are not being translated into practice nearly often enough (NIMH, 2000). As part of an effort to contribute to the knowledge base regarding EBP implementation and effectiveness, this symposium will present findings related to organizational factors and provider perspectives impacting implementation as well as the results of a study examining the effectiveness of Parent-Child Interaction Therapy in community mental health settings. Implications for future implementation and effectiveness research will be discussed. |
| Classroom-level Influences on Literacy Growth Among Early Reading First Students |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sheridan Green, JVA Consulting LLC, sheridan@jvaconsulting.com |
| Susan Hutchinson, University of Northern Colorado, susan.hutchinson@unco.edu |
| Ana Leon, JVA Consulting LLC, ana@jvaconsulting.com |
| Jane Walsh, Denver Public Schools Early Reading First, jane_walsh@dpsk12.org |
| Abstract: The Denver Public Schools Early Reading First program utilizes a literacy-focused curriculum and a substantive professional development component for program teachers. Approximately 180 students are assessed three times annually to gauge improvements in early literacy. This paper will present the results of examining the effects of classroom characteristics on student literacy. Because these effects are nested within schools, hierarchical linear modeling will be used for analysis. Questions addressed: 1) what is the influence of classroom-level factors on literacy growth? And 2) do classroom-level factors differ in their effects on literacy scores? Study data comprise four categories: measures of early literacy skills, measures of classroom teacher qualifications and characteristics, measures of the classroom literacy environment, and measures of student, teacher, and school characteristics that we will employ as analytical controls. Dependent variables include receptive vocabulary and phonological awareness using the Peabody Picture Vocabulary Test-III and the Phonological Awareness and Literacy Screening. |
| Evaluating an Intensive Outpatient Treatment Program for Parolees With Mental Health and Substance Abuse Disorders |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pamela Walsh, Eastern Michigan University, pwalsh@emich.edu |
| David Atkinson, Community Care Services, ripsman1214@yahoo.com |
| Abstract: This is a collaborative project between a community mental health and substance abuse services agency located in Lincoln Park, MI and an university in evaluating the agency's' M-COIT Program, which is an intensive, multi-disciplinary outpatient mental health and substance abuse treatment program for parolees who are mentally ill and/or have substance disorders. M-COIT's goal is to assist parolees' in adjusting to community living and to live a crime-free life style. The premise is that parolees who comply with treatment will have lower recidivism rates, reduced inpatient psychiatric hospitalizations and will comply with parole conditions. Empowerment evaluation is the evaluation method using client record review of parolees who have been discharged from the M-COIT Program from 2001-2005 to identify their recidivism and inpatient psychiatric hospitalization rates and to identify factors (e.g., type of interventions, length of interventions, presence/absence of social supports) that have been significant in affecting recidivism and hospitalization rates. |
| Evaluation of Business Needs in a Diverse Community |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Dennis Watson, Loyola University, Chicago, dwatso2@luc.edu |
| Akilah Watking-Buttler, Loyola University, Chicago, awatki4@luc.edu |
| Abstract: The primary stakeholder of this evaluation is an organization in Chicago that works to promote diversity within their community through business aid and development. A primary problem voiced by this organization is the difficulty they have recruiting ethnic minorities and getting them involved in general activities within the business community. One reason they feel this may be happening is because these organizations may have other networks that they turn to for aid. Noticing a lot of negative friction between different ethnic groups within the community, this organization would like to find ways that they can increase membership in order to foster harmony within the business community. Using a community survey, this project aims to evaluate business needs and look for possible business networks outside of the chamber in an effort to help our community partner provide services that are better suited to the minority population of business owners. |
| Evaluation of Business Needs in a Diverse Community |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Dennis Watson, Loyola University, Chicago, dwatso2@luc.edu |
| Akilah Watking-Buttler, Loyola University, Chicago, awatki4@luc.edu |
| Abstract: The primary stakeholder of this evaluation is an organization in Chicago that works to promote diversity within their community through business aid and development. A primary problem voiced by this organization is the difficulty they have recruiting ethnic minorities and getting them involved in general activities within the business community. One reason they feel this may be happening is because these organizations may have other networks that they turn to for aid. Noticing a lot of negative friction between different ethnic groups within the community, this organization would like to find ways that they can increase membership in order to foster harmony within the business community. Using a community survey, this project aims to evaluate business needs and look for possible business networks outside of the chamber in an effort to help our community partner provide services that are better suited to the minority population of business owners. |
| Development of a Systematic Evaluation Model for Regional Economic Development Projects in the United States |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brad Watts, Western Michigan University, brad.r.watts@wmich.edu |
| Abstract: Most regions of the U.S. engage in some form of economic development effort in an attempt to improve employment, income, and business conditions that impact area residents. Although the government agencies or nonprofit organizations engaged in these activities are forced to make important choices regarding how best to invest limited resources, there is a surprising lack of established evaluation practice in place within the regional economic development community. One approach is to use economic impact analysis to estimate potential project outcomes ahead of time. This provides a basis for comparing and evaluating economic development projects across industries and regions. Considered in a context with other key economic development values such as project cost, appropriateness of fit, and sustainability, economic impact valuation can be key to effective evaluation and decision making. |
| Measuring Program Fidelity Through Factor Analysis: A Developmental Process |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tara Weatherholt, Education Development Center Inc, tweatherholt@edc.org |
| Abstract: The purpose of this paper is to describe the process of developing and validating a set of instruments designed to assess the implementation fidelity of the eMINTS program as it is scaled up to other states. This development process reflects the joint effort of the evaluators and the eMINTS National Center. For the development process we will test each measure of program fidelity of program implementation using a Confirmatory Factor Analysis (CFA) and a reliability analysis. Forty-six observations will be used for this validation process. Overall, these procedures will further explore the validity of the item groupings (per instrument type), thus confirming or disconfirming our hypothesis regarding how the items relate to each other within each instrument, and how well they tap into the eMINTS core concepts. |
| Use of Stakeholder Focus Groups in the Development of Evaluation Protocols for Sexuality Education Programs |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sharon Wilburn, University of North Florida, swilburn@unf.edu |
| Kenneth Wilburn, University of North Florida, kwilburn@comcast.net |
| Dax Weaver, Health-Tech Consultants Inc, htci@comcast.net |
| Abstract: Stakeholder input is a key ingredient in the development of sustainable community-based interventions and the evaluation designs to gauge these interventions. A common method for collecting this input is the focus group. The appropriate design and implementation of focus groups is an undertaking unto itself; however, much of the evaluation effort is often spent in the subsequent data analysis and application phases. As such it is helpful to use methods that pear down redundant information, while at the same time, retain the integrity of the original data. This paper focuses on the process of quantifying qualitative results from two adult and two adolescent focus groups conducted in two rural counties using an open coding process (Strauss and Corbin, 1990). It also illustrates the importance of including stakeholders' input in program planning and evaluation design of sexuality and abstinence education community-based interventions. |
| Assertive Community Treatment in Atlanta: Exploring Particular Outcomes of a Grady Healthcare System's Approach |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Angela Mooss, Georgia State University, amooss1@student.gsu.edu |
| John Barile, Georgia State University, jbarile1@student.gsu.edu |
| Joanna Weinberg, Georgia State University, jweinberg3@student.gsu.edu |
| Doyanne Darnell, Georgia State University, dhorst1@student.gsu.edu |
| Brandeis Green, Georgia State University, bgreen8@student.gsu.edu |
| Abstract: Extensive research indicates that the Assertive Community Treatment (ACT) model of mental health service delivery is effective in treating persons with severe mental illness. The Atlanta based ACT team (ECSEL) at the Grady Healthcare System employs a “housing first” philosophy which holds that effective service delivery cannot occur without stable housing. Supportive housing for homeless people with severe mental illness has been found to reduce overall service use, incarcerations and hospitalizations (Culhane, et. al. 2001). The ACT model is predicted to positively impact overall factors of wellbeing, such as housing satisfaction, familial relations, employment opportunities, prevalence of psychiatric symptoms, and reductions in substance abuse. This presentation examines the outcomes of the above factors among a homeless sample in Atlanta, and explores the unique challenges and successes of data collection in a variety of settings. Subsequent outcomes and issues of diversity among program participants and between the participants and members of the ACT team will also be addressed regarding fidelity to the original ACT model. |
| Use of Stakeholder Focus Groups in the Development of Evaluation Protocols for Sexuality Education Programs |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sharon Wilburn, University of North Florida, swilburn@unf.edu |
| Kenneth Wilburn, University of North Florida, kwilburn@comcast.net |
| Dax Weaver, Health-Tech Consultants Inc, htci@comcast.net |
| Abstract: Stakeholder input is a key ingredient in the development of sustainable community-based interventions and the evaluation designs to gauge these interventions. A common method for collecting this input is the focus group. The appropriate design and implementation of focus groups is an undertaking unto itself; however, much of the evaluation effort is often spent in the subsequent data analysis and application phases. As such it is helpful to use methods that pear down redundant information, while at the same time, retain the integrity of the original data. This paper focuses on the process of quantifying qualitative results from two adult and two adolescent focus groups conducted in two rural counties using an open coding process (Strauss and Corbin, 1990). It also illustrates the importance of including stakeholders' input in program planning and evaluation design of sexuality and abstinence education community-based interventions. |
| Use of Stakeholder Focus Groups in the Development of Evaluation Protocols for Sexuality Education Programs |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sharon Wilburn, University of North Florida, swilburn@unf.edu |
| Kenneth Wilburn, University of North Florida, kwilburn@comcast.net |
| Dax Weaver, Health-Tech Consultants Inc, htci@comcast.net |
| Abstract: Stakeholder input is a key ingredient in the development of sustainable community-based interventions and the evaluation designs to gauge these interventions. A common method for collecting this input is the focus group. The appropriate design and implementation of focus groups is an undertaking unto itself; however, much of the evaluation effort is often spent in the subsequent data analysis and application phases. As such it is helpful to use methods that pear down redundant information, while at the same time, retain the integrity of the original data. This paper focuses on the process of quantifying qualitative results from two adult and two adolescent focus groups conducted in two rural counties using an open coding process (Strauss and Corbin, 1990). It also illustrates the importance of including stakeholders' input in program planning and evaluation design of sexuality and abstinence education community-based interventions. |
| PET: A Practical Evaluation Tool for (Art) Projects |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Machteld Willemse, Artnotion, info@artnotion.nl |
| Abstract: To start and perform a meaningful evaluation process is not an easy task: choosing between indicators, deciding what is important and what should be left out, what information should be gathered and how, transferring information into knowledge, reporting to interested parties, is just a small sample of the many, many factors and activities that concerns a (formative) evaluation process. Even for experienced evaluators it takes an effort to get the evaluation process rolling: imagine how it will be for a well willing, but not so-experienced (self) evaluator. To get started, to keep a grip on the process, to keep track of one's actions and to keep an overview, Art notion has developed an evaluation instrument (firmly based on practical experience) that provides an accessible, practical and visualized tool to perform a formative (self) evaluation process. |
| HIV/AIDS/STD-related Worry and Sexual Activity Within Adolescent Social Networks |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deladem Kusi-Appouh, Cornell University, dnk7@cornell.edu |
| Josephine Allen, Cornell University, jaa7@cornell.edu |
| Eunice Rodriguez, Stanford University, er23@stanford.edu |
| Jennifer Tiffany, Cornell University, jst5@cornell.edu |
| Lindy Williams, Cornell University, lbw@cornell.edu |
| Abstract: This paper examines the relationship of four personal worry factors to sexual activity among 15-19 year-olds. Adolescents were recruited by their peers through Participant-Driven Recruitment (PDR), a methodology that combines participatory research and Respondent-Driven Sampling (RDS). The social networks illustrated by the PDR/RDS recruitment data enabled us to evaluate the tendency for adolescents to form social ties based on similarity (homophily). The recruiter/recruitee social network was our unit of analysis. The survey instrument included measures of worry about 1) getting HIV/AIDS; 2) becoming pregnant/getting someone pregnant; 3) having sex; and 4) getting a sexually transmitted disease. Worry was measured as a continuous variable: never worried, sometimes worried, often worried and constantly worried. Results suggest that those adolescents who worried about these factors had increased odds of engaging in sexual activity during the month preceding the survey. Color graphics displaying the social networks according to each worry factor complement the analysis. |
| Institutionalizing Utilization-focused approaches to Evaluation in Non-Governmental Organizations (NGOs): Creating a Culture of Evaluation |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carrie Willimann, Academy for Educational Development, cwillimann@aed.org |
| Winnehl Tubman, Academy for Educational Development, wtubman@aed.org |
| HyeJin Kim, Academy for Educational Development, hkim@aed.org |
| Arushi Terway, Academy for Educational Development, aterway@aed.org |
| Ana Florez, Academy for Educational Development, aflorez@aed.org |
| Audrey-Marie Moore, Academy for Educational Development, amoore@aed.org |
| Abstract: Evaluation can be a tool for ensuring that education projects provide data and information at the policy level to transfer technical innovation. However, evaluation, as currently structured under many donor funded projects, does not provide the necessary information to inform this process. Nor does it allow for a cohesive approach to draw lessons learned across projects to inform broader policy. Clearing space in the existing political economic landscape so that the substance of sound educational improvement can be implemented is the operational core of what a network of institutions and actors (-reform support infrastructure-) needs to do for reform to succeed. This panel examines how a small evaluation group within an international NGO attempted to clear and fill space for utilization-focused evaluation to flourish in order to create a more cohesive approach to evaluation, build the capacity of staff to implement UFE frameworks, and improve organizational learning across education projects. |
| Validity of Patient-reported Health-related Quality of Life Global Ratings of Change Using Structural Equation Modeling |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-generated single-item global ratings of change (GRCs) are often used as anchors of health-related quality of life (HRQoL) since they are easy for clinicians to interpret and provide clinicians with change score interpretations that may guide clinical decision-making and assist in determining intervention efficacy. Although this approach may be preferred, the validity of the anchor-based approach is currently under scrutiny. We used structural equation modeling to estimate the explained variation in domain-specific GRCs that is accounted for by Time 1 (T1) and Time 2 (T2) domain-specific summary scores from the Short-Form 36, V2 (SF-36) Health Survey in 356 asthma outpatients. Correlations revealed that domain-specific GRCs were more strongly related to T2 than T1 domain summary scores, indicating that patients were not equally relying on T1 and T2 to generate the GRCs. Furthermore, T1-domain summary scores were not of equal magnitude and opposite sign as compared to T2 scores. There is insufficient evidence to establish SF-36 domain-specific GRC validity in asthma outpatients. |
| A Comparison of Traditional and Rasch Cut Points for Assessing Clinically Important Change in Health-Related Quality of Life Among Patients with Asthma |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-perceived change in health-related quality of life (HRQoL) domains has often been classified using a 15-point patient transition rating scale with the arbitrarily-derived change levels of minimal (2, 3 or -2, -3), moderate (4, 5 or -4, -5) and large (6, 7 or -6, -7). Therefore, in order to determine the adequacy of these traditional change levels, item response theory methodology was employed to derive empirically-based cut points for the change levels. Our sample included 396 asthmatic outpatients who completed bimonthly telephone interviews on the Asthma Quality of Life Questionnaire and transition rating items over one year. After computing Rasch partial credit model transition rating cut points, we compared the mean domain-specific HRQoL changes from Time 1 to Time 2 under each traditional transition rating change level with those under Rasch-derived change levels. Although traditional and Rasch categorizations for small, moderate, and large changes slightly differed, nearly all mean changes between classification approaches were comparable. Traditional transition rating cut points remain suitable to assess HRQoL clinical significance in outpatients with asthma. |
| "Apples to Apples": The Experience of Developing an Evaluation Tool for the Workforce Development Performance Benchmarking Project |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacy Woodruff-Bolte, Public/Private Ventures, swoodruff-bolte@ppv.org |
| Martha Miles, Training Inc, mmiles@traininginc.org |
| Carol Clymer, Public/Private Ventures, cclymer@ppv.org |
| Abstract: The Workforce Development Performance Benchmarks Project represents an attempt to determine the feasibility of identifying standards of performance for workforce development organizations and use the benchmarks to improve performance. Following the second wave of survey data collection from workforce organizations across the country serving TANF recipients, lessons related to creating a performance measurement evaluation tool are emerging. Using methods to account for differences among employment and training organizations, we will discuss the development of a much needed evaluation tool in the field for workforce development. |
| A Comparison of Traditional and Rasch Cut Points for Assessing Clinically Important Change in Health-Related Quality of Life Among Patients with Asthma |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-perceived change in health-related quality of life (HRQoL) domains has often been classified using a 15-point patient transition rating scale with the arbitrarily-derived change levels of minimal (2, 3 or -2, -3), moderate (4, 5 or -4, -5) and large (6, 7 or -6, -7). Therefore, in order to determine the adequacy of these traditional change levels, item response theory methodology was employed to derive empirically-based cut points for the change levels. Our sample included 396 asthmatic outpatients who completed bimonthly telephone interviews on the Asthma Quality of Life Questionnaire and transition rating items over one year. After computing Rasch partial credit model transition rating cut points, we compared the mean domain-specific HRQoL changes from Time 1 to Time 2 under each traditional transition rating change level with those under Rasch-derived change levels. Although traditional and Rasch categorizations for small, moderate, and large changes slightly differed, nearly all mean changes between classification approaches were comparable. Traditional transition rating cut points remain suitable to assess HRQoL clinical significance in outpatients with asthma. |
| Validity of Patient-reported Health-related Quality of Life Global Ratings of Change Using Structural Equation Modeling |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacie Metz, West Chester University, smetz@wcupa.edu |
| Kathleen Wyrwich, Saint Louis University, wyrwichk@slu.edu |
| Ajit Babu, Amrita Institute of Medical Sciences, ajitbabu@aimshospital.org |
| Kurt Kroenke, Regenstrief Institute, kkroenke@regenstrief.org |
| William Tierney, Regenstrief Institute, wtierney@iupui.edu |
| Frederic Wolinsky, University of Iowa, frederic-wolinsky@uiowa.edu |
| Abstract: Patient-generated single-item global ratings of change (GRCs) are often used as anchors of health-related quality of life (HRQoL) since they are easy for clinicians to interpret and provide clinicians with change score interpretations that may guide clinical decision-making and assist in determining intervention efficacy. Although this approach may be preferred, the validity of the anchor-based approach is currently under scrutiny. We used structural equation modeling to estimate the explained variation in domain-specific GRCs that is accounted for by Time 1 (T1) and Time 2 (T2) domain-specific summary scores from the Short-Form 36, V2 (SF-36) Health Survey in 356 asthma outpatients. Correlations revealed that domain-specific GRCs were more strongly related to T2 than T1 domain summary scores, indicating that patients were not equally relying on T1 and T2 to generate the GRCs. Furthermore, T1-domain summary scores were not of equal magnitude and opposite sign as compared to T2 scores. There is insufficient evidence to establish SF-36 domain-specific GRC validity in asthma outpatients. |
| Case Studies in Transformative Learning: Interviews, Assessments, and Evidence Used to Evaluate “Ready Schools” |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paula Plonski, Praxis Research Inc, pmplonski@carolina.rr.com |
| Bruce Yelton, Praxis Research Inc, praxis1@att.net |
| Abstract: The concept of “ready schools” grew out of the national concern for children's school readiness. However, helping children to be ready for school is only part of the battle; schools also need to be ready to receive, nurture, and support these “ready children”. As part of the evaluation of the implementation of a W.K. Kellogg SPARK grant in North Carolina, a case study methodology was utilized to assess the readiness of two exemplary schools in the region. The evaluators conducted interviews with teachers, administrators, and parents; observed the administration of High/Scope's Ready School Assessment; and considered documents and other evidence gathered by school personnel. As a result of this process, the evaluators and the Ready School Team simultaneously uncovered the school's level of readiness. |
| Evaluating Mental Health Programs for Children Based on the System of Care Model: A Critical Review |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Mijung Yoon, University of Illinois at Urbana-Champaign, myoon1@uiuc.edu |
| Abstract: This paper critically reviews the evaluations of mental health programs for children that use the System of Care approach. This paper intends to contribute to evaluation theory and practice of mental health programs for children by offering an understanding of evaluation practices of such programs. Specifically, this review focuses on how evaluations have been conducted in response to contextual factors and unique characteristics of such programs. First briefly, this review analytically describes the characteristics of the programs that were evaluated. Second, it analytically describes the characteristics of the evaluations conducted on the programs. Third, it provides a critique of the evaluations. |
| Challenges in Program Evaluation: Lessons Learned From Evaluating Child Maltreatment Prevention Interventions |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lei Zhang, Georgia State University, alhlzx@langate.gsu.edu |
| Angela Snyder, Georgia State University, alhabs@langate.gsu.edu |
| Karen Minyard, Georgia State University, alhkjm@langate.gsu.edu |
| Abstract: Child maltreatment program evaluation raises several ethical/technical issues that have yet to be addressed by evaluation research. A real-world example is Georgia Health Policy Center's evaluation of the Intensive Home Visiting (IHV) program funded by Georgia Division of Public Health. The IHV program features trained, supervised public health nurses that will provide frequent and intense home visits with families of young children identified as at-risk for abuse and neglect. The nurses will use standardized tools and curricula to teach parents about child development, health, and safety. The major ethical/technical issues involved in this evaluation include (1) obtaining sample sizes large enough to detect statistically significant impacts; (2) maintaining victim safety; and (3) tracking participants in a way that minimizes privacy violation. Our feedback from the evaluation indicates (1) evaluation needs to be sensitive to its contexts; and (2) ethical guidance provided by the Joint Committee needs to be more specific. |
| Evaluator Learning Curve: Is it a Gamble for Stakeholders? |
| Poster Presentation 131 to be held in International Ballroom on Wednesday, November 7, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Caroline Lippy, Georgia State University, clippy1@student.gsu.edu |
| Leanne Valentine, Georiga State University, leannevalentine@comcast.net |
| Jim Emshoff, Georgia State University, psyjge@langate.gsu.edu |
| Beth Anthony, Georgia State University, eanthony1@student.gsu.edu |
| Dary Enkhtor, Georgia State University, denkhtor1@student.gsu.edu |
| Ayana Perkins, Georgia State University, ayanaperkins@msn.com |
| Lindsey Zimmerman, Georgia State University, lindsetzimmerman@gmail.com |
| Abstract: This poster examines a series of research projects funded by the Georgia Department of Human Resources, focusing on a comparison of two prevalence studies of problematic gambling behavior. Serving as needs assessments, the studies, conducted in 1999 and 2007, each completed approximately 1600 randomized telephone interviews of Georgia residents and helped to outline the need for problem gambling services in the state. This poster examines several modifications between the first and second study, indicating an evaluator learning curve. Specifically, evaluators utilized an alternative measure of pathological gambling and also became more aware of race/ethnicity interviewer effects on the sample. While such modifications improved the quality of the evaluation, they potentially induced unintended negative consequences for the state, including a decreased understanding of the effectiveness of previous efforts implementing evaluator recommendations from the first study. The implications and recommendations for future research are made for both evaluators and state officials. |