| Evaluation of Research Center and Network Programs at the National Institutes of Health (NIH): An Historical Review and Illustrative Case Example |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jack Scott, The Madrillon Group Inc, jscott@madrillongroup.com |
| Annie Feng, National Institutes of Health, fengx3@mail.nih.gov |
| Kara Hall, National Institutes of Health, hallka@mail.nih.gov |
| Margaret Blasinsky, The Madrillon Group Inc, mblasinsky@madrillongroup.com |
| Brooke Stipelman, National Institutes of Health, stipelmanba@mail.nih.gov |
| Kara Hall, National Institutes of Health, hallka@mail.nih.gov |
| Amanda Vogel, National Institutes of Health, amanda.vogel@nih.gov |
| Dan Stokols, University of California, Irvine, dstokols@uci.edu |
| Abstract: Over the past 30 years federal investments in research center and network grant programs have continued to increase, with approximately ten percent of the current annual research budget of the National Institutes of Health (NIH) allocated toward funding these types of grants. With these growing investments, there is an increasing need to evaluate the scientific and societal merit of these programs. This posters presents findings from a comprehensive review of evaluations of NIH funded research center and network programs. This review explores historical patterns in center- and network-grant evaluations, and extracts lessons learned and recommendations for future evaluations. We then present findings from an evaluation of a large center initiative funded by the National Cancer Institute (NCI) that is currently underway. This case example illustrates innovative methods currently being used at NIH to evaluate the impact of center and network grants. |
| Strategies for Quantative Analysis of Student and Teacher Longitudinal Outcomes in K-12 Settings |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Amy Overbay, North Carolina State University, amy_overbay@ncsu.edu |
| Melinda Mollette, North Carolina State University, melinda_mollette@ncsu.edu |
| Shevaun Neupert, North Carolina State University, shevaun_neupert@ncsu.edu |
| Abstract: This session will present five empirically-validated surveys used to evaluate changes in student & teacher technology skills, instructional activities, & teacher perceptions of professional development. This will include information on how the survey data was quantitatively summarized & reported. Further, presenters will discuss the use of hierarchical linear modeling & logistic regression (odds ratios) to analyze longitudinal student achievement data. Presenters will demonstrate how these quantitative variables were used to measure student/ teacher outcomes over time, as well as ways to incorporate different types of variables with student achievement data. (e.g. scale scores, performance levels) Presenters will also focus on the methodological challenges commonly experienced during K-12 evaluations, including issues related to conducting evaluations across multiple school levels, administering surveys repeatedly over three years, & use of student achievement data when the test is “revised” during the evaluation period. The discussion will highlight strategies used to overcome these challenges. |
| Using Network Analysis in the Evaluation of Community Partnership Building |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Heather Clark, Texas A&M University, hrclark@srph.tamhsc.edu |
| Monica Wendel, Texas A&M University, mlwendel@srph.tamhsc.edu |
| Corliss Outley, Texas A&M University, outley@tamu.edu |
| Abstract: Examining the nature of social relationships in community partnerships can reveal whether, how, and the frequency with which different sectors of the community interface and interact with each other; assessing changes in these relationships over time yields insight into the cooperation among organizations that may contribute to overall capacity of the partnership. In the case of the Brazos Valley Health Partnership, a local collaborative effort in the rural Central Texas, understanding the ways and depths to which organizations are interacting and how those ties change in response to the context of activities in the community is important in facilitating continued efforts to build local capacity for health improvement. |
| Evaluating Nursing Education at the Program Level |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Forneris, St Catherine University, sgforneris@stkate.edu |
| Suellen Campbell, St Catherine University, secampbell@stkate.edu |
| Vicki Schug, St Catherine University, vlschug@stkate.edu |
| Linda Blazovich, St Catherine University, lmblazovich@stkate.edu |
| Suzanne Lehman, St Catherine University, smlehman@stkate.edu |
| Susan Forneris, St Catherine University, sgforneris@stkate.edu |
| Linda Blazovich, St Catherine University, lmblazovich@stkate.edu |
| Abstract: The purposes of this presentation are to 1) share a curricular blueprint that operationalizes the use of national standards as formative/summative evaluation for nurse educators; 2) articulate how these standards guide the evaluation of a nursing program of study with intentional mapping of clinical/classroom-to-course objectives, course-to-level objectives and level to program outcomes and national benchmark standards; and 3) demonstrate the use of these evaluative measures in a course focused on clinical simulation. |
| Using Technology to Improve Collaboration and Stakeholder Communication |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anna de Guzman, University of Denver, anna.deguzman@du.edu |
| Laricia Longworth-Reed, University of Denver, laricia.longworth-reed@du.edu |
| Kathryn Schroeder, University of Denver, kathryn.schroeder@du.edu |
| Abstract: This poster will examine different technologies available to improve collaboration among evaluation partners and to improve extensive data collection efforts, specifically looking at online workspaces and web-based survey tools. We will discuss the free and paid options available and any concerns raised with each option. |
| The DoView Generic Outcomes Model for a Country: A Generic Outcomes Model From Which You Can Borrow Parts to Quickly Build Your Own Outcomes Models (Logic Models) |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paul Duignan, Parker Duignan Consulting, paul@parkerduignan.com |
| Abstract: At any moment on any day, as an evaluator draws a logic model for a particular project, other evaluators in other parts of the world will be trying to draw similar models. This is an inefficient approach. The web-based DoView Generic Outcomes Model for a Country is an attempt to make available a comprehensive set of logic models (outcomes models) from which anyone can quickly borrow just the parts they require for their own models. They can immediately download the parts they want and immediately start amending the model for their own project in DoView outcomes and evaluation software. The logic models developed within the Model are all being built according to a specific set of standards for developing such models to ensure that they can be used for a range of different evaluation and other purposes. How to use the DoView Generic Outcomes Model will be discussed. See http://www.tinyurl.com/otheory274. |
| How to Quickly Assess the Quality of Any Performance Indicator List |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paul Duignan, Parker Duignan Consulting, paul@parkerduignan.com |
| Abstract: Evaluators often need to quickly assess the quality of a performance indicator list. Currently this is done by scanning the list and trying to work out if there are any important measures not included. This requires that evaluators hold their minds both on the list of indicators being assessed and also on a mental model of what the project or organization being measured is trying to do. A much more efficient way of working is to build a visual outcomes model of what it is that the project or organization is trying to do and to visually map onto this the current list of performance indicators. Gaps and overlaps in the indicators currently being collected are immediately revealed. Experience with this approach has shown that many people find this a much easier way of assessing the quality of a set of performance indicators than the traditional approach. See http://www.tinyurl.com/otheory271. |
| Best Practices for Web-based Surveys |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deborah K Lewis, Ohio State University, lewis.205@osu.edu |
| Abstract: The telephone, random sampling, and web-based surveys are probably the three most significant advances in survey technology in the twentieth century (Don Dillman, 2000). However, there is much to learn about how to create and manage web-based surveys. The research pertaining to web-based surveys is still in its infancy, but there are usable guidelines emerging which will be discussed and illustrated. This demonstration will share the latest approaches to conducting surveys through web-based questionnaires. Advantages and limitations of web-based questionnaires as compared to traditional paper, mailed questionnaires will be highlighted. Content of the session will include guidelines for construction and practical tips for application of web-based survey procedures to academic research and evaluation settings. Some commercially available, popular, web based survey products will be compared. This session will highlight the best practices currently available to evaluators who plan to use web-based surveys, with their own special issues, in gathering data. |
| Evaluation in Action Skill-building: Tracking and Timing With the Visitor Studies Association |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Cheryl Kessler, Independent Consultant, kessler.cheryl@gmail.com |
| Joe E Heimlich, Ohio State University, heimlich.1@osu.edu |
| Carey Tisdal, Tisdal Consulting, ctisdal@sbcglobal.net |
| Cláudia Figueiredo, Institute for Learning Innovation, figueiredo@ilinet.org |
| Kathleen Tinworth, Denver Museum of Nature & Science, kathleen.tinworth@dmns.org |
| Abstract: Following a successful and well-attended panel session in 2009, members of the Visitor Studies Association (VSA) will return to AEA to facilitate a skill-building session for AEA attendees on tracking and timing methodologies. We'll conduct this session during the AEA posters/marketplace. While not necessary to attend both sessions, this skill-building session complements a panel session where several tracking and timing methodologies will be presented as case studies. Participants will have the opportunity to both hear case studies where tracking and timing was utilized in a visitor studies setting as well as experiment first-hand with collecting and analyzing the resulting data and applying it to their own work. |
| Educator Sexual Abuse in Virginia: A Policy Study |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Billie-Jo Grant, University of Virginia, billiejogrant@gmail.com |
| Abstract: This paper investigates with an in depth qualitative approach how state K-12 educator sexual abuse polices are defined and implemented by a variety of public school participants in the State of Virginia. Research suggests that such policies may not be implemented or practiced by educational actors and may be contributing to the continued prevalence of educator sexual abuse (Shakeshaft, 2004). Drawing on interviews with 40 educational actors (State board of education members, superintendents, district employees, and law enforcement) and court and district document collection, this project examines: (1) policy implementation and understanding, (2) the intergovernmental policy system, (3) differences in reporting and prevalence rates, and (4) evaluation methods. The author argues that by understanding how state policies are defined and implemented by a variety of public school participants this study may be able to determine if policies are substantive or symbolic in nature. |
| The Impact of Food Supplementation on Infant Weight Gain in Rural Bangladesh: An Assessment of the Bangladesh Integrated Nutritional Program (BINP) |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Begum Housne, University of Dhaka, drhousne@gmail.com |
| Mascie Taylor, University of Cambridge, United Kingdom, nmt1@cam.ac.uk |
| Abstract: Examine the efficiency of the Bangladesh Integrated Nutritional Program (BINP)and whether food supplementation of infants resulted in enhanced weight gain. Of 526 infants, over half of the sample was incorrectly identified for supplementation. In addition <1/4 of the infants received full 90 days of supplementation and close to half the infants exited the program without the requisite weight gain. Infants were assigned to four groups. This classification provided natural controls; the correctly supplemented infants vs the incorrectly non-supplemented infants, and the correctly non-supplemented infants vs the incorrectly supplemented infants. There were no significant differences in weight gain between the correctly supplemented group and the incorrectly non-supplemented group or between the correctly non-supplemented and the incorrectly supplemented groups nor was any evidence of growth faltering in the incorrectly non-supplemented group. This study found serious programmatic deficiencies. There was no evidence that food supplementation had any impact on improving infant weight gain. |
| Evaluation and Organizational Learning and Use |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fahad Khan, Catholic Relief Services, findfahadkhan@yahoo.com |
| Abstract: Catholic Relief Services – Pakistan Program is working in Pakistan since 1954 and contributed very much in the Emergencies of Earthquake in 2005, floods in 2007, Earthquake of 2008 and IDPs crises in 2009. CRS is a learning organization and now CRS Pakistan is in the developing stage of making a Monitoring and Evaluation country team, also our regional office have appreciated the efforts and contribution by the Pakistan country office towards Monitoring and Evaluation. I will share the experiences from our last year successful IDPs emergency response project and share the results of its impact evaluation, in the shape of paper, with the diverse group of workshop participants. The paper will be covering the following; 1. Background of the project 2. Project objectives 3. Use of accountability initiatives (Sphere minimum standards) 4. Impact Evaluation of the project 5. Learning’s from the Impact Evaluation 6. Use and incorporation of learning’s in future programming |
| The Matthew Effect: Enhancing Internal Evaluation Credibility Using a External Regulatory Instrument |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| William Cabin, Richard Stockton College, williamcabin@yahoo.com |
| Abstract: The Matthew Effect was coined by sociologist Robert Merton to capture the concept that enhanced credibility of a source, for whatever reason, increased the likelihood of success, regardless of the content of work. The paper applies the concept to use of the New York State OASAS audit tool to structire internal auditing at a non-profit agency for ex-offenders/formerly-incarcerated persons with co-morbid substance abuse and mental health conditions. The paper presents the tool and its adaptation to an ongoing, longitudinal, comparative process for both program evaluation and evaluation of corporate and regulatory responsiveness to the audits. |
| Strategies and Principles Which Guide the Work of the Staff in Branco Weiss Schools for Youth at Risk: What We Have Learned From Our Successes and Identifying Future Challenges? |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chen Lifshitz, Branco Weiss Institute, chenl@brancoweiss.org.il |
| Michal Yaacov, Branco Weiss Institute, michal.aharoni@mail.huji.ac.il |
| Abstract: The creation of normative educational frameworks for youth who have dropped out is particularly challenging. Branco Weiss schools for youth at risk absorb youth who have dropped out of the regular education system and are from weak socio-economic backgrounds. Our system employs ongoing evaluation; outcomes and outputs highlight a high rate of satisfaction on the part of the pupils, similar reporting by pupils and teachers, but that discipline is still an issue. However, evaluation methodologies to date have done little to examine the strategies used by the schools. During the 2009/10 school year our evaluation methodology has expanded to include case studies which examine the models being applied in 4 of the 10 schools for youth at risk in the network. Preliminary findings highlight critical issues in the methodologies being applied such as the critical role/image played /projected by the principal, and the dichotomy regarding pupil support vs. teacher support. |
| Lessons Learned in the School Evaluations: An Evaluation of a Counter-Marketing Student-Led Campaign to Decrease Sedentary Behavior and Fast Food Intake Among Forth and Fifth Grade Students |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Mary Martinasek, University of South Florida, mmartina@health.usf.edu |
| Karen Perrin, University of South Florida, kperrin@health.usf.edu |
| Marisa Mowat, St Joseph's Children's Advocacy Center, marisa.mowat@baycare.org |
| Melanie Hall, St Joseph's Children's Advocacy Center, melanie.hall@baycare.org |
| Dewey Carouthers, 180 Degree Change, dewey@deweyandassociates.com |
| Abstract: Given that 17.4% of children 2 to 19 years of age are overweight, our youth are a targeted population for interventions and strategies that seek to curb the obesity epidemic. Designing programs that fit within the curriculum of schools combined with the formative evaluation serves to be a challenge for public health practitioners. This research describes the lessons learned from an evaluation of an innovative counter-marketing campaign aimed to increase physical activity and decrease consumption of fast foods and junk foods among 2500 4th and 5th grade students. The evaluation included 16 principals, 118 teachers, 1114 primary caregivers, along with the 2500 students representing 16 schools in two Florida counties. A written 32 item survey was developed based on the research objectives . The surveys for the children were administered on computers in the media centers through Survey Monkey, an online survey administration website, during normal school hours. A proctor was stationed at each computer during survey administration to help children with accessing the survey. Teachers and principals were able to access the surveys through a link that was emailed directly to them. The parents / guardians were delivered a paper copy of the survey with the children’s after-school paperwork. Based on feedback from survey proctors and evaluation results, survey items were changed from Year 1 to Year 2 to better assess the research questions and reach a better understanding among youth. Pretesting survey instruments is a necessary component in ensuring compliance and understanding of items among youth. |
| The Impact of Political Factors on the Evaluation Quality: Lessons Learnt by a Local Non-government Organization in the Developing Country |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jhapendra Baidhya, Community Impacts, jhapendra.baidhya@gmail.com |
| Abstract: Most evaluations reported in the literature seem to have ignored the impact of political factors on evaluation quality while overstating the limitations in resources. This paper presents the case of a local non-government organization in Nepal where the political factors are so strong that they can easily undermine the evaluation quality. This paper first reviews the literature on evaluation quality in the developing countries and identifies the most relevant political factors. Based on the results, a research model is developed and discussed in the context of description of experience of a local non-government organization. Lessons learnt from such experiences are then documented, which can be useful to other developing countries facing with similar situation. As politics is a sensitive matter, this paper does not refer to any particular political party, rather it discusses the general trends and inherent factors which play a greater role to influence the independent evaluation. |
| Methods of Monitoring and Evaluation of Community Projects |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ngole Masango Eugene, Non-governmental Organization (NGO), ruwdeo@yahoo.com |
| Abstract: This paper discusses methods and strategies used for monitoring and evaluation of community project, funded by donor agencies with the beneficiaries ranging from government agencies to grass root level communities. It also gives a clear definition of “monitoring and evaluation” M& E of community project or program. DEFINATION: Monitoring; is a continuing function that uses systematic collection of data on specified indicators to provide management and the stakeholders of an ongoing development intervention with indicators of progress and achievement of objectives and progress in the use of allocated funds. Evaluation; is the systematic assessment of an ongoing or completed project, program or policy, its design implementation and results. (OECD 2002) The aim of evaluation is to determine the relevance and fulfillment of objectives develop efficiency, effectiveness, impact and sustainability. When you have set the planned objectives and outcomes of your project, you can consider the best way of evaluation. |
| Best Practices of Quantitative Methodology in Personnel Evaluation |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lihshing Wang, University of Cincinnati, leigh.wang@uc.edu |
| Abstract: Personnel evaluation is a highly contested area due to the complex nature of the latent constructs and the dynamic interplay of the embedded systems. When psychometrics meets politics, chaotic methods are often applied to tackle the evaluation problem. This paper proposes best practices of quantitative methodology in personnel evaluation involving portfolio and performance assessment. Using a recent faculty search in higher education as a case study, this study demonstrates how quantitative methodology can be systematically integrated into the evaluation process to maximize the rigor and defensibility of evidence-driven decision making. Specific issues addressed include domain specification and behavioral sampling, analytical scaling and item weighting, inter-rater reliability and rater calibration. The data matrix was analyzed by the Many-Facet Rasch Measurement software to demonstrate the feasibility and desirability of adjusting for rater idiosyncrasies. Implications of model robustness and generalizability to other complex evaluations such as program certification are discussed. |
| Dimensions in Self-Assessment Research |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tony Lam, University of Toronto, tonycm.lam@utoronto.ca |
| Abstract: Currently, educators have been using self-assessment as a tool to promote student self-directed learning, and evaluators have been relying on self-assessment to gauge intervention success. This popular use of self-assessment has led to extensive research on both the usage and validity of self-assessments. Unfortunately, results in self-assessment research are moderated by numerous variables inherent in either the nature of self-assessment or the research context. Consequently, findings from these research studies are either inconsistent or difficult to generalize. This paper aims to promote the consolidation of self-assessment research findings by presenting a number of dimensions of self-assessment that researchers can refer to when describing their research designs and reporting their findings. These dimensions, which act as moderating variables in self-assessment research, include functions of self-assessment, domains of self-assessment, levels of self-assessment, task specificity of self-assessment, self-assessment activities, time of self-assessment, unit of measurement, and self-assessment accuracy indices. |
| Heroin Dependent Offenders’ Assessment of Buprenorphine Versus Methadone Treatment in Jail |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ezechukwu Awgu, Western Michigan University, eawgu@yahoo.com |
| Stephen Magura, Western Michigan University, stephen.magura@wmich.edu |
| Andrew Rosenblum, National Development and Research Institutes (NDRI), rosenblum@ndri.org |
| Abstract: The study determined how similar offenders treated in jail with either buprenorphine or methadone experienced and assessed those treatments. The parent study voluntarily randomly assigned male heroin-dependent inmates at the Rikers Island jail in NYC to maintenance with methadone (N=56), the standard of care, or buprenorphine (SuboxoneR) (n=60). Methadone patients were more likely to report feeling uncomfortable the first few days, having side/withdrawal effects during treatment, disliking the meth treatment delivery process in jail and being concerned about continued dependency on medication after release. All buprenorphine patients would recommend buprenorphine to others, with almost all preferring it to methadone because of its perceived lesser addictive potential, fewer side effects, faster treatment effect and availability of prescriptions. Ninety-three percent of buprenorphine vs. 44 percent of methadone patients intended to enroll in those respective treatments after release. The results support increasing access to buprenorphine treatment in the community for heroin-dependent offenders. |
| An Evaluation Coaching Model That Encourages Prolonged Stakeholder Involvement |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tania Rempert, University of Illinois at Urbana-Champaign, trempert@illinois.edu |
| Abstract: Several examples in the literature that refer to the practice of empowerment evaluation identify how stakeholders lose interest in the evaluation and the evaluator takes on the bulk of the evaluation design and conducting most of the evaluation (e.g. Sullins, 2003). This presentation describes a unique evaluation coaching model that has been ongoing for four years and maintained the interest of stakeholders to take on the bulk of the evaluation design and implementation. This coaching model includes an external evaluation consultant providing a monthly evaluation seminar on a mutually agreed upon topic for the organization, followed by one-on-one meetings between the evaluator and each of the staff to apply the seminar information to their particular program’s evaluation. Between seminar dates, the staff spend time working on their evaluation plans according to what was discussed during the one-on-one meetings and the evaluator is “on-call” to provide problem-solving expertise. |
| Are Natural Family Planning Methods Viable Birth Control Options? Providers and Clients Perspectives |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Afsaneh Rahimian, Center for Health Training, afsaneh@jba-cht.com |
| Karen Deluhosh, Center for Health Training, kdluhosh@jba-cht.com |
| Sarah Goldenkranz, Center for Health Training, sarah@jba-cht.com |
| David Fine, Center for Health Training, dfine@jba-cht.com |
| Abstract: We used qualitative approaches to examine family planning providers’ perception of the efficacy of Natural Family Planning (NFP) methods as primary birth control and their experiences with their NFP clients. We also interviewed women about their experiences and satisfaction with these methods, pregnancy outcomes, sex partners’ involvement in birth control as well as their future plans. Preliminary findings suggest that almost all providers believed NFP was effective when used correctly. Both providers and clients associated NFP methods with the concept of empowering women. NFP clients chose this method to avoid hormonal contraceptives because of physical side-effects and emotional impact such as anger, depression, or other mood changes. Most clients were in mutually monogamous relationships. Sex partners were supportive of subjects’ birth control decisions. All women liked this method and reported no pregnancies while using NFP. Most clients planned to continue using NFP in the future. |
| Leading in the New Learning Ecology: A Leadership Framework for Technology Innovation Projects in Schools |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jenifer Corn, North Carolina State University, jocorn@ncsu.edu |
| Abstract: Staff at a university-based research center are conducting a 3-year longitudinal study focused on the evaluation of a statewide one-to-one (1:1) learning technology pilot initiative. This paper focuses on one component of this larger study, sharing the processes and results related to the identification of a leadership framework for technology innovation projects in schools. Researchers used a combination of teacher surveys, staff focus groups, and analysis of existing school-level data from 19 high schools to identify a framework for effective leadership of a technology innovation project. Using this new leadership framework, a 1:1 Leadership Evaluation Scale was developed to assist schools and districts in identifying and assessing the quality of school-based leaders for technology innovation projects. |
| Evaluating Supplemental Educational Services Over the Years: What Have We Learned? |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Judith Inazu, University of Hawaii, Manoa, inazu@hawaii.edu |
| Abstract: The No Child Left Behind law provides Supplemental Educational Services for academically-challenged students from low income families. SES provides free tutoring during non-school hours to improve students’ academic standing. A quantitative evaluation of SES is mandated by federal law. This paper summarizes the impact of the Hawaii program with data spanning a four-year period. Hawaii’s SES evaluation focuses on three questions: (1) Has the provider contributed to increasing student achievement? (2) Are stakeholders satisfied with tutoring services? (3) Are SES providers complying with the terms of their contract and with state and federal rules and regulations? Analysis of covariance was used to determine if students who received free tutoring increased their standardized test scores from one year to the next. Satisfaction measures were obtained by interviewing or surveying parents, school principals, and district staff. Finally, Hawaii’s methods and findings are compared with evaluation outcomes in other states. |
| Quantifying the Strengths, Weaknesses, Opportunities, and Threats (SWOT): a Method for Prioritization and Measuring Progress of a Strategic Plan |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Didi Fahey, Quantitative Research Evaluation & Measurement (QREM), fahey.13@osu.edu |
| Abstract: As part of a strategic planning process, the SWOT analysis typically involves a qualitative review of opinions solicited from key staff or board members. While useful, this type of analyses is limited and does not address prioritization or measuring the progress of the plan. A new methodology is presented that seeks empirical, quantitative review of key issues, rather than opinions of key personnel. This multi-step method utilizes the opinions of multiple groups by first naming, then reviewing the operations or processes of a particular organization. Next, the degree of importance of each process or operation is determined and key features are identified through data-reduction. By graphing the results on an internal/external axis, and then rotating the axis, evaluators can render a visual interpretation of events or practices likely to impact the organization. From there, strategic planning can be applied to maintain or improve desired processes or operations. |
| Iterative Reasoning: The Art of Determining Good Enough |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Karen Zannini Bull, Syracuse University, kbull@syr.edu |
| Abstract: Curriculum evaluation has surfaced under the program evaluation umbrella. Instructional designers are the individuals who systematically reflect the process of translating “principles of learning and instruction into plans for instructional materials, activities, information resources, and evaluation” (Smith and Ragan, 2005, p.4) within a curriculum. Instructional designers conduct front-end analysis’ to determine if there is a training or instructional problem. Throughout the design process, instructional designers participate in making multiple formative and summative decisions, or in other words, iterative reasoning. In evaluation, iterative reasoning could be identified as Patton’s developmental evaluation. How do instructional designers know the instructional solution is good enough to distribute to learners? Which if any of these decisions are formative, summative or simply iterative reasoning? The study will model Fitzpatrick’s work of interviewing exemplary evaluators and offering generalizations about how evaluators make evaluation decisions. |
| The Critical Role of Process Evaluation in Telling the Story |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Linda Winges, Battelle Memorial Institute, winges@battelle.org |
| Margaret J Gunter, LCF Research, maggie@lcfresearch.org |
| Judith Lee Smith, Centers for Disease Control and Prevention, jleesmith@cdc.gov |
| Katarzyna Alderman, Battelle Memorial Institute, aldermank@battelle.org |
| Gary Chovnick, Battelle Memorial Institute, chovnickg@battelle.org |
| Susan Pearce, Battelle Memorial Institute, pearce@battelle.org |
| April Salisbury, LCF Research, april.salisbury@lcfresearch.org |
| Deirdre Shires, Henry Ford Health System, dshires1@hfhs.org |
| Danuta Kasprzyk, Battelle Memorial Institute, kasprzyk@battelle.org |
| Daniel Montano, Battelle Memorial Institute, montano@battelle.org |
| Jennifer Elston Lafata, Virginia Commonwealth University, jelstonlafat@vcu.edu |
| Abstract: Interventions based on a theoretical model supported by research offer several advantages. First, the chances for success are improved; second, the change model is clearly specified; and third, knowledge of how the program theory works in a real world setting is gained. Unfortunately, real world implementation is rarely ideal. This paper describes a theory-driven process evaluation of an intervention to increase colorectal cancer screening rates in primary care clinics. The intervention is based on behavior change theory. Enhancing the quality of the evaluation was the multi-method approach to the process evaluation that relied on the convergence of qualitative and quantitative data from multiple points of view: observers, trainers, and trainees. Interviews were conducted at both intervention and control clinics. The evaluation followed the translation of behavior change theory to intervention design and implementation in order to better understand the story behind future results. |
| External Validity: Documenting Replicability and Transferability in Technology Programs |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Meghan Morris, State University of New York at Albany, morris.mlm@gmail.com |
| Kevin Murphy, State University of New York at Albany, km989754@albany.edu |
| Abstract: Abstract: In this paper, the authors discuss the evaluation, documentation, replicability, and transferability of the innovative Mobile Studio concept. Replicability across instructors is measured within programs and courses while transferability discusses external validity across content and across sites using the program. Descriptive narrative explains the real-time usability of the I/O board that developed under the Mobile Studio project. A myriad of data sources support the findings that multiple content areas (Physics and Electronic Instrumentation) and user characteristics positively serve the use of student-centered mobile technology within the domain of STEM education. Confounding this finding however, was the fact that the ability to accurately and credibly document use decreased the more removed the evaluators were from the site of use. |
| Using Qualitative Methods to Assess Effectiveness of a Public-Private Collaboration to Ensure Healthy And Active Living Through Land Use |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Marina Kaplan, Nemours Health and Prevention Services, makaplan@nemours.org |
| Patricia Miller, Nemours Health and Prevention Services, ppmiller@nemours.org |
| Gregory Benjamin, Nemours Health and Prevention Services, gbenjami@nemours.org |
| Abstract: To prevent continued development of obesogenic environments in Delaware, a non-profit children’s health system collaborated with the state Division of Public Health to ensure that developers received healthy living recommendations in response to land use applications submitted through the state-level review process. Two studies evaluated the effectiveness of this collaboration: • Study 1: Document review and analysis of recommendations to applicants and applicant responses pre- and post-collaboration • Study 2: Key informant interviews assessing stakeholder perceptions of the collaboration Compared to zero active recreation or healthy food access recommendations pre-collaboration, 59% of post-collaboration letters sent to developers included active recreation recommendations and 50% included healthy eating recommendations. Most (74%) developers stated they would amend their plans. Interview results suggest that developers began to change their approach to planning by more proactively incorporating healthy living opportunities in their subsequent plans. |
| Caught in the Middle: Turning Around Middle School Performance |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Gary Timko, Community Research Partners, gtimko@columbus.rr.com |
| Abstract: To improve middle school performance in the Columbus City Schools (CCS), the largest school district in the state of Ohio, the superintendent asked Community Research Partners (CRP) to talk to school administrators, teachers, non-teaching support staff, and parents about what they perceive to be barriers to student learning and school performance and what needs to be done to turn schools around. CRP reviewed research literature to identify middle school best practices, conducted focus groups with teachers and non-teaching support staff at each of the 23 CCS middle schools, conducted 45 individual telephone interviews with principals and vice-principals, and facilitated a group discussion with middle school parents. This paper presents frank and host perceptions of “the good, the bad, and the ugly” aspects of the middle schools in the district. The presentation also includes a brief discussion of how the evaluation findings are being used by the district to make changes. |
| Reviews of Student Achievement Tests as Metaevaluation: Implications for the Evaluation Profession |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Gabriel Della-Piana, Independent Consultant, dellapiana@aol.com |
| Michael Gardner, University of Utah, mike.gardner@ed.utah.edu |
| Abstract: Reviews of achievement tests are a kind of metaevaluation (Scriven, 1991; Stufflebeam (retrieved March 17, 2010). The reviewer evaluates the evaluation of the test. Current test reviews and other lines of evidence reveal signs that professional standards for educational achievement testing are not followed well in practice. This paper is of importance to evaluators of educational programs where student achievement tests are often construct underrepresented outcome measures. The paper proceeds by addressing the following: 1) What are the demands on the teacher and the evaluator for making sense of student achievement test scores? What is revealed by analyses (meta-evaluations) of reviews of achievement tests as to the kinds of validity evidence presented by developers of achievement tests including one widely used achievement test battery? What are the difficulties in sorting out responsibility for appropriate test development and use? What are the implications for evaluator training, evaluation theory and evaluation practice? |
| Evaluating Social Media: The Extension Experience |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michael Lambur, Virginia Tech, lamburmt@vt.edu |
| Abstract: When eXtension (www.extension.org) was first conceived, we had anticipated a more "captured" audience (i.e., people registering on the public site where we could work with them on programming issues and evaluation). This has essentially gone by the wayside with the advent of web 2.0 applications. Social media applications like YouTube, Twitter, and Facebook are examples of ways that individuals are connecting with each other in the digital environment. These have tremendous informal social value and eXtension is adopting them to engage Extension clientele in non-formal educational programming. In this presentation, I will focus, I will discuss challenges and approaches in evaluating three web 2.0 applications: YouTube, Facebook, and Twitter. |
| Participatory Evaluation for Organizational Learning in an Inquiry-based Chemistry Laboratory Curriculum |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Hui-Jung Chen, National Taiwan Normal University, karen3117tw@gmail.com |
| Mei-Hung Chiu, National Taiwan Normal University, mhchiu@ntnu.edu.tw |
| Abstract: The purpose of this study is to rethink the mechanism underlying the links between activities and consequences in participatory evaluation by enacting an empirical study in a school setting. In order to enhance the influence of evaluation results, a new framework as participatory evaluation for organizational learning (PEOL) was proposed referred to constructivism, knowledge utilization theory and adult learning principles. Data collection included interviews, questionnaires, classroom observations and video recording. The results indicate that both teachers and evaluators have considerable instrumental, conceptual and symbolic influence on the evaluation findings. The mechanism of participatory evaluation was suggested as knowledge co-constructed in the whole evaluation process, not only in the particular steps of data analysis or data interpretation by evaluators and stakeholders. The co-constructed knowledge is transformed into actionable knowledge, eventually leading into actions. It effectively facilitated the evaluation influence of evaluation findings. |
| Evaluation of a Social Justice Leadership Institute in Higher Education: Learning From the First Year |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Nicole Cundiff, University of Alaska, Fairbanks, karim@siu.edu |
| Roy Joy, Southern Illinois University Carbondale, rhjoy@siu.edu |
| Jennifer Beckjord, Southern Illinois University Carbondale, jbeckjord@siu.edu |
| Andrea Coulter, Southern Illinois University Carbondale, adcoulter@siu.edu |
| Abstract: A Social Justice Leadership Institute was adopted by a medium-sized Midwest public university for one semester. This paper describes the program development, evaluation process, and learning outcomes and use. Fifty undergraduate and graduate students voluntarily participated in the program, which was assessed using a pre- post comparison group design. Several measures were administered (e.g., ethnocultural empathy, understanding of diversity constructs, knowledge of social justice issues, expectations of the program’s effects, and personal and professional beliefs; among others) in an online survey form. In addition, qualitative data was gathered for assessment of knowledge gained, the emotional impact of the program and its effects on perceived agency in addressing social justice issues. Finally, results and their application are discussed to inform future institutes of best practices. |
| Economic Analysis of National Research and Development (R&D) Program Developing a Geostationary Satellite: The Case of Korea |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Yoon Been Lee, Korea Institute of Science & Technology Evaluation and Planning (KISTEP), yblee@kistep.re.kr |
| Jiyoung Park, Korea Institute of Science & Technology Evaluation and Planning (KISTEP), jypark@kistep.re.kr |
| Jiho Hwang, Korea Institute of Science & Technology Evaluation and Planning (KISTEP), jihoh@kistep.re.kr |
| Abstract: In this paper we present a case about economic analysis of development program of a geostationary satellite. As the program was planned to secure meteorological, environmental and oceanic observation abilities, benefit from each payload was analyzed. In the meteorological field, there are benefits to reduce the social expenses such as the casualties due to the severe weather, the property damage by the disaster and the loss due to the flight delay. The benefit from oceanic payload is the prevention of red tide and the prompt measure against the sea disasters or increasing the fishery production. We also analyzed the benefit from the reduction of death rate by precision improvement in the observation and prediction of air pollution such as micro dusts and ozone. The AHP was applied to integrate the result with that from the viewpoint of technology and policy. |
| Developing Quality Assessments of Observational Research |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| John Pfaff, Fordham Law School, jpfaff@law.fordham.edu |
| Abstract: Empirical research in the social sciences stands at a critical threshold. As the volume of statistical analysis grows, it becomes increasingly difficult to evaluate what is known about a particular phenomenon. Contradictory findings abound, and the social sciences lack effective tools to filter good claims from bad and to synthesize the high-quality findings. In this paper, I examine why this is and what must be done to address the problem. The core problem is that the social sciences have not embraced the evidence-based policy (EBP) movement revolutionizing medicine and other fields. Partly this is due to a misguided philosophy of science, but partly to the fact that the social sciences must rely on non-experimental, observational methods, which EBP has ignored. I consider how to develop rigorous, systematic reviews for observational studies and focus on many of the unique challenges that observational methodology poses for such reviews. |
| An Evaluation of a Capacity Building Performance Management Project: Results From the First Two Years |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lauren Decker, Edvance Research, ldecker@edvanceresearch.com |
| Leslie Grunden, Edvance Research, lgrunden@edvanceresearch.com |
| Abstract: The Statewide Tools for Teaching Excellence (STTE) project aims to empower district leaders to better utilize data, information, and optimize resources for continuous improvement. This is done through the use of a closed-loop, district-wide performance management model that enables educators to set measurable goals, identify research-based interventions, and evaluate chosen interventions in a high quality way. Five case studies are currently being conducted to evaluate the STTE project in two phases. Phase I consists of indicator and data tool development. Phase II consists of professional development instruction and training and serves as a pilot phase for the developed data tool (currently in process). Focus groups and questionnaires are the primary data collection tools being used. Results will be discussed in terms of noted changes in district beliefs, strategies, and practices that result from district participation in the project. The potential for project scale-up will also be discussed. |
| New Logic Model Approach |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Fayez Shafloot, Western Michigan University, eval.p@hotmail.com |
| Abstract: This paper will highlight the new Logic Model approach that will consider other hidden needs during the processes of using logic model. The new logic model has to main advantages in which it will meet most of the organizations' needs by expanding the critical thinking frame for meeting the performance needs and will give specific solutions for specific problems or questions. There are, for example, conscious and unconscious needs that could not be figured out by an individual or organization until we use several methods to collect data and many methods to analyze them. A survey could answer the questions for conscious needs, but other unconscious needs are not easy to be recognized by using this method. This model will worth its implementation and closes the performance gaps by classifying specific needs that serve the optimal organization performance needs. |
| Embracing the Context of Pediatric Rehabilitation Programs: A Survey Investigating the Role of Family-Centered Philosophy in Program Evaluation |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Katherine Moreau, University of Ottawa, kmoreau@cheo.on.ca |
| J Bradley Cousins, University of Ottawa, bcousins@uottawa.ca |
| Abstract: Offering a wide range of medical, social, educational, and recreational services, paediatric rehabilitation settings have transitioned from a medically-focused to family-centred model of programming. This model recognizes that families know their children best and thus, should be actively involved in program development, implementation, and evaluation. However, many researchers believe that the push for accountability and evidence-based practice results in families being seen as sources of data rather than collaborators in evaluation. In this context, researchers believe that many evaluations are incongruent with the family-centred model. To date, researchers have not thoroughly investigated or formally documented the evaluation processes used in paediatric rehabilitation settings, and therefore the degree to which they are incongruent with the family-centred model is unclear. This presentation reports the findings of an empirical study designed to determine and document program evaluation practice in paediatric rehabilitation settings and its congruence with the family-centred model of programming. |
| Using Case Studies to Assess Promising Practices and Lessons Learned in Comprehensive Evaluations of Multi-level Tobacco Control and Childhood Obesity Interventions |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jenica Huddleston, University of California, Berkeley, jenhud@berkeley.edu |
| Abstract: Innovative ways to address tobacco control and childhood obesity include intervening on multiple levels concurrently, such as individual, community and policy levels. Multi-level interventions are being developed, and more widely implemented, but often do not include a comprehensive evaluation. When evaluation is included, it is often piece-meal, with important elements lacking. Examples of comprehensive evaluations within the fields of tobacco control and childhood obesity were identified and explored further through multiple case studies. Some of the areas investigated include: How did these projects end up with comprehensive evaluations? What were some of the contextual factors linked to developing and implementing a comprehensive evaluation? Can lessons learned from one field help to advance work in the other? This presentation will describe elements for comprehensively evaluating multi-level interventions and lessons learned from these projects. This study, and presentation of results, can inform current practices and future work for evaluators in various contexts. |
| Evaluation is a Program Requirement, Now, How Do You Ensure Quality? Evaluation Technical Assistance: An Approach for Center for Disease Control and Prevention's (CDC) National Heart Disease and Stroke Prevention Program |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Eileen Chappelle, Centers for Disease Control and Prevention, ebf1@cdc.gov |
| Abstract: The Division for Heart Disease and Stroke Prevention (DHDSP) from the Centers of Disease Control and Prevention (CDC) funds 41 state programs and the District of Columbia to build capacity in addressing heart disease and stroke prevention at the population-level. This program requires funded states to evaluate specific interventions as well as their partnerships. To ensure the quality of these evaluation efforts, the DHDSP at CDC provides consistent technical assistance on evaluation to the funded state programs. This session will: (1) describe a portfolio approach of evaluation technical assistance and support provided; (2) review the available evaluation resources; (3) illustrate the value of the CDC Evaluation Framework and use of logic models and; (4) share lessons learned in providing technical assistance to state health departments. |
| Overcoming Authoritarian Approach to School Evaluation: Systemic Change Towards Democratization- Polish Case |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Grzegorz Mazurkiewicz, Jagiellonian University, grzegorz@expedition.org.pl |
| Abstract: The main aim of the paper is presentation of the key assumptions that created a base for modernization effort of the Polish schools’ supervision system. In the year 2009 the Ministry of Education in Poland launched the project focusing on designing and building the new model of supervision. The most important organizational decision was to divide control, support and evaluation. This paper will describe goals, values and procedures of the external evaluation. That attempt is a result of the globally shared awareness of the importance of evaluation for school quality. Numerous international initiatives are conducted to increase the effectiveness of the educational evaluation, however there is impossible to state that countries have found the universal solution for evaluation (or other forms of gaining information about educational systems – measurement, research etc). That paper will describe the project in progress and its planned outcomes (new professional culture in Polish schools). |
| Applying Program Evaluation Standards to Improve Education Through Meta-evaluation: The Brazilian Experience |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Thereza Penna-Firme, Cesgranrio Foundation, therezapf@uol.com.br |
| Ana Carolina Letichevsky, Cesgranrio Foundation, anacarolina@cesgranrio.org.br |
| Abstract: The program evaluation standards of Utility, Feasibility, Propriety and Accuracy provide a foundation for ensuring evaluation quality. Yet, their application in a complex evaluation context can be difficult. In Brazil, universal access to primary education in the eighties and the nineties led to the implementation of systems level, national evaluations through standardized tests using Item Response Theory. Immediate attention to psychometric rigor and the tendency to value the accuracy standard without simultaneously considering the other three becomes a concern. Utility of results is yet to be recognized, due to a lack of an evaluation culture in schools. Feasibility is an issue until teachers make timely use of the evaluation results. Regarding propriety, stakeholders are to understand the need to evaluate for program improvement, without offending the values of persons involved. The paper makes a case for formative meta-evaluation to simultaneously improve teacher capacity in evaluation, the evaluative process, and education. |
| Creating Sustainable Structural Changes in State Mental Health and Substance Abuse Systems: Findings From the National Evaluation of the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Co-occurring State Incentive Grant Program |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| John Hornik, Advocates for Human Potential, jhornik@ahpnet.com |
| Abstract: While there is evidence, that mental health and substance abuse disorders have a co-occurrence rate of at least 50%, people are rarely treated for both disorders. Assuming that structural barriers prevent access to treatment, SAMHSA began a program of Co-occurring State Incentive Grants (COSIG) in 2003. We faced several problems in designing a national evaluation to assess structural changes states pursued, how they organized create change, and the sustainability of changes. The evaluation began two years after the program, states selected goals from different domains, state environments varied, and states received different funding levels. We undertook repeated interviews with key informants through site visits and extensive review of project documentation, as well as special studies. Among the findings, screening, workforce development, and services integration were goals that most states pursued showing sustainable change, but the 15 states varied widely in the number and success of their change efforts. |
| The Importance of Context in Understanding Career Academies: A Case Study Approach |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bridget Cotner, University of South Florida, bcotner@cas.usf.edu |
| Maressa Dixon, University of South Florida, mdixon83@gmail.com |
| Margaret Allsopp, University of South Florida, mallsopp@mail.usf.edu |
| Tasha-Neisha Wilson, University of South Florida, tcwilson@usf.edu |
| Margeaux Chavez, University of South Florida, |
| Kathryn Borman, University of South Florida, |
| Abstract: As part of a large, mixed method study looking at the impact of Career Academy participation on high school student outcomes, a case study method was used to document the context as it relates to state policy and student outcomes. While the quantitative team is analyzing state level data to determine student outcomes based on participation in a career academy, the qualitative team identified and visited eight career academies in three Florida school districts to obtain a contextualized understanding of how career academies function. We use a case study approach similar to Saez and Carreto (1998, p. 29) who explain, “Each case provided a unique picture of the implementation…as well as of innovations in particular contexts.” The case studies provide insight into how stakeholders are viewing and implementing policy, such as the new school grade reconfiguration (SB 1908), and how policy is driving practice at the school level. |
| National Institute of Environmental Health Sciences (NIEHS) Small Grants: Do They Lead Down the Intended Path? |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jerry Phelps, National Institute of Environmental Health Sciences, phelps@niehs.nih.gov |
| Elizabeth Ruben, National Institute of Environmental Health Sciences, rubene@niehs.nih.gov |
| Joshua D Schnell, Discovery Logic, joshua.schnell@thomsonreuters.com |
| Duane E Williams, Discovery Logic, duane.williams@thomsonreuters.com |
| Abstract: In this paper we compare applications with a prior history in the National Institute of Environmental Health Sciences (NIEHS) small exploratory research program (R03/R21) to those who do not have prior history. We determine if they score better, compete more successfully, and produce results faster in a subsequent Research Project Grant (RPG) application or award. We use a quantitative approach to explore whether participation in the R03/R21 program increases the output of the subsequent grant. For the study, our dataset is comprised of 408 funded R03/R21 grants awarded FY1996 - FY2008 . We identified subsequent RPGs using a three-step process of 1) matching principal investigators names, 2) comparing dates of submission, and 3) comparing title, abstract, and specific aims of R03/R21 grants and candidate RPGs using a text-based similarity algorithm. These methods use a novel approach for assessing outcomes of NIH grant programs, and can be translated to other analyses. |
| The Road to Behavior Change is Both Bumpy and Smooth: Barriers and Facilitators of Patient Safety Training Outcomes |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristina Moster, Cincinnati Children's Hospital Medical Center, kristina.moster@cchmc.org |
| Stacey Farber, Cincinnati Children's Hospital Medical Center, stacey.farber@cchmc.org |
| Britteny Howell, Cincinnati Children's Hospital Medical Center, britteny.howell@cchmc.org |
| Abstract: Patient safety is a key focus in healthcare today with much work and many resources being devoted to improving and ensuring the safety of patients. This paper discusses the evaluation of one such effort – a patient safety education and training program at a children’s hospital with more than 11,000 staff. A total of 18 on-going patient safety-related courses were evaluated, including classroom, web-based, and simulation courses. Staff participated in one or more of the courses and were surveyed immediately post-training and again two to three months later. The surveys addressed outcomes in terms of participants’ perceptions of satisfaction, learning, application, impact, and value. Results were highly to moderately positive for all areas measured. Perceived barriers to and facilitators of application of training to the workplace were also reported. Program outcomes, barriers and facilitators of application, and challenges to evaluation quality will be discussed. |
| Using Bibliometric Information for Benchmarking and Research and Development (R&D) Portfolio Management |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brian Zuckerman, Science and Technology Policy Institute, bzuckerm@ida.org |
| Kevin Wright, National Institutes of Health, kevin.wright@nih.gov |
| Brandie Taylor, National Institutes of Health, brandie.taylorbumgardner@nih.gov |
| Abstract: At the National Institutes of Health, funding decisions for investigator-initiated research are made on an award-by-award basis, driven by peer review, and not managed as a research portfolio. The National Institute of Allergy and Infectious Diseases (NIAID) has conducted a benchmarking study of the productivity of investigator-initiated awards, focusing on R01 awards first funded in fiscal year 2001. Award size, length of award, and clinical research were all positively correlated with publication output, but only accounted for 25-30% of the variance across awards, suggesting that award-specific factors, rather than such structural variables, play the primary role in determining research productivity. Extensions of the study examined citation-weighting and impact factor weighting, as well as examining cohorts of awards funding in 1990, 1995, and 2000 to assess whether there were regularities with respect to declines in productivity over the lifetime of awards. |
| One Central Florida School District’s Efforts to Help Struggling Readers: An Evaluation of the Effectiveness of the READ 180 Reading Intervention Program on Student Achievement and Student Perceptions on Reading |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Yakup Bilgili, Polk County Public Schools, yakup.bilgili@polk-fl.net |
| Abstract: This study evaluates the effectiveness of the READ 180 reading intervention program for struggling readers on secondary school students in one central Florida school district. Eight and nine grade students whose previous year’s Florida Comprehensive Assessment Test (FCAT) reading scale scores placed them in Level 1 and Level 2 were scheduled/placed into Intensive Reading classes for reading instruction during the 2009-10 school year in a large Florida school district. The goal of the READ 180 program within the district was to increase targeted students’ reading levels to their appropriate-proficiency grade levels (on-grade level). This study includes 33 secondary school students in grades eight and nine enrolled in READ 180 reading intervention program during the 2009-10 school year. A random sample of students not exposed to READ 180 intervention program from the prior year (2008-09), who had similar FCAT reading levels of performance (Level 1 and Level 2) in grades 8 and 9 will be selected to serve as a comparison group to determine effectiveness. |
| Evaluating Program Impact in Alignment With Cultural Expectations of Success |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Leigh D'Amico, University of South Carolina, kale_leigh@yahoo.com |
| Robert Petrulis, University of South Carolina, petrulis@mailbox.sc.edu |
| Cynthia Bolton-Gary, University of South Carolina, Beaufort, cbolton@uscb.edu |
| Abstract: Expectations related to the impact of educational interventions can often be idealistic, unrealistic, or unmeasurable. In a climate that focuses on student achievement as measured by standardized test scores as a primary indicator of program success, many individuals and organizations struggle to develop and implement interventions specifically designed to increase these test scores. When interventions fail to demonstrate impact on test scores, they are often abandoned or labeled insignificant. Meanwhile, evidence may be available to show meaningful impact of these programs and longer-term, multi-faceted evaluations may demonstrate their value in improving student achievement. Evaluators and program implementers will discuss examples from large scale educational programs funded through federal and state initiatives designed to improve student outcomes that have been challenged to meet unrealistic goals, explain lack of progress on these unrealistic goals and objectives, and present stakeholders and decision-makers with important information about the definition of program success. |
| Lessons Learned Coordinating Evaluations of Multi-site Community-based Initiatives |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Laurie Carpenter, University of Michigan, lauriemc@umich.edu |
| Martha Quinn, University of Michigan, marthaq@umich.edu |
| Abstract: This presentation will focus on lessons learned while evaluating multi-site, community-based health initiatives related to the prevention and management of chronic diseases such as asthma, heart disease, epilepsy, and obesity. The insights presented will include experiences related to the development of evaluations which are relevant and useful for all partners, and working with diversity at a number of levels: context, programmatic goals and structures, stakeholders, and beliefs and skills related to evaluation. These lessons will contribute to the literature to which evaluators can turn when developing and implementing multi-site evaluations, and will also support the development of a new research model for community-based health initiatives, which do not fit into the traditional model utilizing randomized controlled trials as the gold standard. |
| Using Mixed Methods to Develop and Evaluate a Safe-Ride Program for Tweens |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Krystall Dunaway, Eastern Virginia Medical School, dunawake@evms.edu |
| Kelli Will, Eastern Virginia Medical School, willke@evms.edu |
| Edward Lorek, Eastern Virginia Medical School, lorekej@evms.edu |
| Abstract: The purpose of this study was to develop, implement, and evaluate a school-based program targeting increased safety belt and backseat use among 8-12 year-old children (tweens), as less than half of tweens involved in crashes wear seatbelts and 73% sit in the front seat. Developed with input from student focus groups, the Make-it-Click program was piloted via a pre-post control-group design in four elementary schools. The program’s six intervention components were implemented throughout the school year. The program resulted in significant increases in belt use (N=762) at Make-it-Click intervention schools (from 31.9% at baseline to 67.9% at follow-up), while belt use at control schools did not differ from follow-up to baseline. Thus, intervention school tweens were 3.3 times more likely to be belted at follow-up than control school tweens. No significant difference was found in backseat use between the intervention and control groups at any phase of the study. |
| What Are the Limits of Human Rights-based Evaluations? |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stephen Porter, University of the Witwatersrand, stephen.porter@wits.ac.za |
| Abstract: This paper presents initial research on evaluative criteria, approaches and limits for human rights-based evaluations grounded in the challenges facing the South African transition. Among the criteria are self-determination and non-discrimination. Principal amongst approaches is analysis of the rights-holder and duty-bearer interface. Limits are both practical and theoretical. Practitioners who claim to implement rights-based evaluations do so within health, gender, and governance and programmes. This demonstrates an emerging approach to evaluation that integrates development and legal literature. Yet, within this approach there is no boundary as to what can be claimed as rights-based. Preliminary frameworks are in place, yet these are very broad. Arguably ‘anything goes.’ If anything does indeed ‘go’ then the unique contribution of rights-based evaluation could be questioned. This paper presents some ideas on the contribution, but delineates them. This demonstrates how there are contributions beyond existing evaluation approaches that centralise Social Justice. |
| The Texas Strategic Prevention Framework State Incentive Grant (SPF-SIG): How the Keep It Simple Stupid (KISS) Principle Works in Evaluation |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Daniel Jorgensen, Texas A&M University, dan.jorgensen@tamucc.edu |
| Abstract: Evaluators often mean well when they use their experience and expertise to develop program evaluation. There are times, however, we when get so wound up in creating our own survey and evaluation regimes we often forget that first and foremost we must look at the goals set for the program by the funding agency. While we are often looking for rigor we forget the simple goals programs must achieve. Perhaps at the detriment of the evaluation. This paper shows how the NOMs are readily achieved for a SPF-SIG program and how they can get lost in an overly rigorous evaluation instrument and methodology. |
| Effect of Read180 Program on Incarcerated Youth’s Reading Performance: An Evaluation Study |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Weijia Ren, The Ohio State University, ren.44@buckeyemail.osu.edu |
| William Loadman, The Ohio State University, loadman.1@osu.edu |
| Raeal Moore, The Ohio State University, moore.1219@osu.edu |
| Jing Zhao, The Ohio State University, zhao.195@osu.edu |
| Deborah Kwon, The Ohio State University, kwon.59@osu.edu |
| Charles Okonkwo, The Ohio State University, okonkwo.1@buckeyemail.osu.edu |
| Abstract: There is increasing emphasis on youth’s reading achievement. This study focuses on evaluating the effectiveness of Scholastic’s Read180 program in improving incarcerated youth’s reading achievement scores in Ohio’s Striving Reader’s project. Hierarchical linear modeling (HLM) is used to analyze the data. Results indicate that race and disability are significant in predicting the reading achievement scores, as well as the growth rate of the scores across three years. Non-black or non-disabled youth tend to have higher scores than black or disabled youth. However, in terms of the growth rate, black or disabled youth tend to increase faster. The results and potential difficulties in analyzing this population will be discussed. |
| Using Developmental Evaluation to Inform Development of a Complex Education Program |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Gretchen Biesecker, City Year Inc, gbiesecker@cityyear.org |
| Belle Brett, Brett Consulting Group, bellebrett@comcast.net |
| Kathryn Race, Race & Associates Ltd, race_associates@msn.com |
| Dannalea D'Amante, City Year Inc, ddamante@cityyear.org |
| Abstract: A developmental approach to evaluation meets the complex social and educational needs of programs as they mature and innovate, as described by Patton (1994) and Jacobs (1988), for example. This presentation will describe a case of how one national education-focused non-profit, evolving to address the academic and social needs of students in grades 3-9, used this approach to develop, pilot, and create scales resulting in useful data from survey tools and other instruments. Program staff, the internal evaluation department, and external evaluation consultants collaborated to: create Theories of Change for work with different age groups of students, pilot and revise a variety of student and teacher/principal instruments and measures, review and interpret data, foster internal learning, and move both the program and its evaluation to the next level over time. We will share how this approach fosters quality in our data collection, reliability of instruments, and use of data. |
| Understanding the Reporting of Bullying to School Officials |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Anthony Petrosino, WestEd, apetros@wested.org |
| Sarah Guckenburg, WestEd, sgucken@wested.org |
| Jill DeVoe, American Institutes for Research, jdevoe@air.org |
| Thomas Hanson, WestEd, thanson@wested.org |
| Abstract: Bullying appears to be quite frequent among U.S. schoolchildren and has a number of short-term and long-term negative consequences. Unfortunately, many victims do not report their victimization to school authorities, hampering educators’ ability to intervene. This project uses national data from the 2007 School Crime Supplement to the National Crime Victimization Survey (Bureau of Justice Statistics) to analyze reporting behavior among bullying victims. Secondary data analyses respond to three questions: 1. To what extent do the characteristics of bullying incidents for students who report their bullying victimization differ from those who do not report? 2. To what extent do the characteristics of the students who report their bullying victimization differ from those that do not report? 3. To what extent do the characteristics of the schools for students who report their bullying victimization differ from those that do not? |
| Enhancing Evaluation Capacity of a Tribal Juvenile Justice Program: Evaluating Creating Restitution and Following Traditions (C.R.A.F.T.) |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Melissa Boeke, Minnesota Institute of Public Health, mboeke@mipg.org |
| Sharon Rosebear, Red Lake Band of Chippewa Indians, srosebear@redlakenation.org |
| Abstract: In 2009, the Red Lake Nation C.R.A.F.T. Program (Creating Restitution and Following Traditions) was nominated to participate in an evaluation capacity building program called “Service to Science Academy”. The Academy was conducted by the Central Center for the Application of Prevention Technologies (CCAPT), a project of the U. S. Department of Health and Human Services Center for Substance Abuse Prevention (CSAP). At the time, the 12-year-old program had never been formally evaluated. After consultation provided at the academy and subsequent award of an evaluation enhancement mini-grant, program staff learned to: create logic models, develop an evaluation plan, create data collection protocols, explore a variety of methods for documenting program activities, and enhance existing cultural assessment instruments. The goal of the enhancement was threefold: conduct a formal evaluation of the program, build capacity of program staff to conduct future evaluations, and add to the limited literature on effective tribal prevention programs. |
| Assessing Reasons for High School Dropout: Dropout Typologies |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Stacey Merola, ICF International, smerola@icfi.com |
| Allan Porowski, ICF International, aporowski@icfi.com |
| Kazuaki Uekawa, ICF International, kuekawa@icfi.com |
| Abstract: In this paper we will investigate the context surrounding students’ decisions to drop out of high school, and whether differences in their reasons for dropping out are linked to differences in later outcomes. Using data from the National Longitudinal Study of Youth 1997 (NLSY97), we will present a typology of dropouts, based on their reasons for dropping out, the risk factors present in their lives prior to dropping out, and the sequence of events in their lives after the decision to drop out. In this presentation we will present preliminary findings from our work, and we will discuss how the use of secondary data can help inform evaluations of dropout prevention programs. |
| From Smolt to Salmon Project: Lessons Learned From a Culturally Responsive Mentoring Project |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Annie Woo, Education Northwest, annie.woo@educationnorthwest.org |
| Abstract: From Smolt to Salmon Project is designed as a school-based mentoring program for tribal community children and youth in grade four through eight, on the premise that children need role models and close relationships with adults of Native American culture. The goals of the program are: a) increase participation in, and enhance the ability to benefit from, elementary and secondary education; and b) provide general guidance and encourage involvement in community cultural activities. The objectives of the study are: a) developing an effective mentoring program in school to empower the youth in dropout and truancy; b) providing the youth with both intellectually challenging and emotionally supportive learning environment; and c) identify ways to provide culturally responsive mentoring to Native American adolescents in the community. |
| Participatory Development of a Logical Framework in Brazil: The Labor Gymnastics Experience |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lorena Vilarins, Social Service Industry, Brazil, lorena.vilarins@sesi.org.br |
| Miguel Fontes, John Snow, Brazil, m.fontes@johnsnow.com.br |
| Rodrigo Laro, John Snow, Brazil, r.laro@johnsnow.com.br |
| Fabrízio Pereira, Social Service Industry, Brazil, fpereira@sesi.org.br |
| Abstract: Objective: To ensure impact evaluation quality using a participatory evaluation process among labor gymnastics program staff Methods: Using a logical framework matrix, program’s staff from all states of Brazil and of different hierarchy levels (mediated by an evaluation team), established consensus about programs values, and also set objectives, goals and indicators aiming a follow-up impact evaluation Results: A participatory evaluation process engages program’s staff with the results and can lead to the effective use of the findings, communication of the outcomes and a continuing rethinking of the program. 30 consensual efficacy and over 50 impact indicators were established and used in the development of evaluative research tools. Staff members also used the final logical framework as a tool for better defining the redesign of the program Conclusions: Participatory evaluation processes produce not just research tools, but they empower those involved in the development of a social program |
| Evaluating the Effectiveness of Warning Systems in Extreme Weather Events |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brandi Gilbert, University of Colorado, Boulder, brandi.gilbert@colorado.edu |
| Liesel Ritchie, Natural Hazards Center, University of Colorado at Boulder, liesel.ritchie@colorado.edu |
| Alexandra Jordan, University of Colorado, Boulder, alexandra.jordan@colorado.edu |
| Abstract: This paper explores how warning information for extreme weather events is acquired, interpreted, and disseminated. Our evaluation of weather warning systems specifically focuses on inter-organizational warning processes for flash floods in Boulder, CO and hurricanes in Miami, FL. This data was collected by conducting face-to-face structured interviews with forecasters, public officials, and broadcasters—representatives of key organizations involved in the warning process. Few evaluations have focused on the ways in which technical weather information is interpreted and translated into warnings and protective actions for the public. However, learning more about how organizations communicate during extreme weather events has the potential to improve the warning process, ultimately reducing injury, loss of life, and property loss. |
| Standardizing Cost-Effectiveness Results of a Social Service Program: A Clustered Sampling Analysis |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lorena Vilarins, Social Service Industry, Brazil, lorena.vilarins@sesi.org.br |
| Miguel Fontes, John Snow, Brazil, m.fontes@johnsnow.com.br |
| Rodrigo Laro, John Snow, Brazil, r.laro@johnsnow.com.br |
| Fabrízio Pereira, Social Service Industry, Brazil, fpereira@sesi.org.br |
| Edson Lopes, John Snow, Brazil, est.edson@gmail.com |
| Abstract: Objective: To produce a cost-effectiveness analysis as part of an impact evaluation of a social service program clustered on 27 states of Brazil Methods: Based on objectives, goals and indicators established in a logical framework of the program, a uniform 4 steps cost-effectiveness analysis was conducted across all 27 Brazilian states (clustered sample, n=300). The 4 steps were: a) identifying financial per capita cost in each state; b) calculating the proportion of 41 efficacy variables; c) defining rates of efficiency and; d) standardizing a multi-site cost-effectiveness index. Results: Even though percapita costs and rates of efficacy showed significant different ranges across states, the use of a standardizing multi-site cost-effectiveness index fit a normal curve. The final normalization test (swilk test) reached p-value>0.05, confirming the null hypothesis of normal fit. Conclusion: The use of a standardizing multi-site cost-effectiveness index formula promotes better comparison of results across geographically clustered samples. |
| Healthy Eating and Physical Activity Policy and Practice Changes in Delaware Schools: 2006 - 2008 |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Gregory Benjamin, Nemours Health and Prevention Services, gbenjami@nemours.org |
| Vonna Drayton, Nemours Health and Prevention Services, vdrayton@nemours.org |
| Zhongcui Gao, Nemours Health and Prevention Services, zgao@nemours.org |
| Abstract: Nemours, an integrated health system, is dedicated to combating childhood obesity in Delaware in partnership with school districts and other community organizations. Nemours provides partners with technical assistance to translate healthy eating and physical activity policies into practices to facilitate behavior change. Nemours’ 5-2-1-Almost None prescription for a healthy lifestyle, recommends daily consumption of 5 or more servings of fruits and vegetables, limiting screen time to less than 2 hours, at least 1 hour of physical activity, and consumption of almost no sugar-sweetened beverages. These behaviors are measured using the Delaware Survey of Children’s Health (DSCH). Early data indicate significant increases in physical activity and moderate to vigorous physical activity in school-age children. This presentation will focus on behavior changes resulting from existing physical activity legislation (e.g., HB 372 and HB 471) and advocacy efforts in support of healthy eating legislation to sustain and further transform schools into healthier environments. |
| Understanding the Structure and Operations of State Offices in a Large-Scale Youth-Serving Organization: A Multi-state Study of Communities In Schools |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Heather Clawson, ICF International, hclawson@icfi.com |
| Aikaterini Passa, ICF International, apassa@icfi.com |
| Sarah Decker, ICF International, sdecker@icfi.com |
| Susan Siegel, Communities In Schools, siegels@cisnet.org |
| Abstract: As part of a comprehensive five-year national evaluation of Communities In Schools, we conducted a study to determine the organizational structure and core functions of state offices across the CIS network. More specifically, this study explored the roles and supports provided by the CIS National Office to state offices and to local affiliates in states without state offices. By comparing the experiences of affiliates with state offices to those without state offices, a better understanding of the key elements of success in working with each of these two models was developed. Using results of phone interviews with key stakeholders and a survey, we provided guidance to the CIS National Office on the needs of state offices and affiliates, identified areas where state offices can improve their relationships with affiliates, and made recommendations on how large federations such as CIS can improve operations. |
| A Teaching Career Path in Science, Technology, Engineering, and Mathmatics (STEM) Education: An Evaluation of a Noyce Teacher Recruitment Initiative in Physics and Chemistry |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Meltem Alemdar, Georgia Institute of Technology, meltem.alemdar@ceismc.gatech.edu |
| Donna Whiting, Georgia Institute of Technology, donna.whiting@ceismc.gatech.edu |
| Greg Rushton, Kennesaw State University, grushton@kennesaw.edu |
| Abstract: This paper presents an evaluation of Teacher Recruitment Initiative in Physics and Chemistry program, a collaboration between Kennesaw State University (KSU) and Georgia Institute of Technology (GT). This program is a part of Noyce Teacher Recruitment Initiative funded through the National Science Foundation. It is designed to encourage and enable GT and KSU undergraduate science and engineering majors to pursue careers in high school chemistry and physics teaching. It provides scholarships to science and engineering majors over a four year period, to be awarded during their senior undergraduate year, and during their enrollment in KSU’s MAT program. This paper focuses on the program during its year of implementation, and its effects on participants. Program impact is evaluated in two areas: (1) recruitment strategies, (2) program effectiveness. Analyses is triangulated across multiple data sources (surveys, interviews, observation & documents) collected. This paper will also elaborate on critical issues in STEM evaluation. |
| Put Your Money Where Your Fat is: Evaluating Healthy Communities Funding and Obesity Rate Patterns, 2000-2010 |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Dave Brewington, YMCA of the USA, dave.brewington@ymca.net |
| Andrea Lee, YMCA of the USA, andrea.lee@ymca.net |
| Abstract: The healthy communities movement, encompassing grantmakers, advocates, and communities, has been championing healthy eating and active living in the places where people live, work, learn, and play since the mid- 1980s. Given the growing obesity epidemic in the U.S., the movement faces an urgency to transform the health of the nation through community-based policy, systems, and environmental changes at the community level. It is currently comprised of partnerships between multiple funding agencies, including foundations, government agencies, and nonprofit organizations, and thousands of communities across the nation. Important policy and environmental changes have been made in many communities, but the question remains whether funding is reaching those communities across the U.S. most affected by obesity. In this paper we will examine healthy communities funding from 2000 to 2010 for healthy community initiatives, comparing the geographic distribution of foundation and government funding with obesity rates, SES, and race/ethnicity demographic information. |
| Exploring Technology Access and Adoption With Low-Income Families |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carlin Llorente, SRI International, carlin.llorente@sri.com |
| Ann House, SRI International, ann.house@sri.com |
| Gabe Novais, SRI International, gabe.novais@sri.com |
| Patrik Lundh, SRI International, patrik.lundh@sri.com |
| Vera Michalchik, SRI International, vera.michalchik@sri.com |
| Abstract: Evaluation researchers explored the ways low-income families adopted and used technologies after receiving free computers and 3 years of broadband Internet access. The percentage of adults in the study that use the Internet on a daily basis doubled over three-year program. Half of children in the families surveyed also used the computer on a daily basis. Income level, education, race/ethnicity, or whether users were born in the United States are user characteristics often considered in previous literature to be critical variables in technology adoption. In this study, however, these features were NOT significantly related to benefits participants reported from their technology use. Instead, study participants who were most comfortable using computers and the Internet were those who reported more frequent use, a broader range of experiences, and greater benefits from the technology. Importantly, this attitude of comfort is not static, but an attitude that can be fostered. |
| Using Collaborative Evaluation Approaches to Improve Implementation and Understanding of a 3-D Virtual World Initiative in Schools |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Nita Matzen, Appalachian State University, matzennj@appstate.edu |
| Rita O'Sullivan, University of North Carolina at Chapel-Hill, ritao@email.unc.edu |
| Sean Williams, Clemson University, sean@clemson.edu |
| Lauren Kendall, University of North Carolina at Chapel-Hill, lkendall@email.unc.edu |
| Johnavae Campbell, University of North Carolina at Chapel-Hill, johnavae@email.unc.edu |
| Abstract: What are the possibilities and the issues of implementing the use of 3D immersive virtual technology for science, technology, engineering, and mathematics (STEM) learning in middle schools? Will this motivate students to pursue careers in STEM? How does a collaborative evaluation approach increase the potential for successful implementation and contribute to this understanding? This paper uses STEM and ICT 3D Instructional Worlds: The 3D Experience (STEM-ICT 3D), a National Science Foundation Innovative Technology Experiences for Students and Teachers (ITEST) project as the lens to explore these questions. |
| Evaluation Techniques to Assess Behavioral Changes Related to the Adoption of Best Management Practices by New Landowners |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Philip Shackelford, Texas A&M University, p-shackelford@tamu.edu |
| Theresa Murphrey, Texas A&M University, t-murphrey@tamu.edu |
| Jeff Ripley, Texas A&M University, j-ripley@tamu.edu |
| Abstract: This presentation will share the results of evaluations of New Land Owner Training programs delivered by Extension during the time period of 2006 – 2009. Detailed reporting of data collected will be shared to illustrate how evaluation efforts have documented the success of the program. Evaluation techniques to be used to assess behavioral changes as a result of this program will also be shared. Individuals interested in assessing behavioral changes will be interested in hearing about the strategy proposed to determine real impact on behavioral changes. Techniques from the field have been reviewed and a detailed strategy will be presented. |
| Measuring Treatment Outcomes Across Two Substance Abuse Treatment Programs for Adults and Adolescents |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Charles Brown, Centerstone Research Institute, charles.brown@centerstone.org |
| Gisoo Barnes, Centerstone Research Institute, gisoo.barnes@centerstone.org |
| Catherine Sewall, Centerstone Research Institute, katie.sewall@centerstone.org |
| Abstract: This paper describes the evaluation of two evidence-based practices designed to treat adults and adolescents with substance abuse disorders. The presenters will discuss the treatment characteristics, evaluation designs and outcomes associated with treating adults and adolescents with substance abuse disorders. The client outcomes measured include drug and alcohol use, mental and physical health symptoms, and social factors. The data obtained in this paper were collected at baseline and 6-month follow-up periods. Descriptive and Generalized Estimating Equation (GEE) Modeling were used to analyze changes in pre-and post-treatment. In addition to the results, the presenters will discuss challenges and lessons learned that could be beneficial for evaluators of such programs. |
| Validating a Labor Individual Competencies Scale for a Distance Education Program in Brazil |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Victor Arzola, Social Service Industry, Brazil, victor.arzola@sesirs.org.br |
| Miguel Fontes, John Snow, Brazil, m.fontes@johnsnow.com.br |
| Lorena Vilarins, Social Service Industry, Brazil, lorena.vilarins@sesi.org.br |
| Mário Ronaldo Oliveira da Silva, Social Service Industry, Brazil, mrsilva@sesirs.org.br |
| Carla Cristina da Silva de Oliveira, Social Service Industry, Brazil, carla.oliveira@sesirs.org.br |
| Abstract: Objective: To perform a statistical analysis of the internal consistency and validity of the Scale of Distance Education Program for Youths and Adults (EJA-EAD) of SESI/RS. The improvement of the scale’s instrument among students is part of a comprehensive Social Impact Evaluation model. Methods: For each of the eight scale’s composites, an exploratory factor analysis was performed to identify the highest loading variables included in a pre-test questionnaire. The internal consistency criteria were calculated using Cronbach's Alpha test for each component and final scale. Results: Out of the initial 107 variables, subdivided into eight composites, 40 variables were kept in the final scale, considering minimal loading thresholds. The final scale reliability reached 0.838 Cronbach’s Alpha. Conclusion: The scale was validated and will be used to evaluate impact of the Distance Education Program together with routine evaluation of academic performance. |
| Evaluation of Online Extension Programs for Improvement and Accountability: Challenges and Alternatives |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Koralalage Jayaratne, North Carolina State University, jay_jayaratne@ncsu.edu |
| Abstract: There is a growing trend of using the Internet for delivering educational programs. This trend is mainly associated with the efficiency of the Internet in reaching a large number of people within a short period of time. When more and more Extension programs are delivered through the Internet, it is important to evaluate online educational programs to determine teaching effectiveness. This paper presentation discusses potential challenges and practical alternatives for online educational program evaluation with the emphasis on program improvement and accountability. This paper is based on the information gathered from literature, Extension educators and the author’s experience. This presentation has direct implications for any organization delivering educational programs online. This is a timely topic for evaluators who face the reality of evaluating online educational programs. It contributes to the evaluation theory and practice by laying out a framework for the evaluation of online educational programs. |
| Developing Core Competence Indicators for General Education |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Che-Hao Liu, National Normal University, haogogo@ntnu.edu.tw |
| Chin-Wen Chang, Ming Chuan University, |
| Abstract: The purpose of the core competence indicator development program is to enhance the teaching quality of general education. The center of general education implements and administers the program. The program began in the 2009 academic year. The core competency indicators are developed by center’s faculty and 30 instructors who are teaching general education core course in semester 2009~2010. The Delphi inventory applied for establishing the validity and revising the competency indicators regularly. Seven core competency indicators are developed including critical thinking, analytical and conflict resolution skills, moral practice and social concern, multicultural and practice, active exploration, participation and initiative, scientific thinking and application, and aesthetics and taste. Later, the center will build a curriculum map which consists of seven core competency indicators and that provides student a guideline when choosing general education course. The University is expecting these indicators to evaluate students’ learning results and provide teaching direction to instructors. |
| Teaching an Evaluation Graduate Course: Use of Learning Assessment as a Teaching Strategy |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Koralalage Jayaratne, North Carolina State University, jay_jayaratne@ncsu.edu |
| Abstract: Effective graduate courses are necessary to prepare future evaluators. Student learning is the outcome of effective teaching. Educators are exploring practical approaches to enhance teaching effectiveness. We use exams and assignments mainly for assessing student learning. Exams and assignments are parts of lesson plans of any course. Can we use student assessment as an effective teaching strategy to enhance learning? Exploring this question to find practical solutions is the main objective of this paper presentation. This paper critically analyzes the challenges and issues in assessing students for learning outcomes in psychological, historical, technological and practical context. Next, the paper presents practical alternatives for designing and conducting student assessments as teaching strategies to enhance learning outcomes. This paper presentation is based on literature and the presenter’s experience in using this approach to teach an evaluation graduate course. This presentation contributes to evaluation practice by providing practical concepts to use learning assessment as a strategy to enhance teaching effectiveness. This paper is practically significant for the educators who teach evaluation graduate courses. |
| On Q: Promoting Quality in Basic Education Through Positive Incentives |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Monika Schaffner, Planning and Learning Technologies Inc, m_schaffner@hotmail.com |
| Abdel-Kareem Badran, National Center for Examinations & Educational Evaluation, drab1985@yahoo.com |
| Abstract: Funded by USAID-Egypt, the School Team Excellence Awards Program (STEAP) was a four-year, standards-based quality assurance program that employed positive incentives to achieve comprehensive school improvement in 10,500 primary and middle schools across Egypt. The program was tasked to disseminate a culture of quality through self-assessment, team-building, school improvement planning, and monitoring and evaluation in a country where none of that was commonly practiced. This included activating the local communities across the country in participating in the school improvement efforts and obtaining private sector support for these activities. To accomplish this, the STEAP team designed an innovative cascade-model for training and management and published the Egyptian School Quality Manual, the guiding text for all training materials and the school improvement process, and provided guidelines for school and community cooperation toward improving school performance and achieving standards-based quality improvement that would ultimately improve student learning. |
| Strengthening Impact Evaluation Quality in Africa Through Partnerships: A Pathway to Sustainable Development |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Winston Allen, Heartlands International, wallen1521@aol.com |
| Abstract: Evaluation quality directly affects its use as a learning tool. Increasingly, international donor agencies working in Africa are emphasizing the need to learn more about the impact of development assistance on the poor. On the other hand, very little impact evaluations are conducted on development programs in the region. Those that are conducted have been found to be of poor quality. Evaluating impact is critical, particularly when resources are scarce, and every dollar is aimed at maximizing development impact.This paper discusses the role partnerships between international development agencies and African institutions can play towards enhancing the quality and use of impact evaluations in Africa. It explores partnerships as opportunities for capacity strengthening of African institutions, in producing quality impact evaluations, and using evaluations as a mutual learning process, and a critical input to the design of future programs. |
| Improving Effectiveness of Household Treatment of Diarrhea of Young Children in Nepal: Increasing Use of Zinc to Reduce Severity of Diarrhea While Maintaining Use of Oral Rehydration Solution (ORS) to Reduce Dehydration – Results From a Population-based Household Survey |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Prakash Dev Pant, N-MARC Project, nepal51@gmail.com |
| Lonna B Shafritz, Academy for Educational Development, lshafrit@aed.org |
| Peter Oyloe, Academy for Educational Development, poyloe@aed.org |
| Abstract: In Nepal, N-MARC initiated a new multi-pronged intervention (June to October 2009) to expand usage of Baby Zinc alongside Oral Rehydration Solution (ORS) for diarrhea among children under-5 in 27 districts. A population-based household survey of 2310 caregivers (completed January 2010) reveals that use of zinc along with ORS increased from baseline of 17% to 25% among all cases of diarrhea in the prior 2 weeks, and that 88% of zinc users also used ORS. 52% of the total sample was exposed to the promotion of Baby Zinc in the prior 3 months, with 77% of these stating that they thought it was “good medicine” for diarrhea, and 80% believing it to be “affordable/inexpensive”. Bivariate and multivariate analysis shows use of zinc in past 2 weeks was significantly related to exposure to the media campaign, as well as to beliefs of high availability and effectiveness of Baby Zinc. |
| Using Technology to Enhance the Quality of Evaluator-Stakeholder Interactions |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Andrew Lejeune, University of Alberta, alejeune@ualberta.ca |
| Cheryl Poth, University of Alberta, cpoth@ualberta.ca |
| Kathy Howery, University of Alberta, khowery@ualberta.ca |
| Abstract: Integrating technology into an evaluation process is becoming an increasingly viable means of developing and maintaining stakeholder relationships. This session examines the impact of emerging technology as a means of communication within the context of an ongoing evaluation that is focussed on the use of web 2.0 tools in the classroom. The session will describe the shift we experienced from a traditional means of communication (i.e., face to face meetings and formal reporting) to the use of emerging technologies (i.e., videoconferencing and blogging). What is investigated is how technology can both enable and hinder the types of interactions experienced between evaluators and stakeholders. The analysis of the first year of this project reveals several themes including issues around risk factors in communicating in a novel environment, access, usability, and time. The session concludes with an examination of strategies implemented to mitigate barriers and promote quality interactions between evaluators and stakeholders. |
| A Person-Oriented Approach to Evaluation: Profiles of Young Adult Volunteers in a National Nonprofit |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Gretchen Biesecker, City Year Inc, gbiesecker@cityyear.org |
| Tavia Lewis, City Year Inc, tlewis@cityyear.org |
| Abstract: In this paper, we share results of a person-oriented study of 17-24 year-old volunteers serving full-time with City Year, a national, multi-site AmeriCorps program. City Year corps members tutor and mentor students across grades 3-9 and conduct additional service learning programs and events. We examined profiles of our corps members to enhance our ability to recruit, retain, and develop them to maximize our impact in schools and develop high quality programming. Using survey data from over 1500 corps members from the fall of 2009, we conducted a k-means cluster analysis, looking at corps members’ motivations for joining City Year, levels of social support, previous education and leadership experiences, grit (e.g., Duckworth, et al., 2007), confidence, and comfort working in schools. Results identified 4 distinct profiles of corps members, who differed in their demographic characteristics and experiences across their year of service, as measured via subsequent surveys and data. |
| Data Based Discussions: Uncovering the Unexamined Assumptions |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Micah Fierstein, University of Alaska, Anchorage, micah@uaa.alaska.edu |
| Abstract: Teacher and evaluator conversations too often appear to be “loaded” and influenced by participants’ silent assumptions and frames. All too often these assumptions are unconsciously held, and unexamined. They are simply taken as granted as “the ways things are…” (Argyris, C.,1974). These silent assumptions can unwittingly constrain, frame, and ultimately debilitate the teachers’ and evaluators’ perspectives, and their ability to understand and appreciate the data’s underlying meaning. They can limit the teachers’ ability to fully appreciate the situation and opportunities at hand - the teachers’ role and potential for change and to influence the future. These blind spots undermine the teacher’s and team’s capacity to change practice - to make a difference. This paper identifies and examines these silent assumptions, and the influence they can have on a team or individual data-based meaning making in program review and evaluation. |
| Using Archival Public School Data In Cross-State Evaluations |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Philip Nickel, AVID Center, pnickel@avidcenter.org |
| Dennis Johnston, AVID Center, djohnston@avidcenter.org |
| Abstract: Evaluation of public school programs usually requires the use of student data collected by states and districts. These data sources can contain a wealth of process and outcome information on individual students and teachers. However, availability of access can vary greatly between districts or states and data quality varies by the importance placed on each piece of information by those submitting their data. Evaluations involving inter-state program implementations will find some data available in some states, but not in others. This paper discusses the kinds of data available in selected states and how the data can be used. Discussion includes difficulties faced in the use of public school data and solutions found by the authors. |
| Experiences of Program Personnel From an Intimate Partner Violence Prevention Program |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lennise Baptiste, Kent State University, lbaptist@kent.edu |
| Abstract: Q methodology was used to understand the experiences of program personnel from an intimate partner prevention program, who participated in an empowerment evaluation. The presenter will share how she developed and piloted a tool designed to identify the sources and types of learning which these personnel experienced. These personnel were impacted to varying degrees by the evaluation purpose, the evaluation context, the data collection approach, the relationships among stakeholders and their relationship with the evaluation team. The findings revealed three perspectives: increased commitment to the program format; increased commitment to the program format; and increased personal commitment from program personnel. |
| An Empirical Study to Develop and Pilot an Instrument to Validate the Essential Competencies for Program Evaluators |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Yuanjing Xue, University of Minnesota, xuex0026@umn.edu |
| Jean A King, University of Minnesota, kingx004@umn.edu |
| Laurie Stevahn, Seattle University, stevahnl@seattleu.edu |
| Abstract: Contemporary program evaluation emerged in the 1960s, and over the past fifty years the field has moved slowly forward its development as a profession. It has not yet, however, become a full-fledged profession. Evaluator competencies, one of the foundations of this development, demonstrate the knowledge, skills, and attitudes required of program evaluators. The Essential Competencies for Program Evaluators (ECPE), developed by King et al. (2001), represent one of the first efforts at specifying professional competencies. The purpose of this study was to develop a survey instrument to validate the ECPE so that the proposed competencies can be used to enhance the professionalization of the field of evaluation. The proposed paper will present the survey development and the results of extensive testing, including a pilot study and a series of interviews with expert evaluators in three areas. King, J. A., Stevahn, L., Ghere, G., & Minnema, J. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22(2), 229-247. |
| Adherence to the Directly Observed Treatment Short-Course's (DOTs) Strategy: An Evaluative Approach From Actors Perspective |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Gisela Cardoso, National School of Public Health (ENSP/Fiocruz), gisela.cardoso@gmail.com |
| Marly Cruz, National School of Public Health (ENSP/Fiocruz), marly@ensp.fiocruz.br |
| Pedro Paulo Chrispim, National School of Public Health (ENSP/Fiocruz), chrispim@ensp.fiocruz.br |
| Dolores Abreu, Laboratório de Avaliaçăo de Situaçőes Endęmicas Regionais, doloresfabreu@hotmail.com |
| Paula Vita Decotelli, National School of Public Health (ENSP/Fiocruz), paulavita@gmail.com |
| Juliana Borenstein, National School of Public Health (ENSP/Fiocruz), ju_borenstein@yahoo.com.br |
| Abstract: The purpose of this study was to evaluate DOTs (Directly Observed Treatment Short-Course) strategy intervention for tuberculosis in an area of the city of Rio de Janeiro with high prevalence of the disease. Two Health Units, with different types of health system structure (one with health family strategy implemented and the other one with a traditional model) were selected in order to compare acceptability, accommodation and compliance to guidelines, presuming them as being associated to adherence. The study design was case-study, with a mixed-method approach in which phenomena are investigated in real life context by using direct observation, semi-structured interviews and documents analysis. Social representation theory was used in order to understand the meaning patients, their families and health professionals give to DOTs treatment, in order to describe and analyze which factors contribute and which are obstacles to patients adherence, considering health service organization and health professional- patients relationship dynamics. |
| Are Workshops Effective? Assessing the Quality and Impact of Evaluation Training Workshops in New Zealand. |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jeffery Adams, Massey University, j.b.adams@massey.ac.nz |
| Pauline Dickinson, Massey University, p.m.dickinson@massey.ac.nz |
| Abstract: Workers in health and community organizations are increasingly required to have an understanding of evaluation and conduct evaluations of their programs/projects. However, the skills of workers to do this are often limited. One common response is to provide training workshops to develop skills – but are workshops effective? We outline our workshop-based training initiative that seeks to build individual workers evaluation capability and the findings of an evaluation of this initiative. Participant feedback on the quality of the workshops (process evaluation) and data on the implementation of evaluative practices (immediate impact evaluation) will be reported. Our evaluation findings are mixed. Even though high quality evaluation training is provided, some workers do not implement many of the skills they are taught. We identify the barriers and enablers for these workers in undertaking evaluation activity, and discuss ways we have supplemented the training workshop to maximise the utilization of skills and knowledge taught. |
| The Critical Role of Data Use in Instructional Improvement and Innovation |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Xinsheng Cai, American Institutes for Research, ccai@air.org |
| Daniel Aladjem, American Institutes for Research, daladjem@air.org |
| Sarah Bardack, American Institutes for Research, sbardack@air.org |
| Victoria R Marks, American Institutes for Research, vmarks@air.org |
| Abstract: The purpose of the study is to examine the role of data use in teachers’ implementation of inquiry-based instructional practices and instructional innovation. The data for the study were teacher and principal surveys conducted as part of an evaluation of the GE Foundation Developing Futures program. A hierarchical linear model was used to analyze the data. The results of the teacher survey showed that data use strongly predicted teacher use of inquiry-based instructional practices. The use of data in the classrooms was a stronger factor than professional development variables for predicting teacher use of interactive teaching strategies while professional development mediates the relationship between data use and instructional practices. The results of the principal survey showed that teacher innovation was strongly predicted by data use especially data use in the classrooms. The findings emphasize the importance of using data in improving instructional practices and in bringing about instructional innovations. |
| What It Means to be Popular: Exploring Chinese Immigrant Youth's Perception of Peer Popularity |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chuan Chang, University of Hawaii, chuanc@hawaii.edu |
| Abstract: This is a pilot study exploring what constitutes peer popularity for Chinese adolescents and its implications for their school engagement. A sample of 8 recent immigrant students enrolled in four Hawaii public schools were interviewed on their school experiences in China and perceptions of the characteristics of popular students in their schools. An admire survey was also used to collect data on what behavior or attributes the immigrant youths often admire in their peers. In consideration of the specific cultural and developmental contexts of Chinese youth, two issues are predicted to be different from previous research findings based on studies conducted with European and American youths: (1) the Chinese adolescents’ perception of good studentship (good academic performance and good efforts) and leadership (reflected in a sense of responsibility and sacrificing self for the sake of the collective) for popularity, and (2) Chinese peer group’s endorsement of the conventional values promoted by parents and teachers. |
| Longitudinal Case Studies of K-8 Little Engineers to Detroit PreCollege Engineering Program (DAPCEP) Academy of Young Scientists (AYS) |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shannan McNair, Oakland University, mcnair@oakland.edu |
| Jason Lee, Detroit Area Pre College Engineering Program, jdlee@dapcep.org |
| Hawra Abigolal, Oakland University, haabogil@oakland.edu |
| Maria Hewitt, Oakland University, |
| Megan Riley, Oakland University, |
| Abstract: 12 students participating in an NSF funded Detroit PreCollege Engineering Program (DAPCEP) Academy of Young Scientists (AYS) were identified as participants in an earlier DAPCEP K-3 Little Engineers cohort group, beginning at age 5. The Little Engineers was a Saturday half-day cohort group learning experience where parents and children attended together. The AYS program was for students in grades 6-8, again a cohort model. In-depth interviews of students and their parent(s) were conducted a two points during grades 6-8. Archival data (children's portfolio, surveys, grades, work samples) from the K-3 program and observation of students in AYS added to interview data. Findings include profiles of childrens' participation in school and out-of-school STEM and other activities from preschool through grade 8, descriptions of student achievement and engagement in STEM programs K-8, future plans and goals for post-secondary and career, and child and parent perceptions of program impact. |
| A Birdseye View of One Community’s Success in Preventing and Ending Homelessness |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Margot Ackermann, Homeward, margot.ackermann@gmail.com |
| Abstract: The Department of Housing and Urban Development (HUD) requires that most homeless shelters that receive HUD funding track people’s experiences of homelessness using a Homeless Management Information System (HMIS). In Richmond, Virginia, community stakeholders involved in homeless services developed a plan to prevent and ultimately end homelessness. Data collected in HMIS, including whether or not people returned to shelter after exiting, the length of time people were homeless, where people go when they exit a shelter, and the length of time people spent in emergency shelter was used to assess whether the community is succeeding in its efforts to reduce homelessness. Looking at the data from the community level, rather than just at the shelter level, highlights significant opportunities to better understand patterns of shelter use among people experiencing homelessness. The present study offers results that can be used for both advocacy and policy change. |
| Effectiveness Evaluation of Brazilian Non-governmental Organizations (NGO) STD/AIDS Prevention Programs Targeting Men Who Have Sex With Men |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Marly Cruz, National School of Public Health (ENSP/Fiocruz), marly@ensp.fiocruz.br |
| Elizabeth Moreira dos Santos, Oswaldo Cruz Foundation (Fiocruz), bmoreira@ensp.fiocruz.br |
| Paula Vita Decotelli, National School of Public Health (ENSP/Fiocruz), paulavita@gmail.com |
| Sonia Santos, National School of Public Health (ENSP/Fiocruz), soniabsantos@yahoo.com |
| Raquel Torres, National School of Public Health (ENSP/Fiocruz), raquelmct@gmail.com |
| Ana Reis, National School of Public Health (ENSP/Fiocruz), anareis@ensp.fiocruz.br |
| Abstract: A case control study to evaluate the effectiveness of HIV/AIDS prevention programs for MSM by NGO in Brazil and to determine favorable aspects and barriers that affect these actions. Four NGO with successful preventive actions were selected for further study to explain relationship between intervention and actions of prevention among cases and controls and determine effectiveness based on evaluation criteria established. Data was collected through documental analysis and interviews with managers and users. Results show that actions developed by NGO do not necessarily follow what is in the scope of project. Preventive actions are more frequent in areas of interaction between MSM and emphasize on distribution of condoms and educational materials. Adult MSM from more favorable socioeconomic conditions were seen to be more protected than younger MSM at higher individual and social vulnerability. Most NGO don’t have records nor monitor their actions, highlighting need to implement a M&E system. |
| Use of Focus Groups in Research, Technology, and Development (RTD) Evaluation |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jim Ryan, CIRCA Group Europe Ltd, jim.ryan@circa.ie |
| Shannon Townsend, National Research Council Canada, shannon.townsend@nrc-cnrc.gc.ca |
| Mike Fitzgerald, CIRCA Group Europe Ltd, mikefitzgerald@ireland.com |
| Abstract: Focus groups are a powerful research tool for qualitative assessment of RTD issues. Both CIRCA Group and the National Research Council of Canada (NRC) have used focus groups in a range of evaluations on different issues. CIRCA and NRC will review the usefulness of Focus groups as a tool for RTD evaluation, using case studies to illustrate their value. |
| Evaluating School-based Arts Integration Projects: Opportunities and Challenges |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Elaine Walker, Seton Hall University, elaine.walker@shu.edu |
| Martin Finkelstein, Seton Hall University, martin.finkelstein@shu.edu |
| Carmine Tabone, Education Arts Team, cvtab@aol.com |
| Abstract: While the heuristic and pedagogical value of arts integration is intellectually understood, it cannot be said that the empirical evidence on the effects of arts integration has yielded consistent results. The goal of the present paper is to describe the evaluation design and findings from two randomized federally funded arts integration projects in which theatre arts strategies are integrated into the teaching of language arts to students in grades four through seven. The findings from the two trials reveal significant cognitive gains for students in the intervention. Moreover, informmation culled from classroom observations and teacher narratives based on their involvement in action research projects underscored how the arts tap into the multimodal dimensions of how students learn and process information. |
| A Tool for Monitoring and Evaluating Pre-service Teachers in Field Experiences |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Hibbard, Florida Gulf Coast University, shibbard@fgcu.edu |
| Robert Triscari, Florida Gulf Coast University, rtriscari@fgcu.edu |
| Diane Kratt, Florida Gulf Coast University, dkratt@fgcu.edu |
| Abstract: Teacher educators use a variety of different methods for evaluating pre-service teachers. An instrument was created to monitor and formatively evaluate the progress and performance of pre-service teachers for continuous improvement during field experiences. This presentation focuses on the development and implementation of an instrument used to evaluative pre-service teacher performance in the classroom. The instrument collected valid and reliable data used to aid in progress monitoring. The instrument was included as a data collection method for the teacher education unit’s accreditation assessment system. We will explain how this tool can be applicable to identifying characteristics of effective teachers. In addition, lessons learned and future uses of the tool will be discussed. |
| Improving Evaluation Quality: Challenges for Evaluating Community Initiatives During Economic Downturns |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brandy Pratt, Western Michigan University, brandypratt@gmail.com |
| Diane Rogers, Western Michigan University, diane.rogers@wmich.edu |
| Mohammed Alyami, Western Michigan University, mohammed.alyami@wmich.edu |
| Abstract: Communities across our nation are struggling to maintain previous levels of services and continue initiatives that began well before the current recession. Many of the initiatives begun before the recession include mandates for funding by the grantors. However, many of these mandates do not include additional funds to support comprehensive evaluation, leaving organizations scrambling to meet requirements and successfully implement initiatives. In this paper, we explore how a community wide initiative to eradicate racism deals with the challenges of implementing high quality evaluation during hard financial times. We propose a community participatory model for how high quality evaluation can be achieved in this context. |
| Evidence-based Guidelines: Evaluating Diffusion in State Tobacco Control Programs |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Max Bryant, Washington University in St Louis, mcbryant@wustl.edu |
| Stephanie Herbers, Washington University in St Louis, sherbers@wustl.edu |
| Lana Wald, Washington University in St Louis, lwald@gwbmail.wustl.edu |
| Douglas Luke, Washington University in St Louis, dluke@wustl.edu |
| Jennifer Hobson, Washington University in St Louis, jhobson@wustl.edu |
| Abstract: There has been little research on how evidence based interventions are disseminated and utilized among state tobacco control programs. To begin to answer this question, the Center for Tobacco Policy Research at Washington University in St. Louis is conducting a multi-year evaluation in partnership with the CDC Office on Smoking and Health. The primary aims of this project are to examine how states are using the Best Practices for Comprehensive Tobacco Control Programs and other evidence-based guidelines for their tobacco control efforts and identify opportunities that encourage guideline use. This presentation will provide background on the field of implementation evaluation, the framework and methods used for this evaluation, and key findings. Lessons learned from assessing diffusion of evidence-based guidelines will also be shared. |
| So Much Information, So Little Time: Evaluating the Impact of Using Brief Addiction Curriculum Infusions as a Teaching Method in Criminal Justice, Nursing, and Social Work Courses |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Joyce Hartje, University of Nevada, Reno, jhartje@casat.org |
| Nancy Roget, University of Nevada, Reno, roget@unr.edu |
| Michelle Berry, University of Nevada, Reno, mberry@casat.org |
| Abstract: The lack of substance abuse and addiction-related academic education leaves students underprepared to enter professional practice and respond to the needs of substance abusing clients. To bridge this gap, brief research-based neuroscience of addiction and FASD curricula were developed and integrated into existing criminal justice, nursing, and social work courses. This presentation will highlight evaluation research activities conducted 1) to gauge the penetration of substance abuse/addiction-related content using a national sample of academic addiction counselor education programs; and 2) the impact of infusing brief, intensive curricula on students’ knowledge and attitudes about substance use disorders (SUDs). Results lend support to the effectiveness of infusing brief research-based curricula into existing academic courses on increasing knowledge and reducing stigmatizing attitudes related to individuals with substance use disorders. Thus, the findings from these studies could have long-term implications for preparing helping professionals work with individuals who have SUDs. |
| Assessing the Quality of Instrumentation Using IRT |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristina Mycek, State University of New York at Albany, km1042@albany.edu |
| Susan Rogers, State University of New York at Albany, susan.rogers.edu@gmail.com |
| Abstract: As evaluators we are constantly creating new instruments to measure outcomes and indicators. As part of this process it is important to assess the reliability and validity of such instrumentation to preserve high quality results. This paper investigates different ways to evaluate reliability and validity of assessments. Several instruments are looked at using percent agreement, Kappa, and Intra-class Correlation Coefficients (ICC) for inter-rater reliability. These types of reliability are often used with observations and rubrics. Reliability for surveys often uses internal consistency measures like Cronbach’s alpha, as well as parallel-forms and test-retest. One option that is often underutilized is Item Response Theory (IRT). This paper will demonstrate the usage of IRT to assess quality of instrumentation in evaluation. |
| Using the Leprosy Control Program evaluation in a Timely |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Elizabeth Moreira dos Santos, Oswaldo Cruz Foundation (Fiocruz), bmoreira@ensp.fiocruz.br |
| Maria Aparecida Patroclo, National School of Public Health (ENSP/Fiocruz), ma.patroclo@uol.com.br |
| Rafaela Souza, National School of Public Health (ENSP/Fiocruz), rafaelabcs@globo.com |
| Ana Reis, National School of Public Health (ENSP/Fiocruz), anareis@ensp.fiocruz.br |
| Gisela Cardoso, National School of Public Health (ENSP/Fiocruz), gisela.cardoso@gmail.com |
| Abstract: It was made in 2009 an evaluative study about the degree of implementation of the brazilian National Leprosy Control Program (NLCP) in 09 states and 23 cities to answer 23 evaluative questions defined by stakeholders distributed according to 05 program operational components: (1) epidemiology, (7) management, (10) integral assistance to leprosy patients and their families, (2) education and communication and (3) research. The analyzed dimensions included: epidemiology - the use; management – compliance; integral care - accessibility, availability, compliance and accommodation; education and communication – effectiveness; research - sustainability, compliance and use. The results have been discussing and using by the Brazilian Ministry of Health to adopt general measures in the country and by states and cities to adopt specific measures. |
| A Commented Review on Cost-Effectiveness of Directly Observed Treatment (DOTs) in Tuberculosis Control Programs |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pedro Paulo Chrispim, National School of Public Health (ENSP/Fiocruz), chrispim@ensp.fiocruz.br |
| Luisa Oliveira, Fluminense Federal University, luisa,dutra@gmail.com |
| Elizabeth Moreira dos Santos, Oswaldo Cruz Foundation (Fiocruz), bmoreira@ensp.fiocruz.br |
| Abstract: Tuberculosis Directly Observed Treatment-short course (DOTs) is believed to ensure a better control of the disease on population. Following World Health Organization guidelines, DOTs is first-choice strategy to treat tuberculosis (TB) no matter what local country or population conditions are. Cochrane’ systematic review recently demonstrated that the effectiveness of DOTs and self-administered strategies don’t differ. In order to review published evidences on health economic of the DOTs, it was made a search on web databases for articles on cost-effectiveness of TB Control Programs comparing DOTs and self-administered strategies. Pubmed, Scopus, BVS, Evidence Portal, Scielo and CRD were searched. Despite the explicit WHO goal of to reduce the abandon rate, this was not an evaluated outcome. Rather, the studies used quality or disability adjusted life years as a measure of effectiveness. In all studies, DOTs was more cost-effective. |
| Lessons Learned From Community Perceptions of Program Evaluation |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Leslie McCallister, East Tennessee State University, mccallis@etsu.edu |
| Abstract: Stakeholders play an important role in program evaluation, especially in a community agency setting. These stakeholders are often central in the process of providing background information to evaluators, as well as ensuring appropriate buy-in to the evaluation process. However, these stakeholders also often lack relevant knowledge about the evaluation process, which can hinder effective program evaluations. This paper examines program evaluation conducted for community agencies, specifically focusing on community agency members’ understanding of program evaluation, how it can impact their organization, and how this understanding impacts the evaluation process. This knowledge will enable evaluators to design and implement evaluation strategies that will better address the needs of community organizations. |
| Using Mixed Method Designs in Evaluating the Effectiveness of Supplemental Instruction Program at a Community College |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sunny Liu, University of Southern California, luliu@usc.edu |
| Abstract: Based on the conceptual framework and data analysis strategies for mixed-method evaluation designs proposed by Greene, Caracelli, and Graham (Greene, Caracelli & Graham, 1989; Caracelli & Greene, 1993), this paper presents an empirical study of applying mixed-method designs in the evaluation of the Supplemental Instruction (SI) program in a community college setting. Four purposes for mixed-method evaluation designs including triangulation, complementarily, development, and expansion were pursued in order to investigate the optimal design characteristics that can best address the research questions of how the unique characteristics of community college students impacting the implementation of SI and what strategies are effective in corresponding to these uniquenesses. Data have been collected longitudinally over three semesters starting from Spring 2009 to Spring 2010 and both the quantitative and qualitative methods have been implemented concurrently and sequentially in the course of the study. It presents a promising new way to mix the methods so that it has “complementary strengths and nonoverlapping weakness” of either quantitative or qualitative method (Johnson & Turner, 2003) |
| Development of an Implementation Fidelity Framework: Implications for District-Wide Multi-classroom Interventions |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Cindi Dunn, Kansas State University, ckdunn@ksu.edu |
| Sheryl Hodge, Kansas State University, shodge@ksu.edu |
| Abstract: Evaluators at the Office of Educational Innovation and Evaluation (OEIE) sought to develop an implementation fidelity framework to encompass the vastness of a federally funded multi-year, multi-classroom, multi-intervention initiative. In establishing the framework, challenges emerged surrounding the need to provide experimental design-based impact for each of the treatment interventions (i.e., nine), while also developing core protocols for parsimony among interventions. This task included detection of overlap across project goals and objectives, as well as differentiation of objectives that were embedded within fidelity of implementation, process, and summative components. Implicit in this framework is a generalizable, functional model that may be used as a resource for evaluators who are charged with the mission of operationalizing key implementation fidelity challenges in the development of complex evaluation plans. As such, the purpose of this presentation is to provide significant strategies for addressing multiple treatment fidelity in a longitudinal evaluation design. |
| Teaching Program Evaluation to Novices: Steps, Pitfalls, and Lessons Learned |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Rogers, State University of New York at Albany, susan.rogers.edu@gmail.com |
| Abstract: The current paper presents a case study outlining the author’s foray into the teaching of Program Evaluation to novices. Having worked in Program Evaluation and taught other courses in higher education, the author was excited when offered the opportunity to teach the first course in a program evaluation concentration offered to graduate students in education. Using a problem-based approach, students worked to evaluate an on-going after-school program under the guidance of the instructor. While the end-product – a formative evaluation report delivered to the client – was acceptable, getting there was an adventure; introducing novices to the disciplinary thinking required by evaluators requires a certain type of thinking all its own! This paper outlines challenges, road-blocks, lessons learned, and recommendations for those considering the dual role of evaluator and educator. Implications are relevant to evaluators from any field of expertise who may find themselves in an educative role. |
| Investigation Into Delivery Modes of Adult Education Programs by a Cooperative Extension |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jane A Rodd, State University of New York at Albany, jr937855@albany.edu |
| Dianna L Newman, State University of New York at Albany, dnewman@uamail.albany.edu |
| Abstract: The Cooperative Extension system is the largest non-formal adult education system in the USA. Extension is in competition for public education funding against other organizations offering formal and non-formal education (McGrath, Conway and Johnson, 2007). Extension professionals have identified that, to remain viable and be maximally effective, programs must be delivered in ways consistent with learning theories (Prawl, Medlin and Gross, 1984; Richardson, 1994; Franz, 2007). This study investigated whether agricultural Extension programs in a representative Cooperative Extension in New York State were delivered in line with principles of androgogy and transformative learning. This paper reports on findings from the evaluation of a random sample of all agricultural and horticultural educational programs delivered by the Extension in 2009. Programs were categorized by level of instructor, group size and intended outcome. Dependent variables were constructs measuring androgogy and transformative learning. Findings will report on variability in program delivery. |
| A Stakeholder-based Evaluation on the Policy of Secondary School Teacher Appointment Examination in Korea |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ye-lim Yu, Seoul National University, dream81@snu.ac.kr |
| Sun-Geun Baek, Seoul National University, ruilin45@naver.com |
| Abstract: The purpose of this study is to evaluate a policy of secondary school teacher appointment examination in Korea with different stakeholder’s perspectives using PUPA model. PUPA model reflects multiple evaluation perspectives of four different groups of stakeholders: Providers, Users, Professionals, and Authorities. Based on the literature reviews and in-depth interviews with 20 stakeholders, evaluation indicators of the policy were developed for each stakeholder group as follows: ? Provider: validity, efficiency, democracy, ? User: satisfaction, fairness, convenience, ? Professional: appropriateness, expertise, effectiveness, and ? Authority: feasibility, legality, public interest. And each stakeholder group evaluated the policy by themselves based on their own evaluation indicators. The subjects of this study were 228 stakeholders. Results indicated that evaluation scores were significantly different among four groups of stakeholders (F=11.30, p<0.01). It suggests that the stakeholder-based evaluation is needed to ensure evaluation quality and balanced evaluation. |
| Developing an Objective Measuring of Tempo Consistency |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tara Pearsall, Savannah College of Art and Design, tpearsal@scad.edu |
| Abstract: Traditionally, music education researchers have measured tempo consistency as a function of the level of agreement among raters. Unfortunately, researchers has shown that raters often experience difficulty agreeing on students’ level of tempo consistency (Schneider, Johnson, & Porchea, 2004; Yap, 2003; Yap, Lewis, & Feldon, 2006). Results from a study by Schneider et al. (2004) indicated that even a 5-judge panel would not produce sufficient reliability to accurately reflect students’ ability to establish and maintain a consistent tempo. The purpose of this study was to develop a procedure appropriate for isolating and objectively measuring the tempo consistency of students’ recorded rhythm performances. The information gained by using such a measure may be of use not only to those interested in measuring rhythm performance achievement for accountability purposes, but also to music educators seeking to improve the rhythm instruction they provide in their classrooms (Schneider et al., 2004). |
| Invited to the Program Planning Dance But Going Home Alone: A Novice’s Strange Journey |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ottis Murray, University of North Carolina at Greensboro, ottis.murray@uncp.edu |
| Abstract: In 2008, Robeson Community College, located in Lumberton, NC, was successful in securing funding from the Kate B. Reynolds Foundation. This support was granted to spearhead an effort to develop a comprehensive strategy to address substance abuse in Robeson County by bringing together the key stakeholders and providers into a developmental and planning matrix. It was believed that collective participation would assure ownership; consequently, collaborative engagement would provide a new framework to address the issue(s) of substance abuse. A strategic plan was developed, which recommended a substance abuse coalition be established to provide for a coordinated process/response to the growing problem of substance abuse. However, the outcome (to date) is this jointly developed plan now sits on the graveyard shelf along side other well-intended, misplaced dead trees. This paper explores the role of an evaluator in facilitating the program planning process and the lessons learned along the way. |
| Linking Communities to Advance Evaluation Capacity Building in Historically Black Colleges and Universities (HBCUs) |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ruth Greene, Johnson C Smith University, rgreene@jcsu.edu |
| Abstract: This paper describes evaluation capacity building (ECB) in Historically Black Colleges and Universities in light of human resource development, partnership development and organizational development and mission. Using video clips and data from HBCU evaluation learning communities of practice, the presentation provides an understanding of the resources, challenges and opportunities for advancing evaluation capacity building in these Universities. |
| Implementation Fidelity of Environmental Strategies: The Strategic Prevention Framework – State Incentive Grant (SPF-SIG) Experience |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Phillip Graham, RTI International, pgraham@rti.org |
| A Monique Clinton-Sherrod, RTI International, mclinton@rti.org |
| Abstract: The Strategic Prevention Framework – State Incentive Grant (SPF-SIG) is the Center for Substance Prevention’s (CSAP) flagship initiative to reduce the onset and progression of substance abuse across the lifespan. A key component of the initiative encourages the implementation of evidence-based environmental prevention strategies to address specific contributing factors to substance use. However, determining if these strategies have been properly implemented has provided significant challenges to program evaluators. In response to this challenge, a group of SPF-SIG evaluators developed a set of fidelity tools to assess the implementation of environmental strategies. To test the efficacy of these tools, this presentation shows findings from the evaluation of Nebraska’s SPF-SIG program. The Nebraska SPF-SIG funds sixteen (16) communities across the state to address three statewide priorities: 1) underage drinking, 2) binge drinking among 18-25 year olds, 3) and driving while impaired. Recommendations for instrument improve are discussed within the context of SPF-SIG. |
| Comparing Comparisons: Contrasting Ethnic-Specific Factor Coefficient Weighted Depression With the Invariant Unit-Weight Measure and Comparing Puerto-Rican and African-American Elderly on Depression Development Processes |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Emil Coman, Institute for Community Research, comanus@netscape.net |
| Jay Schensul, Institute for Community Research, jean.schensul@icrweb.org |
| Abstract: This paper proposes a systematic procedure for assessing differences in measurement properties among ethnic groups and then integrating established measurement differences in further ethnic comparisons. It illustrates formal tests of strict and weak measurement invariance in a continuous latent measure composed of binary indicators (items) and compares the impact of computing factor score coefficients-weighted total individual scores and software estimates scores to ethnic-blind (invariant) unit weight scores, on comparisons between ethnic groups.We compared the same measures across ethnicity, which ultimately yields a set of 4 group covariances comparisons (hence ‘comparing comparisons’) comparing the differences between ethnicities for each measurement type first then, these differences across measurement type, and also comparing differences between measurement type first for each ethnicity, then comparing these found differences across ethnicity. |
| Assessing the Impact of Career Academies on Student Achievement |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Becky Smeardon, Quill Research Associates, bsmerdon@qra-llc.net |
| Frances Burden, Quill Research Associates, francesburden@aol.com |
| Aimee Evan, Quill Research Associates, aimee_evan@yahoo.com |
| Rheta Lanehart, University of South Florida, rlanehar@cas.usf.edu |
| Kellie Kim, Quill Research Associates, kkim@qrc-llc-net |
| Abstract: Career academies are a promising program that typically operate as a school-within-school, providing equal emphasis on academic and vocational curricular content, serving the needs of students with a broad range of academic levels. This study examines enrollment in Florida’s career academies, particularly in STEM-themed career academies. Specifically, we examine the types of career academies that exist, the types of students that enroll in career academies, and whether different types of students are drawn to certain types of career academies. The objectives of the evaluation are to: (1) evaluate the types of career academies that exist across Florida (e.g., What themes are offered?) and (2) evaluate the impact of different types of career academy attendance on student outcomes (e.g., achievement, college readiness, graduation). Findings from this study will provide audiences with insight into how career academies operate, how they impact student outcomes, and the inherent challenges of evaluating state-wide programs. |
| Impact of School Based Evaluation on Learning Outcomes |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Undurthy Lakshmi Narayana, Regional Institute of Education, undurthy@gmail.com |
| Abstract: The main components of school based evaluation were scholastic areas, co-scholastic areas, personal social qualities and monitoring. School based evaluation scheme was implemented in selected schools and was continuously and systematically monitored. A three-tier system of monitoring was followed during implementation of the scheme. The school-based evaluation had been implemented in schools under Central Government and State Government. The overall results on scholastic attainments reveal that school based evaluation scheme contributed to enhance the learning attainments in scholastic areas. School based evaluation scheme helped in mitigating the percentage of non-masters. Supremacy of Central Schools over state schools was evident. This may be due to context variations. Improvements in state schools are encouraging in spite of so many constraints teachers of state schools faced during implementation of scheme. |
| Assessing the Impact of a National Teacher Evaluation Pilot Program |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Hui-Ling Pan, National Taiwan Normal University, joypanling@yahoo.com.tw |
| Abstract: Enhancing student achievement has become a main task of education reforms in the world. And teacher quality is the key. In order to establish teacher professional image, and promote teacher professional development, teacher evaluation has been implemented. Situated in this global village, Taiwan embarked on a pilot program of teacher evaluation from 2006 to 2009. In order to explore if the program works, an impact assessment was appointed by the government. For conducting the assessment, a questionnaire, covering the aspects of program supporting system, program activities and program outcomes, was designed. Collecting survey data from school administrators and teachers in the pilot schools, the study got 2,316 respondents. Two approaches were employed to evaluate the program impact. Firstly, three groups were classified as high, medium and low achievers in terms of their program activities. Then the program outcomes were compared among the three groups. Secondly, controlling the background variables of the respondents, regressions models were analyzed to see how many program outcome variances might be explained by program activities. The findings indicated that the pilot program of teacher evaluation had its impact on teachers’ growth and the changes of the school ecology. |
| Enhancing the Quality of a National Multi-site Evaluation Through Involvement of Site Participants |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Frances Lawrenz, University of Minnesota, lawrenz@umn.edu |
| Abstract: The purpose of this presentation is to describe how participants at local sites in the large multi-site evaluation of the NSF Noyce Teacher Scholarship Program were involved in the evaluation and how that involvement impacted evaluation quality. Several techniques were used to involve local participants including provision of extensive web site based resources, several video conferences, national and regional face-to-face meetings, and individual communication. The web site with its searchable data bases of literature and local evaluation efforts was used extensively. Despite some limitations, video conferences provided two way discussion of evaluation goals and methods and a mechanism to receive group feedback during instrument development. Face-to-face meetings had mixed effects and needed to be structured carefully to contribute to enhancement of the evaluation. Overall, the quality of the evaluation was enhanced through increased response rates, more valid instruments and the inclusion of multiple perspectives in the interpretation of the results. |
| The Strategy for Implementation of Evaluation Methods in the Specific Conditions of Universities for the Increase of Evaluation Culture and Successful Implementation of Their Development Projects: The Case Study of Institute of Tropics and Subtropics in the Czech University of Life Sciences in Prague, Czech Republic (ITS CULS) |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jirina Svitakova, Czech University of Life Sciences, jirina.svitakova@gmail.com |
| Abstract: Teachers and students of the ITS CULS have realized more than twenty projects in developing countries. The research among stakeholders realized by surveys, focus groups and interviews demonstrated that the evaluation culture is significantly low and the monitoring and evaluation (M&E) methods are generally overlooked. The strategy to reform this situation involves: accreditation of a subject focused on M&E for students, expert advisory for project managers and system of information for donors. The important innovation is the solution of the project members turnover – the greatest weakness common for university teams of development projects. When using an IT project management program, the large number of cooperating people become an advantage. With this approach, dozens of students can be involved in the projects to collect data for M&E. It establishes the important link between theoretical and practical training and there are expected innovations also in M&E methods when diploma thesis are realized. |
| How Much Do HIV/AIDS Service Providers Evaluate Their Programs? |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kylie Hutchinson, Community Solutions Planning & Evaluation, kylieh@communitysolutions.ca |
| Bubli Chakraborty, Provincial Health Services Authority of Bristish Columbia, bchakraborty@phsa.ca |
| Francisco Ibanez-Carrasco, Provincial Health Services Authority of Bristish Columbia, francisco-ibanez@shaw.ca |
| Abstract: The British Columbia Provincial Health Services Authority conducted a needs assessment regarding the use of outcome evaluation among HIV/AIDS service providers. Although organizations recognize the need for accountability, outcome evaluation is usually conducted on a minimal and ad-hoc basis due to various constraints including a lack of time, staff, funding, expertise, common outcome measures, and information technology. Other disincentives include different reporting requirements among funders and not seeing their evaluation reports used in meaningful ways. Many service providers do not make a distinction between ‘evaluation’ and ‘reporting to the funder’ and view it as an exercise for the funder rather than an activity that benefits the organization as a whole. Suggestions for increasing outcome evaluation include more financial resources, a provincial evaluation resource person(s), standardized data collection and reporting forms, outcome evaluation software, information technology upgrades, clear and realistic guidelines from funders, greater knowledge exchange, and staff training. |
| Workable Strategies for Assessing Change Among Health Care Professionals Following Continuing Education |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Heather Becker, University of Texas, Austin, heatherbecker@mail.utexas.edu |
| Kristin Hamlett, Nurse Oncology Education Program, khamlett@texasnurses.org |
| Abstract: Those delivering and evaluating professional continuing education programs are challenged to show changes in practice following educational interventions. In this presentation we describe a low-cost method of assessing perceived change in nursing knowledge and practice following both traditional speaker presentations and on-line professional education modules. Approximately 3-6 months after an educational intervention, a short email and electronic survey are sent to participating nurses asking them to rate their knowledge of major content before and after the intervention and to indicate how they have used the information in practice. Results generally show that nurses identify educating patients, other health care professionals, and the public as the most frequent outcomes. The findings also provide insights into those aspects of training that nurses report fewer opportunities to incorporate into practice. These areas may necessitate systems rather than individual provider level interventions. We will provide sample follow-up questions to illustrate the methodology. |
| The Amazing You: An External Evaluation of Museum-based Exhibits About the Human Body |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Wendy Dickinson, Ringling College of Art and Design, wbdickinson@aol.com |
| Bruce Hall, University of South Florida, bwhall@tampabay.rr.com |
| Abstract: The Museum of Science & Industry (MOSI) created The Amazing You exhibition, a unique multi-sensory experience about the human body and the preventive healthcare measures appropriate to each life stage—from conception and birth through late adulthood. The Amazing You consists of seven integrated exhibits: each interactive exhibit allows visitors to immerse themselves within a simulated medical environment, and provides opportunities to explore current medical technologies and procedures. Formative and remedial/summative evaluation procedures were implemented to assist in the development of the various technology exhibits, and examine the completed exhibits for their effectiveness. Data from pilot trials, unobtrusive on-site observations, on-site personal interviews, and post-pre surveying provided an evaluative basis on which decisions were made affecting the development of the medical technology exhibits. This presentation explores the roles of evaluation in the development of these medical technology exhibits—from planning, through initial tryout, to full operational status at the museum. |
| A-Tv-S-Gv Capacity in Cherokee Community Organizations |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Mark Parman, Cherokee Nation, mark-parman@cherokee.org |
| Ronetta Briggs, O Si Yo Communications, rbriggs@osiyo.biz |
| Abstract: In the 20th Century, Cherokee communities endured US public policy that can be characterized as at best neglect and occasionally out right attack. To counteract this, the Cherokee Nation’s Community Services Department created, in 2004, the Community Organization Training and Technical Assistance (COTTA) program. COTTA was designed to help community organizations increase their capacity to provide programs their communities deemed important. The A-Tv-S-Gv curriculum was created to focus COTTA technical assistance. This curriculum condensed the trainings provided by COTTA into a repeatable, comprehensive program. The curriculum is designed to strengthen four areas of the board’s capacity; Leadership and Management, Planning and Effective Communication, Finance and Fundraising and Technology. Three stages of A-tv-s-gv are recognized for each area; Seedling, Sapling and Tree. An assessment tool was developed to individualize technical assistance for each community organization. The curriculum, capacity assessment tool and baseline data from the organizations will be presented. |
| The Assessment of a Professional Development Workshop: Emic and Etic Perspectives on Junior Faculty Experiences |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Deborah Bennett, Purdue University, bennett@purdue.edu |
| Susan Geier, Purdue University, geiers@purdue.edu |
| Abstract: This paper explores the application of anthropological concepts of insider (emic) and outsider (etic) perspectives in the evaluation of a professional development workshop for racially and ethnically underrepresented junior faculty. Using Pike’s (1957) conceptualization of a “stereoscopic window on the world,” we demonstrate how emic knowledge can be captured by asking insiders (underrepresented senior faculty) to identify probing focus group questions to elicit deeper understanding of the experiences of junior faculty. Etic understandings, on the other hand, are captured through rating scales that represent universal or outsider perspectives of the tenure and promotion process. Together, responses provide a more complete and nuanced view of the experience of underrepresented junior faculty in the academy |
| Comparing the Use of Asynchronous Online Groups With In-Person Focus Groups as an Evaluation Method |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| K Anna Douglas, Purdue University, douglask@purdue.edu |
| Carrie Wachter Morris, Purdue University, cwachter@purdue.edu |
| Abstract: The purpose of this study was to examine the differences in information obtained through the use of online asynchronous focus groups and traditional in-person focus groups in evaluating a learning tool for school counselors. The participants’ responses were analyzed through a qualitative research approach. Implications for designing asynchronous online focus groups are discussed. |
| An Evaluation Study of a Multicultural Student Mentoring Program |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kai Chi Yam, Washington State University, kai.yam@email.wsu.edu |
| Gitanjali Shrestha, Washington State University, gitanjali.shrestha@email.wsu.edu |
| Laura Hill, Washington State University, laurahill@wsu.edu |
| Abstract: Researchers have cited academic and social integration as key factors in increasing academic achievement and retention; however, multicultural undergraduates have received less research attention. A large public university has been implementing a student mentoring program to increase multicultural students’ academic achievement and retention. Forty-eight student mentors and over 800 mentees participated in the program in 2009-2010. We report on a process and a formative evaluation with a quasi-experimental design to assess program fidelity and the association of program activities with outcomes. Original items and a well-validated measure, the Institutional Integration Scale, will be distributed to all multicultural students. We will collect information regarding frequency and types of contact between mentors and mentees, as well as academic and social integration among mentees. The present evaluation has important methodological and conceptual strengths including the use of a well-validated self report survey, quasi-experimental design, and emphasis on a vulnerable and infrequently studied population. |
| Examining Aspirations and Expectations for Youth in Foster Care: An Evaluation of Kansas Kids at Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rhonda Lewis-Moss, Wichita State University, rhonda.lewis@wichita.edu |
| Chris Kirk, Wichita State University, cxkirk@wichita.edu |
| Corinne Nilsen, Wichita State University, corinne.nilsen@wichita.edu |
| Delta Q Colvin, Wichita State University, delta.colvin@wichita.edu |
| Abstract: Obtaining a college degree may be the most significant predictor of getting out of poverty, particularly for youth in foster care. In 2002, a program designed to increase awareness about college was created to address this problem for low-income and youth in foster care. The purpose of this evaluation was to determine whether participating in the Gaining Early Awareness and Readiness for Undergraduate Programs in Kansas met its objectives. The goal of the evaluation was to assess the knowledge gained from baseline to follow-up concerning financial aid and college requirements. A total of 865 youth in foster care and limited income were included in this sample. The results showed that student’s knowledge of college requirements and financial aid did increase significantly from baseline to follow-up from 55% at baseline to 63% at follow up and 51% at baseline to 62% follow-up, respectively. Limitations will be discussed. |
| Three Levels of Change on a Retrospective Post as a Program Improvement Tool |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paul Pope, Texas A&M University, ppope@aged.tamu.edu |
| Abstract: This poster presents the three levels of change that can be produced from retrospective post data, and how these levels can be used for reporting and program improvement. Three frequency tables summarize movement, in increasing detail, for each content item on a before vs. after scale of understanding found on a typical retrospective post. A SPSS script generates the tables. The first level of change divides movement of the scale from before to after into one of three categories (positive movement, no movement, negative movement). The second level shows positive and negative movements by the number of levels (steps) of change. The third level presents actual movement on the scale, from the before level to the after level. Taken together, the three levels of change present a more vivid picture of movement among participants, before vs. after the program, and will help educators make precise adjustments to program content. |
| Another Side of Program Effectiveness: Percent Potential Improvement |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paul Pope, Texas A&M University, ppope@aged.tamu.edu |
| Abstract: This poster presents the calculation and use of percent potential improvement as a sometimes overlooked method of demonstrating a program’s effectiveness in producing outcomes such as knowledge gain or attitude change. Percent potential improvement uses a simple calculation that focuses on participants who have the potential to change in a positive direction; that is, those who answered incorrectly or in an undesirable way on a pretest or pre-program score. While overall change on a particular facet of a program may be relatively small, percent potential improvement can be use to demonstrate that the program was successful among those who were “changeable.” |
| Evaluating Social Media Use and Nonprofits’ Quest for Effectiveness |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristen Cici, University of Minnesota, denz0018@umn.edu |
| Abstract: Over the past decade we have seen the rise of social media and its importance in our everyday lives. Nonprofit organizations have begun to invest time and money into using social media tools, often with little to no understanding of the potential impact on their organizations. This leads nonprofit organizations to ask about how they can best evaluate their social media efforts. My research seeks to address this question and provide insight into how nonprofit organizations use social media and how they are currently evaluating their impact. Methods for this study will include a state-wide survey of over 2,000 nonprofit organizations, 10-15 interviews, and an in-depth look at three nonprofit organizations via case studies. |
| Evaluating Communities of Practice in Academic Medicine |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Dorene Balmer, Columbia University, db2595@columbia.edu |
| Boyd Richards, Columbia University, br2217@columbia.edu |
| Rita Charon, Columbia University, rac5@columbia.edu |
| Abstract: In this poster presentation, we will share how we investigated the process of building and maintaining a community of practice (COP), i.e. a group of people who collaborate to carry out common activities for a common purpose, among medical school faculty. We will also share our exploration of the COP’s impact on curricular decisions and on the institution in which it is situated. We systematically collected data from meeting minutes, participant observation and in-depth interviews with members of the COP of interest, whose explicit aim is to improve teaching social and behavioral sciences through narrative methods. For the purpose of comparison, we interviewed members of another COP, whose explicit aim is to improve career advising for medical students. Using ethnographic and case study methods from the qualitative tradition, we went beyond analysis at an individual level and discovered the breadth of impact of COPs at a curricular and institutional level. |
| Evaluating Social Networks using InFlow and Network Genie |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Oliwier Dziadkowiec, Wichita State University, oxdziadkowiec@wichita.edu |
| Trisha Peaster, Wichita State University, trisha.peaster@wichita.edu |
| Scott Wituk, Wichita State University, scott.wituk@wichita.edu |
| Abstract: Wichita State University conducted a social network analysis of a local coalition with 138 active members. This presentation will discuss methods used and insights gained on the process of evaluating social networks using InFlow and Network Genie software programs. Presenters welcome feedback and insights gained from others with experience in social network analysis. |
| A Cost Analysis of the Tandem Teen Parenting Program |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Toni Watt, Texas State University, tw15@txstate.edu |
| Abstract: The Tandem Teen Prenatal and Parenting Program provides medical, mental health, educational/vocational services and social support to pregnant/parenting teens. The program aims to improve the health and well-being of teen mothers and their children and to reduce the incidence of additional pregnancies for these young women. This study provides a cost analysis of the Tandem program, examining costs in three ways a) cost comparison b) cost effectiveness, and c) cost-benefit. Results reveal that Tandem is a moderately priced program. Costs are only slightly higher than existing social service models and less than half that of the Nurse Family Partnership, the most widely endorsed medical model for young parents. Results also suggest that Tandem is a cost effective program and that it provides a substantial return on investment (ROI=$2.34). These results suggest that Tandem is a wise investment, making a contribution to teen mothers, their children, and the community at large. |
| Energy Program Evaluation Building Blocks |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lark Lee, PA Consulting Group, lark.lee@paconsulting.com |
| Laura Schauer, PA Consulting Group, laura.schauer@paconsulting.com |
| Abstract: The Energy Program Evaluation Building Blocks poster will highlight fundamental concepts and issues involved in the evaluation of energy services programs and how evaluation is benefiting stakeholders by ensuring better and more cost-effective energy programs. The poster will highlight the following: 1. Why evaluation is important in the energy services industry 2. The importance of a comprehensive approach to energy program evaluation that includes both process and impact evaluation. 3. The ‘who, what, when, and how’ of market assessment and market transformation evaluation 4. Definition and categorization of Demand Side Management (DSM) program strategies 5. The utilization of a theory-based evaluation approach 6. The key factors that produce an effective process evaluation 7. Appropriate impact evaluation approaches |
| International Evaluation: Lessons Learned From Collecting Data in Kenya |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shayne Harrison, University of Tennessee, Knoxville, sharrison1976@comcast.net |
| Abstract: In 2009, the CDC contracted with The Oak Ridge Institute for Science and Education (ORISE) to assist in the development of an infection control training course and evaluate its ability to eliminate cases of tuberculosis in Kenya. The evaluation for this training was limited by a number of factors, largely due to limited resources and technology. True application on the job depended on the resolution of resource issues and limited support from hospital management. Incomplete survey data received from training administrators made analysis difficult and extremely time-consuming. As a result of this project, the student researcher learned the importance of effective logic modeling, strict data cleaning processes, and regular communication with Kenyan personnel. This poster describes the importance of assessing the knowledge, attitudes, and practices of a study’s participant pool before developing a comprehensive strategy, as well as the F.A.C.T.s behind international evaluation: Flexibility, Awareness, Culture and Time. |
| Using Systems Engineering Tools to Inform a Geographic Needs Assessment |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Paul Griffin, Pennsylvania State University, pmg14@psu.edu |
| Angela Snyder, Georgia State University, angiesnyder@gsu.edu |
| Natalie Towns, Georgia State University, alhnlt@langate.gsu.edu |
| Christina Scherer, Southern Polytechnic State University, cscherre@spsu.edu |
| Abstract: The goal of this evaluation was to create a model to optimize the geographic placement of shelters in Georgia to meet the current demand for family violence services. The evaluation utilized a systems engineer to create a mathematical model to balance the supply of shelter beds with the demand for violence services. County-level demand for shelter services was estimated using intentional injury hospital visits per population, the percentage of the population living in poverty, the number of temporary protective orders issued, and an indicator for whether or not a county contained a shelter already. Geographic, distance, and capacity constraints were added to the model based on recommendations by experienced program staff. Several model scenarios were run to identify areas of the state in need of additional shelter services as well as areas of the State with excess shelter capacity where limited resources could be redirected to outreach activities. |
| A Statewide Evaluation of Supplemental Educational Services (SES) Programming: Challenges and Lessons Learned |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Pamela Finney, University of North Carolina at Greensboro, pfinney@serve.org |
| Melissa Williams, University of North Carolina at Greensboro, mwilliam@serve.org |
| Alexa Edwards, University of North Carolina at Greensboro, aedwards@serve.org |
| Abstract: In 2010, the SERVE Center at UNCG served as an external statewide evaluator for the Supplemental Educational Services (SES) programming which falls under Title I of the Elementary and Secondary Education Act, as reauthorized by the No Child Left Behind Act of 2001. The program provides free-of-cost academic assistance for eligible students (e.g., tutoring), over and above regular school hours. This external evaluation examined the performance of the statewide SES providers on the basis of: (1) student attendance in the program, (2) parental satisfaction, and (3) academic achievement of participating students. These evaluation criteria were previously identified in the state SES evaluation policy as the outcomes on which to determine the effectiveness of each SES program provider. This proposal provides an overview of the methods for evaluating the state SES program as well as lessons learned and challenges to conducting such a comprehensive evaluation of a widely diverse base of SES providers. |
| Process Evaluation of Data Collection and Reporting Systems for United States Environmental Protection Agency Office of Brownfields and Land Revitalization |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| David Shellard, SRA International Inc, dshellard@gmail.com |
| Abstract: In 2004 the United States Environmental Protection Agency Office of Brownfields and Land Revitalization (EPA/OBLR) initiated a process evaluation to review its data collection and reporting procedures that were implemented by a contractor’s project team for the previous five years. Project and EPA/OBLR staff thought the procedures were resource intensive and welcomed the evaluation. The evaluation required a review of documentation and interviews with stakeholders and project staff. The Evaluation Team participated in a full cycle of the system for nine months. Discussions were held to discuss why some procedures diverged from the documentation. The Evaluation Team’s analysis of the documentation and interview and discussion comments resulted in multiple logic models and feedback on the strengths and weaknesses that EPA/OBLR utilized to implement changes to its data collection and reporting processes. Those changes reduced the delay in reporting by six months in four years and saved time and money. |
| Optimizing the Baseline Phase of a Pay for Performance Multi-Year Evaluation in the Federal Government |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Heather Prather, United States Office of Personnel Management, heather.prather@opm.gov |
| Sarah Johnson, United States Office of Personnel Management, sarah.johnson@opm.gov |
| Abstract: This poster explores the importance of the baseline phase in a longitudinal effectiveness evaluation by examining what special factors to consider in designing and conducting a baseline evaluation and what actions to take following the completion of the baseline stage. Program evaluators can optimize the validity and reliability of the data and contribute to the program’s effectiveness prior to the final stages of the program by following systematic, rigorous evaluation methods. Using lessons learned from a quasi-experimental baseline evaluation of a Federal agency converting from the Federal-wide tenure based General Schedule (GS) system to a new pay for performance system, as well as best practices in evaluation, this poster will address specific topics including selecting and creating appropriate metrics, survey design and planning, data analysis, and how best to communicate the baseline results before beginning the next phase of the evaluation. |
| Promoting Sound Evaluation Practices in the Field of Suicide Prevention |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Philip Rodgers, American Foundation for Suicide Prevention, philiplrodgers@yahoo.com |
| Abstract: The Best Practices Registry for Suicide Prevention (BPR) is a widely recognized source of information about suicide prevention programs and practices. One of the goals of the BPR is to promote sound evaluation practices in the field of suicide prevention. Unfortunately, evaluation in the field of suicide prevention can be limited by ethical and methodological considerations. This poster will answer two questions: (1) What is the BPR, and by what criteria are evidence-based programs and best practices defined? (2) How can these definitions be used to plan and implement sound evaluations that take into consideration the unique ethical and methodological limitations found in the field of suicide prevention? |
| Longitudinal Changes in Capacity and Readiness to Provide Training and Technical Assistance (TA): Developing Effective TA Providers |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Catherine Lesesne, Centers for Disease Control and Prevention, clesesne@cdc.gov |
| Duane House, Centers for Disease Control and Prevention, lhouse1@cdc.gov |
| Kristin Brown, Centers for Disease Control and Prevention, khbrown@cdc.gov |
| Kim Burley, Centers for Disease Control and Prevention, kburley@cdc.gov |
| Ndidi Nwangwu, Centers for Disease Control and Prevention, nnwangwu@cdc.gov |
| Abstract: Background: Centers for Disease Control and Prevention aimed to build grantee organizations’ capacity to provide training and technical assistance (T&TA) around using science-based approaches to teen pregnancy prevention. Method: Annual needs assessments were conducted with staff from twelve grantee organizations (2006-2009). The assessment measured capacity using multiple indicators, including knowledge, training/experience, and confidence in providing T&TA. Grantees received T&TA to increase their capacity to work effectively with local partners. Preliminary Results: Grantees collectively received 1292 hrs of T&TA to build their capacity. Grantees reported increased training/experience providing T&TA to local partners. Specifically, grantees reported capacity in 58% of all of the T&TA topics measured at baseline (e.g. program planning and evaluation topics) and in 87% of all of the topics measured four years later. Trends over time showed significant gains in self-reported confidence to provide T&TA to local partners across a range of program planning and evaluation topics F(2,20)=4.96, p<.05. |
| Evaluation of the Satisfaction Level Among Pregnant Patients Who Attended the Prenatal Care Clinics in a Teaching Hospital |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Ida Sanabria, University of Puerto Rico, wiwi62@onelinkpr.net; ida.sanabria@upr.edu |
| Ruth Ríos, University of Puerto Rico, ruth.rios2@upr.edu |
| Abstract: Patient satisfaction is considered a measure of realized access and care quality from the consumer’s point of view. This evaluation project examined patient satisfaction with prenatal care services at a teaching hospital clinic in Puerto Rico. The sample consisted of 150 pregnant women aged 21 years of age or older in their third trimester selected by a non-probabilistic sampling process. General and multidimensional satisfaction (convenience, coordination, quality of services, health care information and courtesy of health professionals) was measured using a Spanish version of a 41-item-self-administered questionnaire (Aday & Andersen, 1974; Erci & Ivanov, 2004). Results from multivariate analyses revealed that time committed by doctors during prenatal care visits was the most important factor significantly associated to patient satisfaction. However, adequacy of prenatal care was not statistically associated to patient satisfaction. Measuring patient satisfaction is a valuable tool to identify gaps in service delivery that could be improved. |
| Evaluation and Adjustment of Performance Evaluation System in a Middle-Size East European Industrial Organization |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jana Kotrcova, University of Saskatchewan, jana.kotrcova@usask.ca |
| Abstract: Some businesses and industrial organizations in Eastern European countries still function on principles on which they were built during the communist era. One of the limiting factors in introducing modern business and HR procedures is the old-time work values that some long-time employees hold. Therefore, designing and implementing changes to organizational processes, structure, and culture can be met with skepticism and opposition. This work focuses on the assessment of performance evaluation system and its relation to work motivation in a middle-size industrial organization in the Czech Republic. Document reviews, key informant interviews and the Motivation Sources Inventory were used to assess the current state of work motivation, job satisfaction, and performance evaluation system. Major adjustments to the performance evaluation system were proposed based on the results of the aforementioned methods and a literature review. Despite anonymity and confidentiality, resistance and lack of cooperation presented a major challenge to the evaluation. |
| Enhancing Traditional and Innovative Approaches to Advanced Composition in Academic Disciplines |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Maria Jimenez, University of Illinois at Urbana-Champaign, mjimene2@illinois.edu |
| Ayesha Boyce, University of Illinois at Urbana-Champaign, boyce3@illinois.edu |
| Abstract: Stakeholders in advanced composition instruction at the University of Illinois at Urbana-Champaign (UIUC) have had limited capacity to evaluate learning outcomes for the purpose of regularizing iterative improvement in writing instruction. This evaluation was the University’s first step toward evolving robust, regular assessments that will provide the stakeholders in composition and critical thinking courses with empirical evidence needed to fulfill their instructional, administrative, and oversight roles with enhanced confidence and impact. Throughout all phases of the evaluation, issues of quality were considered in order to provide stakeholders with useful and valid information. To this end, a mix of methods were employed in the evaluation including classroom observations, course evaluation surveys, writing attitude surveys, instructor interviews, and the Measure of Academic Proficiency and Progress (Proficiency Profile, formerly MAPP, a standardized test constructed by ETS). Preliminary findings and implications for UIUC will be discussed. |
| An Evaluation of the Introduction of Concurrency Concepts Across a Four-Year Postsecondary Curriculum |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Melissa Yale, Purdue University, myale@purdue.edu |
| Deborah Bennett, Purdue University, bennett@purdue.edu |
| Abstract: Evaluating post secondary curricula has a unique set of challenges, particularly if the evaluator is attempting to assess the impact of infusing a concept across four years of coursework. The National Science Foundation has funded the Purdue CPATH (CISE Pathways to Revitalizing Undergraduate Computing Education) project to integrate concurrency concepts throughout the Computer Engineering (ECE) curriculum. Basic concurrency principles, such as atomicity, synchronization, and parallelism, are being introduced in existing hardware, software, and algorithm courses as a way of meeting the growing need for engineers who can apply concurrency to real world problems. This poster presentation will outline the formative and summative evaluation of this curriculum development project, including the creation of concept and curriculum maps, the development of the Concurrency Concept Inventory (CCI), and the adaptation of an engineering student attitude survey |
| Facilitating Data Interpretation With Preschool Teachers |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Juanita Pelt, The Evaluation Group, juanita@evaluationgroup.com |
| Joel Philp, The Evaluation Group, joel@evaluationgroup.com |
| Abstract: The interpretation and utilization of information obtained in early literacy environments are critical to the implementation of interventions for future reading success. This poster presentation presents strategies and techniques for promoting the use and interpretation of data within the context of an evaluation of an early childhood literacy initiative. Using a web-based data collection system, both implementation and outcome-based results are presented in varying formats all designed to promote utility and ease application in the classroom. |
| Measuring and Reporting on School Climate in a Rigorous Magnet Evaluation |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bianca Montrosse, Western Carolina University, bianca.montrosse@gmail.com |
| Jane Meli, University of North Carolina at Greensboro, jmeli@serve.org |
| Holli Bayonas, University of North Carolina at Greensboro, hbayonas@serve.org |
| Abstract: Little is known regarding under what conditions, if any, magnet schools produce intended outcomes. Some have hypothesized that school climate plays a role in the success of magnet schools (Christenson, Eaton, Garet, Miller, Hilawa, & DuBois, 2003; Griffin, Allen, Kimura-Walsh, & Yamamura, 2007; Lou & Dappen, 2005; McLean, 2006). As one component of a larger rigorous, magnet evaluation, the SERVE Center developed a student and teacher self-report survey to measure the dimensions of school climate identified in the program logic model. This poster will provide an overview of the survey development process, as well as survey results. Results will be presented in the format developed by SERVE to convey climate information in both technical and non-technical language to various stakeholders (e.g., teachers, board of education). |
| Evaluation of Adherence to AIDS Treatment in Patients Suffering Variation in the Distribution of Body Fat Due to Lypodistrophy |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rachel Baccarini, Ministry of Health of Brazil, rachelbaccarini@hotmail.com |
| Abstract: We intend to evaluate the levels of adherence to AIDS treatment in patients suffering variation in the distribution of body fat due to lypodistrophy. The adherence levels to the use of antiretroviral will be evaluated according to antiretroviral schemes in use, and the abandonment of the treatment by those patients. The users’ satisfaction regarding health services will also be evaluated. This study will be carried out in the Public Health System outpatients. All the five regions of Brazil will be encompassed, in cities with population over 50,000 people. A questionnaire will be applied to patients using antiretroviral for at least one year, presenting lypodistrophy caused by the use of those drugs. The adherence will be evaluated by the criterion of 95% of pills ingested in the last 04 days. This project is part of the Masters in Monitoring and Evaluation program, of the National School of Public Health. |
| Evaluation Quality and Dynamic Systems |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jeffrey Wasbes, Research Works Inc, jwasbes@researchworks.org |
| Abstract: This work explores the systemic variables that affect the overall quality of evaluation projects. It presents a dynamic hypothesis (stock and flow model) that is based on an eroding goals system archetype, and attempts to answer the following questions: Which systemic variables most affect the level of quality of the evaluation process and findings? How can evaluators leverage these insights to make efforts and develop policies to ensure high quality evaluation across a broad range of substantive areas? The model presented is qualitative in the sense that values used for formal model construction are based on empirical observation, but are estimated for extrapolation to a broader range of applications. Through simulation runs that test a variety of parameters, the information generated will be used to identify systemic structures that affect evaluation quality and how to develop policies that will contribute to high quality in evaluation projects. |
| Using Propensity Scores and the National Educational Longitudinal Survey, 88:2000, to Evaluate the Effectiveness of Upward Bound Programs |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Bradley Coverdale, University of Maryland, College Park, bjcoverdale@gmail.com |
| Abstract: To determine the effectiveness of Upward Bound programs by meeting the national program goal, increasing the percentage of low-income, first-generation college students who successfully pursue postsecondary education opportunities, data from a nationally represented sample (NELS:88/00) were examined. In particular, responses that matched the program objectives: increasing high school graduation and post-secondary education enrollment rate, as well as ensuring students generate the skills needed to succeed post high school. Upward Bound participants were compared with the remainder of the sample who matched the background characteristics of Upward Bound students via propensity scoring. Students were placed in subgroups based on the number of years enrolled in Upward Bound. Responses from the students were captured and analyzed via binary logistic regression testing. Students that participated in Upward Bound were on average twice as likely to apply for post-secondary education and financial aid, enroll in a post-secondary institution and graduate from a post-secondary institution. |
| Supplemental Education Services (SES) Provision of No Child Left Behind (NCLB): A Synthesis of Provider Effects. |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shanan Chappell, Old Dominion University, shananc@embarqmail.com |
| Shana Pribesh, Old Dominion University, spribesh@odu.edu |
| John Nunnery, Old Dominion University, jnunnery@odu.edu |
| Jane Hager, Old Dominion University, jhager@odu.edu |
| Abstract: The 2001 renewal of the United States’ Title I program in the No Child Left Behind Act instituted the Supplemental Educational Services (SES) program, in which schools in their third year of failing to make Adequate Yearly Progress (AYP) are required to offer out-of-school-time tutoring in core subjects to low-income students. This evaluation synthesizes SES provider effects reported in the extant body of SES provider evaluations to generate an estimate of the overall effectiveness of the SES policy in terms of improving student achievement and to identify provider characteristics that are associated with variation in student achievement effects. |
| Disproportionality in the Child Welfare System: Using Geographic Information Systems (GIS) to Evaluate Differences in Child Welfare Outcomes |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Virginia Dick, University of Georgia, vdick@cviog.uga.edu |
| Abstract: In recent years researchers have begun to examine the issue of racial disproportionality in the child welfare system. Building on two previous projects, The Carl Vinson Institute of Government at The University of Georgia has undertaken a project to examine child welfare data and socio-economic data in graphical representation using maps. Five counties are the basis for this pilot project, representing over 70 census tracts in the state of Georgia. Key demographic variables and child welfare outcomes are mapped together on each of the census tracts and county maps are the basis for the analysis of the relationship between the two sets of variables. This presentation will review the preliminary findings from the review of the completed maps, presentation of the maps, and a description of how the findings will be used in coordination with the Division of Family and Children Services to inform practice within the agency. |
| Informing Truth Through Formative Evaluation: A Needs Assessment of One Federally Funded Homeless Education Center |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Alexa Edwards, University of North Carolina at Greensboro, aedwards@serve.org |
| Lynn Amwake, University of North Carolina at Greensboro, camwake@serve.org |
| Abstract: There are a variety of dynamic issues related to the education of homeless students that requires the collaboration of LEA’s, state agencies, and organizations. As part of a national homeless education program statewide homeless education centers provide research, resources, and information that enable schools and communities to address the educational needs of children experiencing homelessness. This poster presentation will highlight the various components of the evaluation design of one statewide homeless education center and will describe the quantitative and qualitative methods used to assess the needs of the sub grantees in a large southeastern state. It is anticipated that discussion generated from this poster will facilitate an open forum by which conference participants can engage in dialogue that leads to an advancement in what we already know about needs assessment within evaluation as well as spark innovative ideas and solutions that have not yet been explored. |
| Using an Empowerment Approach to Plan and Evaluate a Sexual Health Education Summit for Minority Teens |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lynn Chen, University of Texas, Austin, lynnchen@mail.utexas.edu |
| Adama Brown, University of Texas, Austin, abrown@mail.nur.utexas.edu |
| Heather Becker, University of Texas, Austin, hbecker@mail.nur.utexas.edu |
| Abstract: This evaluation focused on the planning and implementation of a one day, sexual health education summit for minority teens, ages 13 -17. The summit provided health education information by utilizing informed decision making strategies. We described how a teen empowerment approach was used to involve stakeholders and participants in the planning phase and evaluation data collection process. Multiple data sources were collected and analyzed to meet the goals, including 1) planning notes; 2) video tapes; 3) short surveys and open-ended questions directed to parent participants and teen participants; 4) and online-survey from planning committee members, and other stakeholders including the teen advisory committee, agency members, and volunteers participating in the summit. Results showed that teens provided concrete suggestions for future planning and implementation. Online survey results gathered from planning committee members and stakeholders provided helpful insight into ways to generate a more collaborative planning process for future teen summits. |
| Parents as Reporters on Program Effectiveness: The Need to Disaggregate by Parent and Child Characteristics |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tiffany Berry, Claremont Graduate University, tiffany.berry@cgu.edu |
| Susan Menkes, Claremont Graduate University, susan.menkes@cgu.edu |
| Desiree Nangle, Claremont Graduate University, desiree.nangle@cgu.edu |
| Abstract: In the educational evaluation literature, parent surveys are often included in evaluations to triangulate conclusions regarding the effectiveness of programs. However, parent survey information is usually treated in the aggregate; rarely do evaluators systematically examine how parents’ characteristics (e.g., home language, level of involvement, etc.) or their children’s characteristics (e.g., grade level, number of years in program, number of children in program, etc.) affect parents’ perceptions of program effectiveness. The purpose of this presentation is to illustrate how evaluation outcomes (and interpretation of outcomes) are adapted when parental perceptions are disaggregated by parent and child characteristics. Including this information in educational evaluations highlights the need to consider the context of not only direct program participants, but also parents whom are an invaluable reporting source in educational program evaluation. The data for this presentation was collected as part of an on-going evaluation of After-School All-Stars, Los Angeles. |
| Connecting Evaluation Quality and Object Effectiveness: Evaluative Inquiry Practice Patterns in a High Performing School |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kathryn Hill, Minnesota Office of Higher Education, kathy.hill@state.mn.us |
| Abstract: The poster presentation will provide key findings from a study which explored a specific set of indicators associated with evaluative inquiry. Evaluative inquiry (EI) has been defined as the systematic process of gathering and analyzing data for organizational problem-solving and decision-making (Cousins, Goh, & Clark, 2006; Preskill & Torres, 1999). A teacher survey and principal interview questions were developed to gather information about EI-related practices and the use of achievement test data for instructional decisions. The teacher survey and principal interviews were conducted in a sample of higher and lower performing schools. The findings are interesting because they suggest several EI practices are especially predictive of high performance, while other indicators that were previously thought to be predictive appear to be either less important or unimportant. |
| The Development and Validation of an Evaluation Tool Designed to Measure the Distinctive Needs of High School Students After School |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tiffany Berry, Claremont Graduate University, tiffany.berry@cgu.edu |
| John Coffey, Claremont Graduate University, john.coffey@cgu.edu |
| Abstract: Although evaluations of afterschool programs in elementary and middle school are abundant, there are comparatively fewer evaluations of high school afterschool programs, and even fewer valid and reliable survey instruments available to measure outcomes for high school youth. As part of an on-going evaluation of a local afterschool program in Los Angeles, we have developed a high school survey assessing the following constructs: program satisfaction, autonomy, leadership, preparation of college, future plans, goal orientation, and academic preparation. The purpose of this presentation is to present our validation methods, examine survey outcomes as a function of youth characteristics (e.g., gender, grade, ethnicity, motivation) and program characteristics (activity type, program dosage, etc.), and discuss ways to align measures to capture the distinctive context of high school afterschool programs. |
| Using Evaluation to Strengthen Program Quality in New Models of Teacher Preparation |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Edith Stevens, ICF Macro, edith.s.stevens@macrointernational.com |
| Abstract: Alternative teacher preparation is critical to addressing teacher shortages in many states. The expansion of these programs has occurred at a rapid pace in Maryland, from eight to 20 programs in four years. With this kind of growth, the State wanted to develop an accountability structure to ensure that current and future alternative teacher preparation programs are of high quality and are sustainable. The project evaluator, ICF Macro, played a pivotal role in this effort. In this poster session, particular attention will be given to the role of the evaluator at each stage of the process: developing Standards of Practice to ensure consistency in program outcomes, creating Developmental Guidelines to inform program implementation, and piloting a review process for alternative preparation programs. The major questions that drove each phase of the process and how those questions framed the context for the role and activities of the evaluator will be presented. |
| Using Observational Methods and Geographic Information Systems (GIS) to Evaluate Relationships between Physical and Social Environmental Features and Community Walking |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Duncan Meyers, University of South Carolina, meyersd@mailbox.sc.edu |
| Dawn Wilson, University of South Carolina, wilsondk@mailbox.sc.edu |
| Abstract: Environmental supports for physical activity (PA) – such as recreational trails – are important contextual influences for healthy behaviors. While evidence links attributes of the physical environment to health behaviors such as walking, gaps still exist in regard to how such contextual factors interact with social environmental influences (e.g., community connectedness, social support). The evaluation of an NIH funded grant “Positive Action for Today’s Health” (PATH) was expanded to include spatial analyses (via standardized observational methods in conjunction with geographic information systems [GIS]) and additional survey items to assess these factors over time. This poster will present how this supplemental evaluation 1) targets theoretically-derived mechanisms of change in the social and physical environment; 2) conceptualizes and measures variables used in GIS analyses; and 3) present evaluation findings related to the relationships between geospatial variables (e.g., physical features of the trail) and participant perceptions of social environmental supports (i.e., socio-cultural contextual influences). |
| Two-Year Evaluation of an Initiative Designed to Encourage Young Women to Seek Engineering Careers: The Engineer Your Life Initiative |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Christine Paulsen, Concord Evaluation Group LLC, cpaulsen@concordevaluation.com |
| Christopher Bransfield, Concord Evaluation Group LLC, cbransfield@concordevaluation.com |
| Abstract: This poster describes the three-year evaluation of an effort funded by the National Science Foundation to: 1) increase college-bound females’ understanding of engineering, 2) inspire young women to explore engineering as a career and 3) help counselors and engineers encourage young women to investigate engineering opportunities. The evaluation study was designed to assess the impact of the initiative by surveying members of the three main cohorts of interest: professional engineers, career counselors, and college-bound females. These data provide a “snapshot” of attitudes, knowledge and beliefs among key audiences at three points in time: before the initiative was launched, the year of its launch, and one year after the initiative was launched. This poster will present a summary of key findings regarding the impact of the NSF-funded initiative, as well as a summary of methodological lessons learned, with a discussion of the implications for other evaluators who are studying STEM-related initiatives. |
| The Role of Exploratory Factor Analysis in Evaluation |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kai Chi Yam, Washington State University, kai.yam@email.wsu.edu |
| Brian French, Washington State University, frenchb@wsu.edu |
| Laura Hill, Washington State University, laurahill@wsu.edu |
| Abstract: Despite its popularity and usefulness in educational and psychological research, exploratory factor analysis (EFA) may not be as frequent in use in evaluation studies. We argue that EFA is especially important in such studies when study-specific measures are employed. EFA can begin to provide score validity evidence through the examination of internal structure of the measurements. Such evidence can increase the utility of the instrument in evaluation. However, too often EFA is conducted using software default settings (e.g., principal component analysis with varimax rotation) which may not be most appropriate for the (a) data, (b) theory underlying the internal structure, or (c) purpose of the analysis. We provide guidelines for appropriate uses of EFA from decisions related to extraction and rotation methods to sample size and criteria for retaining factors. We also provide interpretations for outputs generated from SPSS from an example of an evaluation study. |
| A Culturally Sensitive Community-Participative Program to Promote Minority Support for HIV/AIDS Vaccine Research: A Social Networking Approach |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Robin Kelley, National Minority AIDS Council, rkelley@nmac.org |
| Abstract: Goal: to foster the increased participation of African Americans and Latino Americans in HIV vaccine research educational activities: Methods NMAC worked in with others to create a culturally sensitive message the liaisons would use in their communities to recruit African Americans and Latinos to help spread the word about HIV vaccine research. Results Within seven-weeks, the seven community liaisons reached a total of 644 community residents, of whom 343 were willing to share the message with three other community members. Furthermore, 62% (400) stated that the information they received from the liaisons increased their awareness about the potential benefits of HIV vaccine research. Conclusion This exploratory program exceeded expectations. This project provides encouraging data suggesting that health communication interventions that use culturally sensitive social networking strategies can successfully promote minority community awareness and support (information dissemination) for HIV vaccine research. |
| Strengths, Weaknesses, Opportunities, Threats (SWOT) Approach: Analysis of Needs Assessments to Assess Availability of Mental Health and Substance Abuse Services for HIV Infected Individuals in Texas |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michele Rountree, University of Texas, Austin, mrountree@mail.utexas.edu |
| Lynn Chen, University of Texas, Austin, lynnchen@mail.utexas.edu |
| Abstract: In this presentation, we describe a SWOT (Strengths, Weaknesses, Opportunities, and Threats) analysis to assess the availability of mental health and substance abuse services to HIV infected individuals in the state of Texas. Eight statewide HIV needs assessment reports were selected. Triangulation of the results was ensured by having three coders review and identify central themes using the SWOT approach and meetings with project investigators to review the results. Systematic rules were established to determine the information to be included or excluded from the analysis. Hence, the primary themes using the SWOT analysis were illuminated and future recommendations highlighted. Findings provide a comprehensive and holistic view of mental health and substance abuse services and the gaps in the continuum of care related to access, delivery, and utilization of services to this population. |
| Improving Evaluation Quality by Engaging Community Agencies in the Evaluation Process |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Korinne Chiu, University of North Carolina at Greensboro, k_chiu@uncg.edu |
| Kelly Graves, University of North Carolina at Greensboro, kngrave3@uncg.edu |
| Abstract: This study surveyed community-based programs to assess the benefits, barriers, and levels of community engagement in the in the evaluation process. Community agencies were asked to describe and discuss the benefits of program evaluation, including identifying what program components to evaluate, addressing community needs, and identifying program strengths and weaknesses. Barriers to the evaluation process in community-based programs were identified such as insufficient funding for the evaluation and absence of cultural competence in the evaluation process. In addition, community agencies provided responses regarding the roles of different stakeholders that are involved in the evaluation process. Open-ended responses also provided a more comprehensive understanding of how and why evaluations were being conducted in various community-based programs. Findings from this study provide insight on how to improve the quality of community-based programs and evaluator partnerships in order to better serve the community and provide quality information on program outcomes and impacts. |
| Technology-Enabled Evaluation: Advantages and Disadvantages |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kristy Jang, University of British Columbia, younjung@interchange.ubc.ca |
| Arwa Alkhalaf, University of British Columbia, arwaalkhalaf@hotmail.com |
| Sandra Mathison, University of British Columbia, sandra.mathison@ubc.ca |
| Abstract: This presentation analyzes the use of three different web-based data collection and analysis tools in an evaluation of a graduate-level professional development program, which aims to expand the employability of graduate and post-graduate students through group mentoring and training networks. First, through an online survey development tool Lime Survey, data for social networks analysis (SNA) were collected. Secondly, Google Doc Forms were used to collect and compile data on mentoring interactions. Next, software UCINET was employed to visualize the interactions among the project members and to analyse SNA data. These tools were efficient and useful in this program evaluation, but each have advantages and disadvantages, which are discussed. |
| Meta-evaluation Model for Assessing Evaluations of Professional Development Programs |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Julie Sexton, University of Northern Colorado, julie.sexton@unco.edu |
| Leslie Reid, University of Calgary, lfreid@ucalgary.ca |
| Erin Dokter, University of Arizona, edokter@email.arizona.edu |
| Abstract: Quality and effective evaluations of professional development programs are important for ensuring there is ongoing and relevant feedback for program developers. Effective evaluations allow program developers to act on the information gathered in a timely and meaningful way. To ensure the quality of our evaluation plan of a university faculty instructional professional development program, we developed and conducted a metaevaluation to identify strengths and weaknesses of the evaluation. We developed a metaevaluation survey and procedure and an external evaluation expert used the survey and procedure to conduct the metaevaluation. The results of this metaevaluation were used to improve the plan so that it met criteria for a high quality evaluation. In this poster presentation we will share our process and survey and will engage in conversation about best practices for ensuring quality evaluations. |
| Assessing Context and Culture to Evaluate Sexual Health Programs for African American Youth in Metropolitan Atlanta |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Tabia Henry Akintobi, Morehouse School of Medicine, takintobi@msm.edu |
| Audra Woodard, Wholistic Stress Center Institute Inc, mizaudra03@aol.com |
| Cagney Stigger, Morehouse School of Medicine, cstigger@msm.edu |
| Nastassia Laster, Morehouse School of Medicine, nlaster@msm.edu |
| DeBran Jacobs, Morehouse School of Medicine, djacobs@msm.edu |
| Donoria Evans, Morehouse School of Medicine, devans@msm.edu |
| Jennie Trotter, Wholistic Stress Center Institute Inc, jt@wholistic1.com |
| Tarita Johnson, Wholistic Stress Center Institute Inc, tjohnson@wholistic1.com |
| Melody Jackson, Wholistic Stress Center Institute Inc, melandkhael@yahoo.com |
| Jewel Crawford, Wholistic Stress Center Institute Inc, jewel_crawford@yahoo.com |
| Abstract: Background: The Morehouse School of Medicine Prevention Research Center (MSM-PRC) evaluation partnership with The Wholistic Stress Control Institute, Incorporated centers on assessment of the 2 HYPE Abstinence Education Club targeting African-American youth ages 12-18 within Metropolitan Atlanta. Methods: Critical to a community-based participatory evaluation design was MSM-PRC pilot-testing of a pre- and post-intervention intervention survey to determine its face validity and cultural appropriateness. Evaluation was conducted using four focus groups comprised of 30 African American youth ages 12 to 19. Results: Structured conversations with youth resulted in an expanded understanding of how African American youth conceptualize of marriage, family and their futures; themes traditionally central to abstinence education. Evaluation Implications: Findings signal the need for an expanded approach to implementing and assessing programs designed to reduce adverse sexual health outcomes among African American youth in urban settings. |
| Impact of Support on Application of Training |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Sandhya Rao Hermon, Pricewaterhouse Coopers LLP, sandhya.rao@us.pwc.com |
| Steven Torkel, Pricewaterhouse Coopers LLP, steven.torkel@us.pwc.com |
| Abstract: An important aspect of corporate learning is the extent to which the learning is applied back on the job. Seventy new hires in a large accounting and professional services firm participated in a four-day blended program on introductory knowledge and technical skills. After being back on the job for six months, they completed a survey that included measures of knowledge/skill application and the amount of support for learning received back on the job. Results demonstrated that employees who received more support on the job were more likely to successfully implement what they learned. Implications for optimizing the impact of applied training programs will be discussed. |
| Finding the Right Target: The Case for Informal Science Education |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Chris Caveny-Cox, North Carolina State University, chris_caveny-cox@ncsu.edu |
| Benjamin Silliman, North Carolina State University, ben_silliman@ncsu.edu |
| Abstract: This Poster, “Finding the Right Target, Shooting from the Right Distance” reviews recent research on programs in informal science education and provides examples of appropriate program designs and measures for diverse types of community-based programs. Drawing from theory and practice research in youth development, informal science education, and evaluation, the poster will illustrate the importance of matching program processes and outcomes to the intensity, frequency, duration, and breath of the program and the competence, commitment, and consistency of program leaders. Understanding by design provides a framework for educational programming and evaluation in this approach, including accurately targeting process and outcome objectives and scaffolding learning experiences. Implications for Extension program improvement, accountability, and capacity-building, especially related to informal science education, will be discussed. |
| Mapping Teacher Education Evaluative Use to Evaluation Use Standards |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Georgetta Myhlhousen-Leak, University of Iowa, leakjg@msn.com |
| Abstract: This research took the program evaluation use standards and mapped them to findings from research investigating the types of use and the factors affecting use in the context of teacher education program accreditation in Iowa. The original research study explored program administrators’, and faculty member’s perceptions and experiences with use in the context of teacher education program review (i.e., accreditation). As potential evaluators and/or intended users of the assessment process faculty members represent key stakeholders in the evaluative efforts of program review. This study identified the types of use (i.e. process use-instrumental, conceptual, persuasive and finding use instrumental, conceptual, symbolic) and factors effecting use (i.e. human, context, procedural) that administrators and faculty, found most salient. The population for the study was drawn from four volunteering four-year liberal arts institutions in the state of Iowa. Respondents completed a structured open-ended interview and five related scales. The mapping of use standards provides a direct connection between theory and practice. |
| Evaluating a Special Education Doctoral Preparation Program From the Outside In |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Michelle Bonati, University of Illinois at Urbana-Champaign, michelle.bonati@gmail.com |
| Lori Meyer, University of Illinois at Urbana-Champaign, l.erbrederis@gmail.com |
| Montrischa Williams, University of Illinois at Urbana-Champaign, willia52@illinois.edu |
| Randy Fletcher, University of Illinois at Urbana-Champaign, rfletcher@parkland.edu |
| Abstract: A shortage of special education teachers and faculty has prompted federal funding of personnel preparation programs in higher education. This evaluation focused on the practice-related outcomes of one such doctoral program, that is, outcomes related to the instructional practices of teachers of students with disabilities. Compared to evaluations typically conducted from within a program or organization, this evaluation took an external standpoint and gathered data on the unique practice-related perspectives of special education teachers, leading special education experts, and contemporary research on priority practice-related needs in special education. The findings from all three data sources were then analyzed to assess the alignment between the practice-related components of the doctoral program and: a) what researchers are presenting as best practices, b) the views of special education faculty regarding doctoral leadership preparation and practice related needs of teachers, and c) the views of local special education teachers. |
| Assessing Individual Substance Use Based on Information from Multiple Data Sources: Lessons for Evaluation about Understanding and Reconciling Differences |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Lynn Wallisch, University of Texas, Austin, lwallisch@mail.utexas.edu |
| Thomas Bohman, University of Texas, Austin, bohman@austin.utexas.edu |
| Kristin Christensen, University of Texas, Austin, kchristensen@austin.utexas.edu |
| Dena Stoner, Texas Department of State Health Services, dena.stoner@dshs.state.tx.us |
| Abstract: A sample of 1,616 patients of a large publicly-funded hospital district who had behavioral (mental health and/or substance abuse) and physical disorders participated in a demonstration project designed to prevent potential disability. Participants were randomized to intervention and control groups. Data on participants’ substance misuse were obtained from medical records (ICD-9 diagnoses), annual patient surveys (self-reported use and problems), case manager activity reports (pre-authorization for substance assessment or treatment services), and post-intervention participant interviews (whether substance abuse was a barrier to working and whether participant used substance treatment services). This study compares the conclusions about participants’ substance use that would be drawn from each source, examines differences between the intervention and control group, discusses possible reasons for apparent discrepancies, and suggests lessons learned that can be applied in future evaluations to build upon the strengths offered by having different perspectives on the same topic while minimizing discrepancies. |
| Linking Evaluation Quality and Program Implementation: A Quasi-Study of A National Implementation |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Melissa Rivera, National Center for Prevention and Research Solutions, mrivera@ncprs.org |
| Scott Steger, National Center for Prevention and Research Solutions, ssteger@ncprs.org |
| Abstract: This study investigated the efficacy of Stay on Track, an evidence-based program, through a quasi-experimental design. The national evaluation methodology was replicated to examine the quality of implementation and adherence to program material within several classrooms in Indiana, New Mexico, and Florida were analyzed to determine if student outcomes could be differentiated based on the program implementer (National Guardsmen, Teacher, or Combination), the quality of program implementation, or the students’ characteristics (race, age, military families, and gender). Findings from the national study reveal that the program is able to empower youth by providing knowledge and life skills relevant to health-promoting behavior. The intervention has been associated with short-term improvements in student: motivation to abstain from substance use (knowledge of drug effects and risk of harm), personal competence skills (decision making and goal setting skills), healthy interpersonal and social skills (refusal skills, media awareness, and communication skills), and selection of a healthy social environment (advocacy skills). |
| Evaluating Teachers’ Use of Technology for Improving High School Academic Achievement: A Multi-level Analysis |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| George Chitiyo, Tennessee Technological University, gchitiyo@tntech.edu |
| Jamie Coburn, Tennessee Technological University, jamie.coburn@tn.gov |
| Abstract: This study is an evaluation of teachers’ use of technology to improve students’ achievement as measured by their performance on standardized tests. Data for the study was obtained from the Educational Longitudinal Study of 2002 from which a nationally representative sample of approximately 6,000 public school 10th graders was drawn. Several measures of technology use by the students’ teachers were then used in a multi-level analysis to predict students’ performance in mathematics and reading after controlling for the students’ estimated abilities. The incorporation of computers and the internet in instruction was positively related to students’ academic achievement. This was generally consistent for both mathematics and reading and also across the different types of school urbanicity (urban, suburban, and rural). |
| New Visions for Civic Engagement: Evaluating the Impact of Human Service Transformation and Social Change Work |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Laura Pinsoneault, University of Wisconsin, Madison, ltpinson@wisc.edu |
| Abstract: The current funding climate suggests that human service organizations in order to survive will need to re-engineer themselves in ways that move beyond traditional service. More importantly that they establish that resources devoted to this transformation can help improves lives and strengthen the long-term impacts of their work. There are many innovative organizations developing plans to transform the way they approach human services. However, the challenge continues to be developing effective multi-level evaluation tools and methods to demonstrate the effectiveness and value of investing in this transformation. This poster details the evaluation strategy and findings from the first 18-months of a 3-year initiative of the Alliance for Children and Families to encourage and support its HSO membership in engaging boards, volunteers, clients, and residents in mission-based advocacy and civic engagement efforts and essentially transforming the ways organizations and communities view contributions of HSO and the constituency they serve. |
| Evaluation Capacity Building at the National Institute of Allergy and Infectious Diseases (NIAID): One Block at a Time |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Brandie K Taylor, National Institute of Allergy & Infectious Diseases, taylorbr@mail.nih.gov |
| Tarsha McCrae, National Institute of Allergy & Infectious Diseases, mccraet@mail.nih.gov |
| Kevin Wright, National Institute of Allergy & Infectious Diseases, wrightk@mail.nih.gov |
| Abstract: NIAID is working to build institute evaluation capacity through two ongoing learning activities in order to engage program staff, as most are unfamiliar with evaluation fundamentals. The NIAID Evaluation Forum (NEF) provides an opportunity for staff to learn about and discuss evaluation-related topics important to the organization, while the Evaluation Training Seminars offer the occasion to learn about a topic in more depth. The activities provide a venue to inform staff on ways to incorporate evaluation into programmatic routines and provide a common understanding on improving performance. A significant challenge for NIAID is to identify steps to best share the wealth of knowledge and experience of program evaluation methods and practices. The forums/trainings allow staff to interact around evaluation processes and results and learn tools and resources for planning, prioritization and decision-making. Examples of implementation, received feedback and challenges will be illustrated. |
| Conducting Comprehensive Workforce Assessments of Public and Tribal Child Welfare Agencies: Lessons Learned |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Kathryn Schroeder, University of Denver, kathryn.schroeder@du.edu |
| Laricia Longworth-Reed, University of Denver, laricia.longworth-reed@du.edu |
| Anna de Guzman, University of Denver, anna.deguzman@du.edu |
| Sheridan Green, University of Denver, sheridan.green@du.edu |
| Julie Morales, University of Denver, julie.morales@du.edu |
| Robin Leake, University of Denver, robin.leake@du.edu |
| Joe Walker, Native American Training Institute, joew@nativeinstitute.org |
| Abstract: The Comprehensive Organizational Health Assessment (COHA) was designed as part of a federally funded grant to assess and evaluate the organizational health and functioning of public and tribal child welfare agencies. Assessment results are used diagnostically to identify strengths and challenges and to guide the development of targeted systems-change interventions. The evaluators sought to obtain a complete picture of each agency through surveys, interviews and focus groups with staff at all levels of the organization, community partners, and child welfare clients. Implementing an assessment of this magnitude in four diverse and geographically distinct child welfare agencies presented many challenges and opportunities. In administering a quality evaluation, the evaluators learned many lessons during each of the phases of planning, coordination, and logistics. These lessons not only impacted current and future administrations of the COHA, but the interventions that were developed. |
| Randomized Control Evaluation of a Farm Safety Intervention: Using Theory of Planned Behavior for Changing Power-Take Off (PTO) Related Behaviors of Youth on Farms |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Haminda JIinnah-Ghelani, University of Georgia, hamidajinnah@gmail.com |
| Zolinda Stoneman, University of Georgia, zo@ihdd.uga.edu |
| Abstract: The rate of injury to America's farm youth is unacceptably high. Most approaches in farm injury prevention have focused on knowledge as the primary outcome. Gains in knowledge are minimally effective in changing behavior. Results will be presented from a three-year randomized control trial evaluating the effectiveness of a family-based farm safety intervention that utilizes the Theory of Planned Behavior (Azjen, 1985) as a framework for changing safety behaviors. Ninety families having children between 10 and 19 years, who are active on the farm, were randomly assigned to one of three groups - parent-led, peer led and control group. Analysis of covariance models on the final wave of data, controlling for pre-intervention levels of the outcome variables, will be preformed. We postulate that one group will be more effective than others in positively changing the knowledge, attitudes, intentions, and behaviors of youth related to working with Power-Take Offs (PTOs) on the farm. |
| Evaluating a Pan-Canadian Population Health Research Centre: Integrating Systems and Developmental Approaches |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Barbara Riley, Propel Centre for Population Health Impact, briley@uwaterloo.ca |
| Roy Cameron, Propel Centre for Population Health Impact, cameron@uwaterloo.ca |
| Sharon Campbell, Propel Centre for Population Health Impact, sharoncm@uwaterloo.ca |
| Steve Manske, Propel Centre for Population Health Impact, manske@uwaterloo.ca |
| Steve Brown, Propel Centre for Population Health Impact, ksbrown@uwaterloo.ca |
| Abstract: Interest is growing in centers, teams and networks that link evidence and action to improve society. The Propel Centre for Population Health Impact is a collaborative enterprise at the University of Waterloo, funded by the Canadian Cancer Society, to conduct solution-oriented research, evaluation and knowledge exchange to accelerate improvements in the health of populations in Canada and around the world. Propel’s mandate has evolved from traditional research to a focus on continuous improvement and impact. This new orientation requires a new approach to Centre evaluation that embraces the complex nature of social and policy change. To this end, Propel is developing a theory-driven evaluation, integrating systems and developmental approaches to evaluation. The framework will be described, including implications for metrics (e.g. illustrations of conceptual and instrumental use of evidence, connections within and across systems), matching the evaluation with system dynamics, and adapting the theory of change and evaluation over time. |
| The Program Might Work, but Does it Fit the Context? Looking at One Special Education Leadership Program in Terms of National Policy and Research Priorities |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Matthew Linick, University of Illinois at Urbana-Champaign, mlinic1@gmail.com |
| Cristin Geoghegan, University of Illinois at Urbana-Champaign, geogheg1@illinois.edu |
| Kim Wolowiec-Fisher, University of Illinois at Urbana-Champaign, kwolow1@illinois.edu |
| Daniel Araya, University of Illinois at Urbana-Champaign, daniel@levelsixmedia.com |
| Abstract: The Individuals with Disabilities Education Improvement Act of 2004 created an opportunity for many personnel training grants in special education. One such program, OPAL (Opportunities for Preparation, Access, and Leadership), began at a major university in 2006. Our evaluation focused on OPAL’s fit into the national conversation surrounding special education. We analyzed the context in which this program functions through interviews with researchers and policy makers, and archival analysis of grants and research, which generated a broad interpretive framework. We sought to determine if OPAL’s focus on “access to the general curriculum” was congruent with national priorities in research and policy, and where OPAL sits in the discourse of civil rights surrounding special education. |
| Implementation Evaluation of the Disability Prevention in Leprosy in the Cabo Frio, Rio de Janeiro, Brazil |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Rafaela Souza, National School of Public Health (ENSP/Fiocruz), rafaelabcs@globo.com |
| Elizabeth Moreira dos Santos, Oswaldo Cruz Foundation (Fiocruz), bmoreira@ensp.fiocruz.br |
| Marly Cruz, National School of Public Health (ENSP/Fiocruz), marly@ensp.fiocruz.br |
| Abstract: It was made a study case about Cabo Frio to evaluate the implementation of the rehabilitation and disability prevention (RDP) actions in this city. Many actors related to leprosy control have participated, like health workers, patients and managers. Based on the norms praised for the National Health Department, the results of this research show that RDP actions are partially implemented there. The existence of a multiprofessional staff and the daily schedule of the health unity are good points in Cabo Frio’s program; however, the centralized attendance in only one health unity is a challenge. Although this city is not a priority city to the National Leprosy Program, we see that permanent activities at the beginning and the end of the treatment can contribute for the decreasing of stigma and discrimination caused by the illness and the increasing of life quality of the people who have leprosy. |
| Monitoring and Evaluation Network to Control Tuberculosis in Brazil |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Verônica Machado, Independent Consultant, veronicafmachado@gmail.com |
| Claudia Fonseca, Independent Consultant, claufcost@gmail.com |
| Abstract: The Brazilian Monitoring and Evaluation Network to Control Tuberculosis in Brazil was launched in June of 2008, by the Brazilian Global Fund Project. The 57 municipalities covered by the Project adopted the network as a strategy. The networking group decided to establish four regional points. These points are responsible for mobilizing the strength of Surveillance System and disseminate the M&E culture among the municipalities. The M&E Network joints the participation of governmental programs for tuberculosis, AIDS and Primary Care and several civil society members from local, state and federal levels. During these twenty-one months of existence, the M&E network proposed a collective assessment of epidemiological data analysis, discussions and this year the local governments planed M&E activities in their own plans. This process addressed the importance of planning actions using M&E data. The M&E network can evidence the real demand to control tuberculosis by data evidences and the sponsor’s involvement. |
| Evaluation of the Implementation of an Education Program in Nutrition in Order to Control Obesity in Rio De Janeiro |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Aline Leal, National School of Public Health (ENSP/Fiocruz), leal.aline@gmail.com |
| Elizabeth Moreira dos Santos, Oswaldo Cruz Foundation (Fiocruz), bmoreira@ensp.fiocruz.br |
| Marly Cruz, National School of Public Health (ENSP/Fiocruz), marly@ensp.fiocruz.br |
| Abstract: The changing nutrition’s landscape of the Brazilian population shows that while declines the incidence of sub nutrition in children and adults at accelerated pace, the obesity rates increases. This study aims to evaluate the implementation of an education program in nutrition in order to control obesity in Rio de Janeiro. A methodological study of multiple cases has been done. Two units were selected considering the time of intervention unit deployed in the organization and the magnitude of institutional care. The analysis was made from the characterization of the degree of implementation of actions and their relation to the organizational context. Results :The intervention in both cases was partially implemented. We conclude that the implementation of such educational proposal is restricted by the context and the structures of services’ organization. We suggested: a closer attention to the organization of physical space where the actions occur and more careful planning and actions. |
| Constructivist Success Case Method (SCM): An Alternative Use of the SCM for the Evaluation of New Initiatives |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Julien Kouame, Western Michigan University, julienkb@hotmail.com |
| Abstract: The success Case Method (SCM) was created by Brinkerhoff (2003 ) as an easy and fast way to identify and explain the contextual factors that differentiate successful from unsuccessful adopters of new initiatives. However in the search for extreme cases of users, the initial model imposes external criteria as evidence for success. The constructivism approach to the success case method provides an alternative that really takes in consideration the contextual factors by using internal criteria to determine evidence of success. |
| Supporting the Needs of Parents of Children With Autism Who Speak English as a Second Language: A Qualitative Pilot Study |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Yoko Kitami, University of Hawaii, Manoa, kitami@hawaii.edu |
| Chuan Chang, University of Hawaii, chuanc@hawaii.edu |
| Abstract: The number of foreign nationals who immigrate to the US is growing from 7.9 to 12.6 percentages of US in past 18 years based on Migration Policy Institute 2007 and 2000 U.S Census. In Hawaii, 17.3 percent of Hawaii's total population is immigrants. The purpose of this study is to find out the experiences and needs of children in the autism spectrum disorder (ASD) where one or more of the parents are foreign-born. The research participants are four immigrant Japanese caregivers who have children with autism. Thus, a psychological perspective of person’s living with child with autism and the needs of parents and children will be revealed. Themes and patterns that emerging from these semi-structured interviews will be presented before the time of the conference, Qualitative analysis will be completed and evaluated to provide findings on this issue. |
| Quality Evaluation and Dynamic Initiatives: Mixing Methods and Creativity |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Shaunti Knauth, National Louis University, shaunti.knauth@nl.edu |
| Maryann Durland, Durland Consulting, mdurland@durlandconsulting.com |
| John Durland, Durland Consulting, johnrdurland@gmail.com |
| Abstract: Often evaluators evaluate programs that are already moving full steam ahead. This is how we found Science Chicago, a year-long celebration of science in Chicago. We needed to provide sound, highly focused methodology and a clear theoretical framework – in a short amount of time! Additionally, we wanted to capture the ongoing, dynamic experience of both participants and stakeholders. This poster session will illustrate the process that we engaged in to develop a meaningful framework and choose appropriate methodologies. The process required not just using our methodological skills, but also developing and using our insights from an immersion in historical research on the Chicago’s World’s Fair held in Chicago and other large scale events, to develop a theoretical framework to ground the evaluation. This framework also allowed us to understand the long-term impacts of a large scale event. The poster session will illustrate our process, the framework, the measures and context. |
| Striving for a Safer Campus Alcohol Environment: A Process Evaluation |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jason Black, University of Tennessee, Knoxville, jblack21@utk.edu |
| Daniel Reilly, University of Tennessee, Knoxville, dreilly@utk.edu |
| Jennifer Morrow, University of Tennessee, Knoxville, jamorrow@utk.edu |
| Abstract: This is a process evaluation of the campus alcohol prevention program at a large southern university. The program consists of social norms media campaign and a Resident Assistant training session. The social norms media campaign provides accurate normative information regarding typical student drinking on campus. Studies suggest that students drink in accordance with their normative beliefs, which are often exaggerated. Normative corrections can influence reductions in alcohol consumption and associated negative consequences. The Resident Assistant alcohol policy awareness and enforcement training is delivered by a prevention professional in a 50 minute presentation format to all student residence hall assistants. It targets awareness of campus alcohol policy, and seeks to facilitate RA confidence and competence in consistent policy enforcement. Program outcomes appear promising. The process elements of implementation fidelity, dosage, and reach have not been examined. The current project addresses these issues using student survey and interview data. |
| Metaphors We Evaluate By: Randomized Controlled Trials and the Definition of Evaluands |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Aaron Pannone, University of Virginia, afp2n@virginia.edu |
| Walter Heinecke, University of Virginia, wfh3y@virginia.edu |
| Abstract: We conduct an analysis of the metaphorical reasoning behind the language of the Randomized Controlled Trial (RCT) in medical education and general education (Lakoff, 2003). The medical literature does not reveal widespread use of the RCT to evaluate Undergraduate Medical Education (UME) programs. Researchers in the field of medicine have realized that it is language, not statistics that make a good doctor (Groopman, 2007). Groopman wrote, “Numbers only complement a physician's personal experience with a drug or a procedure, as well as his knowledge of whether a ‘best’ therapy from a clinical trial fits a patient's particular needs and values.” This point exemplifies an understanding that the metaphor may not work for understanding educational research. The implications for research designs are discussed. References Lakoff G, Johnson M. (2003) Metaphors We Live By. Chicago: The University of Chicago Press. Groopman J. (2007) How Doctors Think. New York: Houghton Mifflin Company. |
| Evaluating Recidivism and Risk Factors of Juvenile Offenders Participating in Jail Diversion Programs |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Xuan Stevens, Florida International University, xos0197@alumni.nd.edu |
| Jesus Perez, Lou Panci, Citrus Health Network, perez_29993@msn.com |
| Nicole Davis-White, Family Resource Center of South Florida, babs1300@gmail.com |
| Abstract: This project evaluated both risk factors and recidivism of 45 14 - to 18-year old juvenile offenders. Based on data from the Juvenile Detention Diversion Project, the primary hypothesis was supported: The relationship between the predictors can explain more than 20% of the variance for recidivism. A set of factors that was proposed to predict recidivism was also supported. The current study examined risk factors in the lives of juvenile offenders combined with models to prevent recidivism that contributed to successful socio-behavioral outcomes in adolescent children. Reducing reoffending by juvenile offenders is a public health issue that has positive implications for not only reducing violence but also improving mental health treatment for adolescent youth. Furthermore, results from this study directly impacted the quality of program evaluation and use of evaluation findings for program improvement by including these factors when designing interventions and the effectiveness of the program to reduce recidivism. |
| The Importance of Using Multiple Metrics to Investigate a Single Program Goal |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carl Westine, Western Michigan University, carl.d.westine@wmich.edu |
| Lori Wingate, Western Michigan University, lori.wingate@wmich.edu |
| Arlen Gullickson, Western Michigan University, arlen.gullickson@wmich.edu |
| Abstract: Longitudinal data on a single National Science Foundation (NSF) program presents an opportunity to explore the long-term impact of initial program goals for a large national program. Using ten years of data from the annual survey of the NSF’s Advanced Technological Education (ATE) program grantees, the researchers explore to what extent the ATE program has been successful at increasing the participation of women in technological disciplines, an initial goal of the NSF at the initiation of the ATE program in 1999. A look at the average female participation rate for all grantees over ten years shows a flat trend line. However, presentation of a total female participation trend line at the aggregate level for all ATE grantee programs reveals a large decline in female participation. Careful scrutiny of program goals and representative metrics is necessary to improve evaluation use, but requires knowledge of relevant policy considerations to address multiple perspectives. |
| The Evaluation of the Geology, Environmental Science and Meteorology (GEM) Scholar Program: A Participatory Approach |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Geier, Purdue University, geiers@purdue.edu |
| Deborah Bennett, Purdue University, bennett@purdue.edu |
| Suzanne M Zurn-Birkhimer, Purdue University, zurnbirk@purdue.edu |
| Abstract: The GEMscholar (Geology, Environmental Science and Meteorology) program seeks to increase the number of Native American students pursuing graduate degrees in the geosciences. Drawing on research from Native American student education models to address three key themes of mentoring, culturally relevant valuations of geosciences and possible career paths, and connections to community and family, the GEMscholar program was designed to provide research opportunities and a support network for the participants. Given the cultural foundation of this program, a participatory evaluation approach was well suited. Participatory evaluation strategies, processes, benefits, challenges and lessons learned will be presented as applied to the GEMscholar program evaluation that spanned four years. |
| Co-Constructing Knowledge in Participatory Evaluation: A Case Study of Logic Model Development for a Power-based Community Organization |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Charles Collins, Michigan State University, colli443@msu.edu |
| Tiffeny Jimenez, Michigan State University, jimene17@msu.edu |
| Abstract: Process evaluations require a strong understanding of the program or strategy of implementation to meet organizational goals. Obtaining an accurate representation of the work depends on the creation of a precise logic model. Utilizing participatory evaluation methods may be one of the most effective ways in which to gain a more accurate representation of the process in that model. This poster will present a case study of logic model development with a national power-based community organizing network that utilized a participatory evaluative approach. I will present the stages of logic model development beginning with preliminary stakeholder input through model finalization utilizing the knowledge co-construction process of PPE. Visualizations of each iteration will be presented along with discussion of stages of model development, practical and theoretical implications and lessons learned. |
| Conducting Anonymous Focus Groups: Escaping Politics and Bias? |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Susan Hibbard, Florida Gulf Coast University, shibbard@fgcu.edu |
| Abstract: Focus groups are often used as a form of data collection in program evaluations. This presentation highlights the use of "face-less" focus groups to allow participants to remain as anonymous as possible. The evaluator conducted focus groups over the phone/internet, to increase participation and discussion of job sensitive topics. The logistics of organizing and conducting the focus group to allow participants to feel at ease will be presented. Lessons learned will be shared. |
| Decision Process for Integration of Design and Methods |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jason Burkhardt, Western Michigan University, jason.t.burkhardt@wmich.edu |
| Lee Balcom, Western Michigan University, lee.a.balcom@wmich.edu |
| Abstract: Theoretically, many design options are available to the modern practicing evaluator (Bickman & Rog, 2009). These tools can be complementary or adversarial in their relationship to each other. How can evaluators in contemporary practice navigate their way through the many choices available to them, in pursuit of the highest quality and most efficient research/evaluation design? This poster proposes a framework for a decision model in evaluation that can take the evaluator from Evaluation/Research question to evaluative conclusion in a consistent and valid manner. Topics covered include quality of assumptions, quality of estimators, tailored methods, and quality conclusions. Poster presentation will be interactive, and viewers will have the opportunity to provide feedback, with their comments gaining acknowledgment in the final product. |
| Performance Evaluation of a Cervical Cancer Screening Program: An Integrated Model for Local Analysis at Rio De Janeiro, Brazil |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Oliveira Thais, Oswaldo Cruz Foundation, thscoutinho@yahoo.com.br |
| Elizabeth Moreira dos Santos, Oswaldo Cruz Foundation (Fiocruz), bmoreira@ensp.fiocruz.br |
| Cruz Marly, Oswaldo Cruz Foundation, marly@ensp.fiocruz.br |
| Abstract: Despite being easily treated as early diagnosed Cervical Cancer is still an important public health issue for worldwide women population. National or local control programs face a number of challenges in order to achieve lower incidence and mortality rates, especially at developing countries. Based upon an extended bibliographic review, an integrated model for performance evaluation of cervical cancer control practices has been developed and applied on a testing base as a case study on a local program at Rio de Janeiro/Brazil. Evidences show that although guidelines and indicators for performance monitoring are reasonably well established and new technologies as HPV vaccines are available on some particular situations, health professionals and program managers need evaluations that could indicate how individual, programmatic, and social vulnerability factors influence local programs performance. Networking organization should be considered for enhancing coverage and adhesion of target women to systematic attendance of screening based programs. |
| Evaluating the Diffusion and Implementation of Health Policies Within and Across School Districts |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jennifer Mortensen, Michigan State University, morten19@msu.edu |
| Abstract: Findings from the National Health and Nutrition Education Survey (NHANES) indicate that 31.9% of children and adolescents ages 2 through 19 are overweight, and 16.3% are obese (Ogden, Carroll, & Flegal, 2008). Coordinated School Health Programs (CSHP) have become increasingly popular in school districts across the country over the past 20 years and have been charged with tackling the childhood obesity epidemic. Schools tend to approach coordinated school health through the addition of targeted programs rather than policies aimed at improving the school environment. Creating policies around health can help to establish new healthy norms, shift behaviors, and provide explicit information on what is valued within the system, yet little is known about what factors facilitate and impede adoption and implementation of these policies. This study considers three school districts in Michigan that implemented new health policies to promote wellness and ultimately reduce the prevalence of childhood obesity. |
| Revisiting the Post Plus Retrospective Pretest: Its Use in Evaluating a National Health Training Program for Health Care Professionals |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Elaine Cohen, National Parkinson Foundation, elainevc14@hotmail.com |
| Gladys Gonzalez-Ramos, New York University, gmg1@nyu.edu |
| Ruth Hagestuen, National Parkinson Foundation, ruth.hagestuen@nyumc.org |
| Abstract: This poster reports findings from a multi-year, multi-site national interdisciplinary health training program in Parkinson’s disease (PD) which successfully used post + retrospective pretest methodology to evaluate the training effectiveness. Allied Team Training for Parkinson (ATTP) is a 4 ˝ day interdisciplinary training program for physicians, nurses, and allied health professionals, sponsored by the National Parkinson Foundation, to learn about the latest assessment and treatment of PD and about how to build an interdisciplinary team. Trainees completed post + retrospective pretests at the end of each training day to self-report on their knowledge level on each of that day’s learning objectives. In select training sites, the post + retrospective pretest methodology was compared to traditional pre-post test methodology. This poster session compares results from the post + retrospective pretest and traditional pre-post testing methodology, with recommendations about effective strategies for maintaining quality in post + retrospective pretest methodology. |
| A Culturally Responsive Evaluation of a Historically Black Colleges and Universities (HBCU) Health Service |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| April Waterford, Lincoln University, aprilwaterford@yahoo.com |
| Anthony Fluellen, Lincoln University, anthony.fluellen@lincoln.edu |
| Pamela Frazier-Anderson, Lincoln University, pfanderson@lincoln.edu |
| Kevin E Favor, Lincoln University, kfavor@lincoln.edu |
| Abstract: Access to healthcare is one of the most contentious issues facing the U.S. public. There are many facets of the healthcare subject where issues of quality have eluded the valuation process from the perspective of underserved consumers. The gravity of concerns pertaining to the college attending public led to the development of Healthy Campus 2010 that established national college health objectives. College students, generally, are challenged with acquiring, maintaining, and transitioning to responsibility for affordable healthcare due to age related transitional pressures. In accord with this effort to determine the worth and adequacy of health care for college students, the present evaluation of the Health Service of a Historically Black University employs culturally responsive evaluation in an effort to illuminate the adequacy and quality of services perceived to be available by stakeholders in this seldom examined environment. |
| The Value of Impact Statements to Evaluate Participants’ Experience in a National Interdisciplinary Training Program for Health Care Professionals |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Elaine Cohen, National Parkinson Foundation, elainevc14@hotmail.com |
| Gladys Gonzalez-Ramos, New York University, gmg1@nyu.edu |
| Ruth Hagestuen, National Parkinson Foundation, ruth.hagestuen@nyumc.org |
| Abstract: This poster reports findings from a multi-year, multi-site national interdisciplinary training program in Parkinson’s disease (PD) which successfully used Impact Statement methodology to evaluate the effectiveness of the training program. Allied Team Training for Parkinson (ATTP) is a 4 ˝ day interdisciplinary training program, using discipline-specific and cross-discipline teamwork, for physicians, nurses, and allied health professionals. Sponsored by the National Parkinson Foundation, participants in ATTP learn about assessing and treating PD and about building interdisciplinary teams. At the end of the 4 ˝ day training program, participants wrote, as part of a larger mixed method evaluation, about the professional and personal impacts of the training on them. Content analysis of these statements from five separate ATTP trainings, triangulated with satisfaction data, show the wide range and strength of training impacts. Impact Statements can provide rich evaluation data and are a useful tool for qualitative researchers. |
| Learning From Change: A Collaborative Evaluation of a Community-Focused Cancer Outreach Program |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Carolla Belle, University of South Florida, carollab@mail.usf.edu |
| Lokman Akbay, University of South Florida, lokmanakbay@mail.usf.edu |
| Huaye Li, University of South Florida, lhuaye@mail.usf.edu |
| Jane-Van Trumpet, University of South Florida, jtrumpet@mail.usf.edu |
| Abstract: Many studies indicate that early detection of cancer is crucial. This is the summary of an evaluation for M-POWER (Moffitt Program for Outreach Wellness Education and Resources) which focuses on strengthening partnership with local groups targeting underserved populations to develop and improve methods for reducing cancer disparities in and around Tampa Bay. The purpose of the evaluation is to identify areas needing improvement through community collaboration and partnership, providing health education to empower communities to make positive health choices. A collaborative approach using the Model for Collaborative Evaluation checklist is assigned as the approach of the evaluation (Rodriguez-Campos, 2005, Collaborative Evaluations: A step-by-step Model for the Evaluator). To gather the data, a thousand surveys will be administered to the target community. Using the results, a final evaluation report including recommendations is given to the client to make changes in the defined areas to reach their goals in and around Tampa Bay. |
| Overcoming Barriers: A Case Study in a Program Evaluation of Alcohol and Other Drug Abuse Outpatient Treatment |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Linda Jourdan, University of Wisconsin, Green Bay, jourlk02@uwgb.edu |
| Lora Warner, University of Wisconsin, Green Bay, warnerl@uwgb.edu |
| Abstract: University students and faculty teamed with alcohol and other drug abuse treatment staff to develop and implement a practical outcome measurement system that will be used to monitor and enhance the outpatient treatment program. This system offers a new approach to evaluate the outcomes of the program overall rather than an individual client. Based on the literature and insights of staff, the evaluation team developed a brief survey for clients to assess their status at three points in time on five domains (healthy habits, resisting alcohol and drugs, life satisfaction, support system, and life goals). Demographic items allow for analysis of subgroups. Implementation of this system highlighted many of the typical challenges that small nonprofits face in outcome measurement, such as lack of commitment of the board, insufficient evaluation capacity, difficult choice of which outcomes to measure, collecting data efficiently, protecting confidentiality, and integrating outcome data with existing management systems. |
| Essential Nutrition Action (ENA) in Bangladesh: Using Administrative Data to Show Evidence of Impact |
| Poster Presentation 137 to be held in Texas A on Wednesday, Nov 10, 6:30 PM to 8:00 PM |
| Presenter(s): |
| Jillian Waid, Washington University in St Louis, waid@wustl.edu |
| Abstract: The ENA approach is an evidence-based set of cost-effective, integrated nutrition actions shown to reduce the rate of childhood and maternal malnutrition. Primarily a preventive approach, ENA delivers information about seven key nutrition actions through multiple community actors at critical contact points in the lifecycle. Funded by USAID, the ENA Pilot Project (ENAPP) was implemented by HKI-Bangladesh from August 2008 to September 2009. A robust evaluation for ENAPP was not planned and no quantitative evaluation data was collected. This evaluation relies on administrative, growth monitoring data of children <2 years of age to construct a longitudinal, case control study. Multiple matching methods were used to pair children from the six ENAPP unions to children similar in age and pre-intervention weight, drawn from eleven geographically adjacent unions where intensive ENA training did not take place. Both longitudinal and cross-sectional analyses indicate that children’s weight-for-age z-score improved significantly in ENAPP unions. |