| Session Title: Expanding the Independent Evaluator's Tool Box: First Forays Into Using Cost/Economic Data, Social Network Analysis, Reflective Assessment, and GIS Data |
| Multipaper Session 250 to be held in Capitol Ballroom Section 3 on Thursday, Nov 6, 10:55 AM to 12:25 PM |
| Sponsored by the Independent Consulting TIG |
| Chair(s): |
| Sally Bond, The Program Evaluation Group LLC, usbond@mindspring.com |
| Abstract: As evaluators, we are always looking for ways to strengthen our findings of the 'value' of a program. In some cases, even where budgets may limit us, there may be ways to incorporate aspects of cutting-edge analyses that provide a better context for program findings, without conducting full-scale studies. In this session, four evaluators will share ideas about including aspects of the following 1) cost/economic analysis, 2) social network analysis, 3) reflective assessment, and 4) GIS data. The presenters will provide information about the 'full-blown' method, describe the modifications they made, and share the impact that inclusion of such analyses had on their understanding of the program outcomes and impact as well as on the funder's understanding of his or her program. The presentation will also provide time for the audience to ask questions or brainstorm other ways in which such analyses can be incorporated. |
| Making Use of Economic Data: Cost Study of a Teacher Induction Program |
| Amy Germuth, Compass Consulting Group LLC, agermuth@mindspring.com |
| As part of a study of a district-level teacher induction program, the evaluator gathered economic data from the program director regarding the program budget and costs. Additionally the evaluator sought out comparison economic data in order to better understand these costs in context. Without conducting a true cost-benefit analysis, this analysis still allowed for a more refined examination of the costs of implementing the program and how these costs compared with costs associated with teacher attrition, transfer, and recruitment. This approach provided a clearer picture of costs by category and the costs required for successful retention and placement of lateral entry and new teachers. It also allowed one to estimate the cost offsets associated with such an intervention. This analysis provides information that is useful for assessing the value and importance of program outcomes relative to the cost of program implementation. |
| Using Concepts of Social Network Analysis: A Study of Collaboration in a Media and Technology Initiative |
| Sally Bond, The Program Evaluation Group LLC, usbond@mindspring.com |
| From 2003-2006, the state of North Carolina funded a number of schools to implement its IMPACT model for Media and Technology Programs. Collaboration between teachers, the school library media coordinator, and a technology facilitator is key to the success of the model. In order to assess change in collaboration at one of these schools, the evaluator developed a pre/post Collaboration Wheel exercise for participating staff members. With participating school personnel identified as a social network, the Collaboration Wheel was used to capture connections between actors in the network (who interacts with whom, and how often) as well as the nature of interactions between actors. Through the use of this method, the evaluator was able to better understand how program participants defined collaboration. Program participants were then able to consider ways to deepen their collaboration beyond simply sharing information or providing collegial support to one another. |
| Reflective Assessment: A Streamlined Evaluation Technique |
| Dawn Hanson Smart, Clegg and Associates Inc, dsmart@cleggassociates.com |
| Reflective Assessment is a structured, streamlined form of evaluation which captures data on program operations and achievements and generates information on improvements to service delivery. Widely used in education, the evaluator has applied the method in a variety of housing-related evaluations. RA provides an opportunity for multiple stakeholders within an organization to carefully think about a program, assess progress, document accomplishments, and consider the choices ahead before continuing with implementation. It is conducted in 'real time' during implementation. It is the immediacy of the learning that makes RA valuable ' changes are identified and incorporated more quickly than is often true in other evaluation approaches. Involving multiple stakeholders ensures that various points of view are considered. With clients, staff, management, and board members at the table, there is greater understanding of what is needed and why, and greater buy-in to the recommendations developed. |
| Improving Findings through Mapping: First Steps in Using Geographic Information Systems |
| Sarah Heinemeier, Compass Consulting Group LLC, sarahhei@mindspring.com |
| Geographic Information Systems (GIS) analyses are popular for the visual expression of data, especially when that data focuses on the spatial distribution of clients, landmarks, programs, etc. There is an ever-growing field of study behind the use of this approach with specialized training and software applications dedicated to the processing of the data and production of maps and visual exhibits. This presentation will present several techniques for creating relatively simple maps that can be incorporated into evaluation reports and presentations, as was done within the context of an early childhood systems evaluation. These maps may prove to be useful heuristics in presenting and explaining patterns in data, if not triggers for the identification of patterns or trends that were otherwise unnoticed. |