| Session Title: Data Management and Acquisition for Large-scale Evaluations of Public Agencies |
| Multipaper Session 506 to be held in Sebastian Section I3 on Friday, Nov 13, 10:55 AM to 11:40 AM |
| Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG |
| Chair(s): |
| Hannah Betesh, Berkeley Policy Associates, hannah@bpacal.com |
| Abstract: In evaluation practice, participant data is often messy or non-standardized, and acquiring data from public agencies calls for careful negotiations around consent and confidentiality. In this session, we share lessons learned and practical approaches for acquiring and managing data from public agency clients, including school districts, county departments, and community-based organizations contracted to conduct public services. We will draw on our experiences with a multi-year random assignment evaluation of a teacher professional development program in 50 schools and an evaluation of a city-funded youth violence prevention program. We will focus on methods that ensure data integrity and maintain accuracy and quality while consolidating data from multiple sources. Specific strategies to be discussed include the use of a third-party encoder to handle sensitive data matching, development of memoranda of understanding with public agencies, and integration of multiple public data sources. |
| Data Acquisition and Management Practices for a Teacher Professional Development Evaluation Across Eight School Districts |
| Lorena Ortiz, Berkeley Policy Associates, lorena@bpacal.com |
| Assessing student outcomes based on standardized test scores and administrative student records is challenging under the best of circumstances. This presentation will address these challenges of data collection and management in a multi-year, multi-school district random assignment evaluation of a teacher professional development program in 50 schools. As with many studies, this research hinges on effectively managing data collection and management processes to get clean, organized student data--yet these important processes are often not discussed in proposals or study designs. This presentation highlights the necessity of incorporating such processes into proposal writing and study designs. Topics to be discussed include developing and maintaining data contacts with each school district; deciding which data elements are necessary and attainable; requesting data elements in ways that require the least amount of 'cleaning'; developing secure protocols for data transfers; normalizing data elements across multiple data sources, and merging data files. |
| Triangulating Outcomes Across Multiple Public Data Sets: Lessons Learned From an Evaluation of a Violence Prevention Program |
| Hannah Betesh, Berkeley Policy Associates, hannah@bpacal.com |
| Measure Y is the Violence Prevention and Public Safety Act of 2004, a City of Oakland voter approved initiative to fund, among other services, violence prevention programs provided in partnership with schools, community-based organizations and the county probation department. The evaluation of Measure Y's violence prevention programs necessitated a unique data management process to handle confidential data gathered from multiple sources: program participation data from a centralized city database, arrest records from the Alameda County Probation Department, suspension, attendance data from the Oakland Unified School District, and satisfaction and outcome surveys conducted by partner community-based organizations. Because of the high-risk target population and the confidential nature of the outcome data, a key feature of our approach was the use of a third-party encoder to match participation records with school and probation records. This session will discuss the process and lessons learned from this evaluation. |