|
Session Title: How Evaluation Policies Affect Evaluation Quality in a Texas Public School District
|
|
Panel Session 509 to be held in MISSION B on Friday, Nov 12, 9:15 AM to 10:45 AM
|
|
Sponsored by the
|
| Chair(s): |
| Karen Looby, Austin Independent School District, karen.looby@austinisd.org
|
| Discussant(s):
|
| Whitsett Maria, Moak, Casey & Associates, mwhitsett@moakcasey.com
|
| Abstract:
Federal and state departments of education and local school boards regularly institute policies requiring the evaluation of educational programs. Often, these policies support the development and implementation of high quality program evaluations. However, they also may have unintended consequences which undermine the evaluation quality. Program evaluators from Austin Independent School District will illustrate how evaluation policies affect program evaluation work in the district. Panelists will describe policy influences on district program evaluations and highlight theoretical and practical issues that arise in the work. The establishment of district evaluation priorities and the evaluations of the district’s teacher pay for performance program, American Recovery and Reinvestment Act of 2009 (ARRA)funded programs, and externally provided programs operating within the district will be used as illustrations. The panelists’ presentations will set the stage for a collegial discussion about conducting quality evaluation work that is characterized by integrity, accuracy, and usefulness within a policy-driven environment.
|
|
Setting Evaluation Priorities in a Public School District: Who Decides What Gets Evaluated?
|
| Karen Looby, Austin Independent School District, karen.looby@austinisd.org
|
|
The Department of Program Evaluation (DPE) in Austin ISD established a process by which district stakeholders identify evaluation priorities to ensure that resources are available to conduct high quality program evaluations and to ensure that the evaluation work will be used to inform decision-making and improve educational practices. In this process, the DPE staff have increased the rigor of their work while responding to evaluation policy requirements. In this portion of the panel discussion, Dr. Karen Looby will describe the process of engaging decision-makers in setting evaluation priorities and discuss how the quality of evaluation work has evolved in the district in terms of employing increasingly rigorous evaluation methods and creating a variety of formats to better communicate evaluation results to an assortment of stakeholders.
|
|
|
Policy Influences on the Evaluation of a Teacher Incentive Pay Program and Program Decision Making
|
| Lisa Schmitt, Austin Independent School District, lschmitt@austinisd.org
|
|
In 2006, the Austin ISD Board approved a revision of policy to include differentiated pay that considers performance. The AISD REACH program, currently funded with both local and grant allocations, focuses on developing high quality teachers and principals to reach high student achievement. Potential future funding includes a voter approved local tax rate increase, federal Teacher Incentive Funds, and state District Awards for Teacher Excellence (DATE). A Steering Committee guides and oversees the policies related to the pilot, and the Chamber of Commerce serves as a critical friend. The complex landscape of policy, stakeholders, funding sources, and reporting requirements creates a challenging environment for program evaluation. In this portion of the presentation, Dr. Lisa Schmitt will describe the ways district evaluators have navigated a variety of circumstances to provide accurate, usable, and appropriately targeted information to drive program decisions.
| |
|
Addressing Challenges Associated With the Mandated Evaluation of Projects Funded by Federal Stimulus Dollars
|
| Martha Doolittle, Austin Independent School District, marthad@austinisd.org
|
|
Austin ISD received federal stimulus funds (Title I, Part A and IDEA) for 2009-2011 that support multiple projects aimed primarily at improving student achievement and teacher quality. These funds are time-limited to support short-term or one-time projects to boost efforts that can be sustained when the funding runs out in 2011 (the "funding cliff"). While the state currently asks for quarterly updates on expenditures, and jobs created and saved by these grants, the district's staff, Superintendent, Board of Trustees, and the community want to know who is benefitting from the projects and how effectively these federal funds are being spent. From a program evaluation perspective, Dr. Martha Doolittle will describe challenges to tailoring appropriate interim and long-term progress measures for each project, to designing meaningful evaluation logic models that can be used across projects, and to providing an efficient and effective means to communicate project results to key district decision makers.
| |
|
Creating a Partnership: How a School District Helps Community Based Service Providers Evaluate Their Programs
|
| Cinda Christian, Austin Independent School District, cchristian@austinisd.org
|
|
Austin ISD partners with a variety of community based service providers. These organizations conduct activities during and after school including tutoring, enrichment, individual and group counseling, case management, prevention programming, mental health services, etc. Many of these services are provided as components of grants received by the service agencies. As accountability policies established by grantors become more stringent, service providers are overwhelming the district’s capacity to complete ad-hoc data requests. Further, because service agencies define outcome variables differently, the aggregated data make it difficult for district stakeholders and grantors to compare various programs or service delivery methods across agencies. Consequently, DPE has engaged stakeholders, both in and out of the district, including granting agencies, program managers, and direct service providers in a dialog to develop standardized aggregate reports that can be used across purposes. Dr. Cinda Christian will discuss the pros and cons of using these reports for evaluation purposes.
| |