|
Session Title: Evaluating Educational Materials in the Profit-Making World: Issues Associated With Evaluating Curricular Materials for Textbook Publishers
|
|
Panel Session 640 to be held in Suwannee 14 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Jerry Jordan, University of Cincinnati, jordanjm@uc.edu
|
| Abstract:
Decisions about the adoption of curriculum materials in schools have significant consequences. For teachers and children, these decisions dictate the basic materials teachers use to teach our children. These adoptions also represent huge financial transactions as districts, or states, invest millions of dollars in materials. Recently, decision makers in education have become more research driven in their decisions about the adoption of curriculum materials. Text book publishers have responded to this environment by sponsoring formal evaluations of the materials they promote to schools. These evaluations have substantial social and economic implications. This panel explores the issues that emerge when evaluators assess the effectiveness of educational materials in collaboration with the publishers who create and market those materials. Panelists will explicate the specifics of this context and highlight how evaluators can manage the unique challenges and opportunities inherent in conducting evaluations in the high stakes context of the "for-profit" publishing world.
|
|
The Bottom Line in the For-Profit World: How the Publishing Industry Shapes the Context of Curriculum Evaluations
|
| Stephanie Baird Wilkerson, Magnolia Consulting LLC, stephanie@magnoliaconsulting.org
|
|
This presentation will focus on how the education publishing industry shapes the context of curriculum efficacy evaluations. The presentation will examine the context of evaluation across various dimensions including purpose, focus, paradigm, organizational evaluation capacity, funding, communication, timeframe, organizational culture, political factors, stakes, and "the bottom line." Attendees will gain a better understanding of how the context of textbook evaluations is both unique and similar to general evaluation contexts. The presentation gives a global overview of the context of conducting textbook evaluations and sets the stage for following presentations to go into more depth in many of the highlighted areas. For over eight years, Stephanie Baird Wilkerson has been evaluating supplemental and basal curriculum materials for education publishers including Pearson Education, Houghton Mifflin Harcourt, National Geographic School Publishers, Northpoint Horizons, Study Island, Curriculum Associates, and EducationCity.com.
|
|
|
Different Points of View: The Fundamental Differences in How Publishers and Researchers See Evaluation Research
|
| Martin Saperstein, Saperstein Associates, msaperstein@sapersteinassociates.com
|
|
To some extent, the assessment of curriculum materials represents the collision of two different worlds: business and academics.-á The primary agents within each area often bring with them sharply constrasting sets of assumptions and goals. This presentation will focus on the most significant areas of potential misunderstanding and confusion. Differences range from very simple issues of vocabulary (e.g. the meaning of "signficant" findings) to deep-rooted differences in philosophies and ethics. Martin Saperstein has been president of Saperstein Associates for nearly 30 years and conducted dozens of studies for major book publishers. In this presentation he will share his experience regarding the major differences that affect collaborations between academic researchers and large publishing companies.
| |
|
Who Said it was Easy? Design Challenges and Considerations Associated With Evaluating the Effects of Educational Curricula
|
| Miriam Resendez, PRES Associates Inc, mresendez@presassociates.com
|
| Mariam Azin, PRES Associates Inc, mazin@presassociates.com
|
|
Presenters, having conducted numerous large-scale experimental and quasi-experimental evaluations examining the effects of various educational curricula on student learning, will discuss the unique design challenges and considerations associated with conducting these types of studies. Design topics that may be addressed include, but are not limited to: a) site recruitment; b) unique issues associated with conducting evaluations of core curricula (rather than supplemental programs); c) implementation monitoring; d) measurement of extraneous variables that can influence study findings (such as teacher quality); e) selection of outcome measures; and f) strategies for enhancing the power and sensitivity of evaluations in detecting any treatment effects.
| |
|
Good News and Bad News: Representing and Reporting Mixed Evaluation Results in the High-stakes World Of Textbook Publishing
|
| Rebecca Eddy, Claremont Graduate University, rebecca.eddy@cgu.edu
|
|
Preskill (2005) and others have provided guidelines about how evaluators should report evaluation results to make them most useful to stakeholders. However, evaluators have to balance competing interests when reporting on evaluations of educational curricula. For example, evaluators must work to meet rigorous design standards set forth by the What Works Clearinghouse (WWC), and simultaneously make the evaluation findings interpretable enough for the client (textbook publisher) and the client's stakeholders (states, districts, and schools). In addition, evaluators often have the responsibility of making the results of advanced statistical analyses interpretable to consumers that accurately conveys implications of these findings. Textbook publishing is not unlike most other areas, where clients are biased to report only positive results and evaluators must resist this temptation. Evaluators should take advantage of their unique role in the publishing world and advocate a strategy where evaluation findings influence product development.
| |
|
Empowering the Consumers of Curriculum Efficacy Studies: How Can We Help the Schools Interpret Our Research?
|
| Jerry Jordan, University of Cincinnati, jordanjm@uc.edu
|
|
The ultimate users of assessments of curriculum materials, teachers and administrators, often are not experienced consumers of research. Anecdotal evidence implies that some decision-makers respond to superficial characteristics of reports and may misunderstand primary claims. In this presentation, three direct actions are suggested to help with the decisions made at the grass roots level: 1) Research should be conducted from the perspective of the teachers to discern which dimensions of the materials are most consequential to them. 2) School PD activities should include education on how to interpret rigorous evaluations of curriculum materials. 3) Consultants with evaluation experience should reach out to schools and offer their expertise in interpreting the implications of the research presented to the schools. Jerry Jordan is a research associate in the Evaluation Services Center at the University of Cincinnati. Most recently, he has conducted several field tests of curriculum materials for Macmillan/McGraw-Hill and Glencoe Publishers.
| |