|
Session Title: The Fight for Evaluation Quality: Perspectives From the Trenches
|
|
Panel Session 290 to be held in BOWIE A on Thursday, Nov 11, 1:40 PM to 3:10 PM
|
|
Sponsored by the Independent Consulting TIG
|
| Chair(s): |
| Carol Haden, Magnolia Consulting LLC, carol@magnoliaconsulting.org
|
| Abstract:
Teaching students about evaluation quality in a graduate class is one thing. Finding ways to ensure it when conducting applied evaluation studies is another. Evaluators can enter into the conduct of evaluations with idealistic notions of how their work will espouse quality. What they will find are a myriad of challenges that can threaten the quality of all aspects of evaluation. The job of the evaluator is to anticipate, plan for, and respond to these challenges as they present themselves during evaluation studies. This panel is comprised of independent evaluation consultants who will share their experiences and perspectives related to evaluation quality using examples from a variety of studies. Presenters will share insights about how local context influences evaluation quality, the nature of relationships in promoting evaluation quality, and evaluators’ role in ensuring quality through reporting.
|
|
Evaluation Quality and Local Context: When Anything Can Happen in the Trenches
|
| Stephanie Wilkerson, Magnolia Consulting LLC, stephanie@magnoliaconsulting.org
|
|
This presentation will address how local context influences evaluation quality. Using examples from multisite randomized control trials, the presenter will share how the politics, organizational culture, and institutional capacity of districts and schools affect evaluators’ abilities to ensure evaluation quality. This presentation conceptualizes evaluation quality as it relates to maintaining design integrity and coherence, fidelity of implementation, and participant engagement. Contextual factors related to Institutional Review Boards, local budgets, district priorities, and staff capacity support or threaten evaluation quality. This presentation aims to increase evaluators’ awareness of potential threats to evaluation quality during the development and implementation of large-scale studies. The presenter will offer lessons learned in ensuring evaluation quality despite challenges.
|
|
|
Evaluation Quality and Relationships: When Nurturing Makes a Difference in the Trenches
|
| Lisa Shannon, Magnolia Consulting LLC, lisa@magnoliaconsulting.org
|
|
For independent evaluators, developing positive relationships with clients and study participants is critical to ensuring evaluation quality. Relationships with clients set the tone for studies and can affect the degree to which evaluators can interact with study participants. When clients have confidence in evaluators, they trust them to communicate freely with study participants, which facilitates the collection of meaningful data. Likewise, when participants feel supported and appreciated, they are more likely to abide by study specifications, respect implementation guidelines, participate in data collection activities, and share relevant feedback, all of which will contribute to evaluation quality. In some situations, such as when participation is mandated, study participants might feel mistrustful initially, making it even more critical to nurture affirmative relationships. This presentation will provide an overview of relationship-building techniques for evaluators and demonstrate how positive relationships with clients and study participants can ensure evaluation quality, particularly in challenging situations.
| |
|
Evaluation Quality and Reporting: When Working Together Improves the Trenches
|
| Mary Styers, Magnolia Consulting LLC, mary@magnoliaconsulting.org
|
|
Weiss (2004) suggested that evaluation reports are not journal articles, but are often considered of high importance to a particular program or district. As independent consultants, our work is critical in helping a particular group of individuals to improve their capacity and deepen their understanding. The work may not necessarily be publishable but is of critical importance to those individuals directly impacted by the program. This presentation will speak to evaluation quality as related to what an evaluation means for stakeholders and participants, with specific examples from our work with various clients to improve their program. We work collaboratively with program users and stakeholders to provide a voice for program benefits, disadvantages and suggestions for improvement. As independent consultants, we judge the quality of our evaluations by their utility for clients and not by whether it is published.
Reference:
Weiss, C. H. (2004b). Rooting for evaluation: A cliff notes version of my work. In M. C. Alkin (Ed.), Evaluation roots (pp. 153-168). Thousand Oaks: Sage.
| |