|
Session Title: Advances in Evaluating Evaluation Theory
|
|
Panel Session 221 to be held in Sebastian Section L4 on Thursday, Nov 12, 9:15 AM to 10:45 AM
|
|
Sponsored by the Theories of Evaluation TIG
|
| Chair(s): |
| Nick L Smith, Syracuse University, nlsmith@syr.edu
|
| Discussant(s):
|
| Veronica Thomas, Howard University, vthomas@howard.edu
|
| Amy Germuth, EvalWorks LLC, agermuth@mindspring.com
|
| Abstract:
Responding to repeated calls for stronger evidence about the quality and utility of evaluation theories, the three papers in this panel each provide a careful analysis and specific guidelines for better evaluation of evaluation theories. The first paper presents five standards (operationalizability, relevance, feasibility, impact, and replicability) for empirically examining theories. The second paper considers a cultural critique of evaluation theories. Nine recommendations are provided for examining evaluation theories through a cultural lens. The third paper examines how the assessment of evaluation theories depends on how one views such theory: as a type of intervention, an ideology, or as a set of principled procedures of practice. Two discussants will react to the presented papers and provide additional observations based on their own research and experience articulating, implementing, and testing evaluation theories. Discussant and audience reactions will challenge and sharpen these proposed advances.
|
|
Developing Standards for Empirical Examinations of Evaluation Theory
|
| Robin Miller, Michigan State University, mill1493@msu.edu
|
|
Calls to evaluate the relationship between evaluation theory and its practice have struck a chord in the evaluation community. Evaluation theories are intended to provide evaluators with the normative bases for making the myriad of decisions that are part of designing and conducting an evaluation. Sorting through theories and determining their ultimate merit requires empirical examination of how evaluation theories function in practice and whether they consistently and reliably lead to successful evaluation under precisely defined circumstances. This paper will elaborate a framework for evaluating theory in practice, using the criteria of procedural operationalizability, contextual relevance, feasibility in practice, evidence of impact, and replicability. These criteria can guide empirical inquiry on the theory-practice relationship and suggest standards for reporting on the use of theory in practice. Systematic and rigorous research using the proposed framework can aid in the development of an evidence-base for a theoretically rooted evaluation practice.
|
|
|
Examining Evaluation Theory in Cultural Context
|
| Karen Kirkhart, Syracuse University, kirkhart@syr.edu
|
|
Theory determines how evaluation is understood, where the boundaries of the enterprise are placed, and how they are legitimated. Multiple theoretical perspectives are available for evaluators' reflection and guidance. Each casts its own worldview, infused with a cultural footprint. Because evaluation theory guides epistemological, methodological, and practical choices, it is critical to question how well evaluation theory addresses dimensions of cultural context and conversely to reflect on how culturally biased assumptions may enter evaluation theory. This paper opens with a succinct discussion of culture, and then moves to examine the complex intersection of culture and evaluation theory. Culture shapes evaluation theory through the historical and cultural contexts of theory development, including the individual theorists and their institutional and social locations. Evaluation theory shapes cultural understanding by offering a framework for addressing cultural context in the evaluation of social programs. The paper closes with practice recommendations for cultural critique of theory.
| |
|
Characterizing the Evaluand in Evaluating Theory
|
| Nick L Smith, Syracuse University, nlsmith@syr.edu
|
|
This paper discusses alternative approaches to evaluating theory depending on how one characterizes the evaluand: as intervention, as ideology, or as principled procedures of practice. Evaluation theories are frequently described as interventions to be implemented in a standard way and assessed in terms of whether results match promises. Testing a theory as an intervention raises familiar concerns of definition, operationalization, implementation, and impact assessment. An alternate view is that evaluation theories are essentially ideological commitments promoting particular sets of social and professional values. Evaluating an evaluation theory under this view, requires methods of moral and value analysis. Finally, a third view is that an evaluation theory is primarily a theory of practice, consisting of a set of principles and associated procedures designed to produce a certain category of desired outcomes. This view combines elements of the two prior views and requires impact evidence, values analysis, and judgments of practical benefit.
| |