|
Session Title: Practice-based Assessment of Evaluation Practice, Method, and Theory
|
|
Panel Session 845 to be held in Sebastian Section I4 on Saturday, Nov 14, 1:40 PM to 3:10 PM
|
|
Sponsored by the Research on Evaluation TIG
|
| Chair(s): |
| Nick L Smith, Syracuse University, nlsmith@syr.edu
|
| Discussant(s):
|
| Beverly Parsons, InSites, bparsons@insites.org
|
| Melanie Hwalek, SPEC Associates, mhwalek@specassociates.org
|
| Abstract:
Although recent writers have generally advocated external, formal studies in researching evaluation, much can also be learned from practice-based assessments conducted by evaluators themselves. Developing effective evaluation practice requires iterative tests of well designed variations that are sensitive to differences in context, evaluand, personal attributes, and changing local conditions. Understanding what influences the development and effectiveness of evaluation requires such situation rich information. The papers in this panel provide such information: a summary of lessons learned from 25 years of research developing stakeholder participation approaches, reflections on how practitioner feedback from training and use of logic models has improved the utility of this method, and an examination of how critical self-assessment by practitioners can provide formative research information for improving evaluation theory. Evaluation practitioners have training and opportunity to conduct developmental research on evaluation, providing the profession with critical insights not attainable from more external investigations.
|
|
Conducting Research on Program Evaluation in Conjunction With Evaluation Studies
|
| Paul Brandon, University of Hawaii at Manoa, brandon@hawaii.edu
|
|
Calls for conducting research on evaluation by and large have not recognized the utility of existing small bodies of published research on program evaluation. In this paper, I describe a body of research on stakeholder participation in evaluation that my colleagues and I have conducted in conjunction with evaluation projects over a 25-year period. Despite having mostly been published in refereed journals, the research has received less attention than might be expected. I give a brief overview of the research methods and results, discuss their utility for the refinement of evaluation practice and methods, and identify the limitations of the research. I speculate why there has been little attention given to the research and suggest how to enhance the effects of similar bodies of research on evaluation practice and on the discussion about empirical research on program evaluation.
|
|
|
Developing Evaluation Theory From Practice: The Case of Program Theory
|
| Patricia Rogers, Royal Melbourne Institute of Technology, patricia.rogers@rmit.edu.au
|
|
Program theory (also known as logic models and theory-based evaluation) has now been widely used for over 20 years in some organizations and regions. This presentation reports on a current project to distil and impart practice wisdom from this experience, drawing from different sources - the researchers' own experiences using program theory (based on their observations, feelings and conversations with other participants before, during and after projects), the experience of others working on these projects (gathered in conversations, anonymous interviews and written feedback), others' experiences using program theory (reported in formal papers, conference presentations and informal discussions), and research students' dissertations (which are more formally systematic in data sources and critical analysis of interpretations). One of the enduring challenges is how we, as professionals and academics, can feel safe enough to share uncertainties and errors, rather than triumphant narratives to show us, and our preferred methods, in our best light.
| |
|
Reflective Practice as Formative Research on Evaluation Theory
|
| Nick L Smith, Syracuse University, nlsmith@syr.edu
|
|
Although recent calls for improved research on evaluation theory have emphasized formal external studies, the profession has historically relied heavily on the reflective practice of theorists and practitioners in evaluating theory. This paper reviews:
(a) the benefits of relying on reflective studies for evaluating evaluation theory, especially their preservation of the complexity of evaluation practice, and their provision of direct, first-hand knowledge of a theory's value, (b) the problems that arise from reflective studies in terms of bias, lack of controls, and limited generality of findings, and (c) improvements that are needed in the way reflective studies are conducted if they are to provide convincing evidence of a theory's quality and contribution, for example, full disclosure and independent confirmation.
Conducted properly, reflective studies of evaluation practice constitute an important form of formative research on evaluation theory that provide a type of primary, contextually-rich evidence not available from formal external studies.
| |