| Session Title: Research on Program Evaluation Theory and Practice: Overviews and Critiques of Exemplars |
| Multipaper Session 874 to be held in Centennial Section D on Saturday, Nov 8, 1:20 PM to 2:50 PM |
| Sponsored by the Research on Evaluation TIG |
| Chair(s): |
| Melvin Mark, Pennsylvania State University, m5m@psu.edu |
| Discussant(s): |
| J Bradley Cousins, University of Ottawa, bcousins@uottawa.ca |
| Abstract: In this session, four presenters and a discussant will address issues in conducting conceptual and empirical research on program evaluation theory and practice. The papers will present (a) an overview and analysis of conceptual and empirical research on evaluation theory, (b) an overview and analysis of empirical studies on evaluation practice, (c) a critique of the limitations of empirical research on evaluation use, which has been studied empirically more than any other topic in the evaluation literature, and (d) an overview and discussion of a small body of research on stakeholder participation in evaluation that has been conducted in conjunction with evaluation studies but largely ignored in the evaluation literature. Collectively and individually, these papers move from broad overviews to increasingly focused analysis, and each paper provides a careful assessment of the current status, important limitations, and needed improvements in research on evaluation theory and practice. |
| Issues in the Conceptual and Empirical Study of Evaluation Theories |
| Karen Zannini, Syracuse University, klzannin@syr.edu |
| Nick L Smith, Syracuse University, nlsmith@syr.edu |
| Although much writing has been devoted to advocating and critiquing particular evaluation theories or approaches, less attention has been devoted to the difficult problems of conducting research on evaluation theories. This paper provides an analysis of both conceptual and empirical modes of conducting research on evaluation theory, including a review of the methods and findings of selected prior studies. Attention is devoted to addressing crucial problems in this work, such as (a) what exactly is the nature of theory, models, or approaches in evaluation and (b) whether and how conceptual statements of theory can in fact be empirically tested. The paper concludes with an assessment of the benefits and limitations of both the conceptual and empirical approaches to studying evaluation theory, including recommendations for how to improve future research on evaluation theory. |
| An Analysis of Empirical Studies of Evaluation Practice |
| Jie Zhang, Syracuse University, jzhang08@syr.edu |
| Nick L Smith, Syracuse University, nlsmith@syr.edu |
| Studies of evaluation practice provide descriptive information on how evaluations are actually conducted and evaluative information on possible improvements. In assessing the current status of research on evaluation practice, this paper first reviews the range of designs and methods that have been used to study practice including practitioner self-reports, meta-evaluations, independent case studies, surveys of practice, and comparative studies of alternatives. Strengths and weaknesses of alternative designs are assessed, as well as their frequency and scope of use. Second, the paper examines four primary issues that have been addressed through studies of practice, reviewing results to date: What is the technical quality of evaluation practice? What is the utility and impact of evaluation? What is the feasibility and effectiveness of alternative evaluation methods? And what is the relevance and utility of alternative evaluation theories or models? The paper closes with a summary of needed improvements in research on evaluation practice. |
| Conclusions from Research on Evaluation Use: How Strong Are the Methodological Warrants? |
| Paul R Brandon, University of Hawaii Manoa, brandon@hawaii.edu |
| J Malkeet Singh, University of Hawaii Manoa, malkeet@hawaii.edu |
| Leviton (2003, p. 526) stated that those who conduct research on the use of evaluation findings often accept 'a [low] standard of evidence that many of us would never dream of applying to the conduct of evaluations.' Many of the studies of evaluation use are reflective accounts of individual practitioners' experiences, and too few use strong designs that result in conclusive results. Furthermore, no studies that we know of have systematically reviewed the large body of studies of evaluation use for their methodological warrants. We address the methodological warrants for the conclusions of research on evaluation use in our paper. We examine the studies identified in reviews of the literature on evaluation use published since about 1980, classify the studies according to their choice of research methodology, and arrive at conclusions about the strength of the warrants for the findings of the research. Leviton, L. C. Evaluation use: Advances, challenges and applications. American Journal of Evaluation, 24, 525- 535. |