|
Evaluating Participants' Conceptual Changes Around Complex Program Outcomes: Measuring Thinking Around Integrated Food Systems
|
| Presenter(s):
|
| Rita O'Sullivan,
University of North Carolina, Chapel Hill,
ritao@email.unc.edu
|
| John O'Sullivan,
North Carolina A & T State University,
johno@ncat.edu
|
| Jeni Corn,
Technology in Learning SERVE,
jocorn@serve.org
|
| Abstract:
Measuring systems change and the way individuals change in their systems thinking pose very interesting and important challenges for program evaluators. As the complexity of desired program effects accelerates into systems outcomes versus discrete changes in individual indicators, evaluators need more tools to track these involved outcome changes. This paper presents the methodology and results of measuring one group of program participants' changes in how they think about integrated food systems as the result their participation in a leadership development program. Evaluators used Inspiration, a concept mapping software, to develop pre and post depictions of participants' concepts of integrated food systems. Then using a rubric developed by the evaluators, differences in the concept maps were assessed.
|
|
Independent Science Review in Watershed Management Projects: What Insights Does Critical Systems Heuristics Provide in Understanding the Quest for Best Available Science?
|
| Presenter(s):
|
| Mary A McEathron,
University of Minnesota,
mceat001@umn.edu
|
| Abstract:
Scientific evidence or scientifically-based evaluations are phrases spoken often these days, conjuring up images of a nearly singular path to increased clarity and incisive decision-making. The intersection of science and policy in watershed management projects tells a murkier, more twisted tale. Interviews conducted with natural resource agency staff and independent science panelists who were involved in recent reviews on the Columbia and Missouri Rivers revealed significant pockets of disconnection and uncertainty among the scientific disciplines needed to inform decisions in large ecosystems. Frequently occurring themes included boundary issues, which shifted and divided levels of expertise, roles, and the assessment of knowledge. This paper will focus on how the author used Ulrich's Critical Systems Heuristics to guide and inform her analysis. In addition to the insights gained, the author will discuss the successes and challenges she encountered in applying this (new to her) approach from the Systems world.
|
|
How Do Evaluation Concepts Travel? Using Social Network Analysis to Trace Knowledge Transfer in the International Program for Development Evaluation Training
|
| Presenter(s):
|
| Rahel Kahlert,
University of Texas, Austin,
kahlert@mail.utexas.edu
|
| Robert Kahlert,
University of Vienna,
robert.kahlert@gmail.com
|
| Abstract:
The increasing demand for political accountability in governments has propelled the diffusion and institutionalization of evaluation in the public sector on a global scale—especially from the Western societies to the developing world. Social network analysis can aid in the detection of information flow and knowledge transfer. With regard to the evaluation field, it can help determine whether a particular tendency exists towards directionality in the diffusion of evaluation theories and practices. In particular, social network analysis can be used to trace the diffusion of evaluation models across cultural boundaries. The paper utilizes information from the international setting of the International Program for Development Evaluation Training (IPDET). IPDET has trained approximately 1,500 evaluators from more than 100 countries since its inception in 2001. A social network will be constructed from trainers, trainees, and their respective institutional, professional, and academic affiliations. By annotating the social network with the conceptualizations communicated, we can track the shifts these conceptualizations undergo as they are transported between individuals and institutions with different socio-cultural contexts and political needs, generating a network of a very different topography.
|
| | |