What is Evaluation?

This statement was developed by a previous  AEA Task Force commissioned by the AEA Board to “define and communicate the value of evaluation to the media, the public, and other audiences as well as be used comfortably by evaluators throughout the field without regard for specialty or area of expertise.” The statement is meant to encourage dialogue -- so based on comments and responses it will be revised periodically. Thus, your reactions and comments are encouraged (see comment section below).The Task Force was comprised of both long-time evaluation professionals and AEA members newer to the profession. All have experience and expertise in communicating to others about evaluation. The task force included:

Michael Quinn Patton, Chair, Edith Asibey, Jara Dean-Coffey, Robin Kelley, Roger Miranda, Susan Parker, and Gwen Fariss Newman

What is Evaluation? 

Evaluation is a systematic process to determine merit, worth, value or significance. So what does that mean in practice?  Let’s use one kind of evaluation, program evaluation, to illustrate. Programs and projects of all kinds aspire to make the world a better place. Program evaluation answers questions like: To what extent does the program achieve its goals? How can it be improved? Should it continue? Are the results worth what the program costs? Program evaluators gather and analyze data about what programs are doing and accomplishing to answer these kinds of questions.

A program evaluation has to be designed to be appropriate for the specific program being evaluated. Health programs aim to make people healthier and prevent disease. School programs strive to increase student learning. Employment training programs try to help the unemployed get jobs. Homelessness initiatives work to get people off the streets and into safe housing. Chemical dependency programs help people using alcohol and drugs. Community development programs plan initiatives to increase prosperity among those in poverty. Juvenile diversion programs try to keep kids out of jail and put them on a path to becoming productive adults. For each kind of program, an evaluation would gather and analyze data about that program’s effectiveness. But program evaluation is only one kind of evaluation.

 

What are the Different Kinds of Evaluation?

All of us have conducted some sort of evaluation, whether formally or not. We do it almost every day when we decide what to wear or how to prioritize the various tasks that lay before us. A more specific example is when it comes to purchasing expensive items such as a car or home.  We tend to weigh various criteria in order to make a decision, for example, price, location, number of rooms in the case of a house or miles per gallon and safety features in a car. That’s evaluation. The evaluation profession has developed systematic methods and approaches that can be used to inform judgments and decisions.  Because making judgments and decisions is involved in everything people do, evaluation is important in every discipline, field, profession and sector, including government, businesses, and not-for-profit organizations. 

Different types of evaluation include product evaluation, program evaluation, policy evaluation, and personnel evaluation. Personnel evaluations aim to make people more effective. Product evaluations help inform consumer decisions. Policy evaluation helps policy makers judge the effectiveness and consequences of specific policies. So, there are many different types of evaluation depending on the purpose of the evaluation and what is being evaluated. Program evaluations, as illustrated earlier, can improve program effectiveness, efficiency, and results. Examples of different kinds of evaluation questions include:

  • What is the quality of program or policy implementation?
  • What outcomes are being achieved?
  • Are the real needs of people being met?
  • What works for different people in what ways and under what conditions?
  • How do cultural and diversity variations affect what is done and achieved?
  • What are the costs and benefits of a program, policy, product, or training effort for personnel evaluation?
  • What unintended consequences or negative side effects are appearing that need to be addressed?
  • What are key success factors that others can learn from and use?

These are just a few of the many kinds of evaluation questions that can be asked – and answered with evaluation information and data.

How are evaluations used? 

Evaluations are used in different ways depending on the primary purposes for the evaluation. Evaluations can be used to monitor how an effort is progressing, like tracking implementation of a vaccination campaign. Sometimes evaluations improve a program by getting and using feedback from participants in the program, like a professional development course or parent education program. Evaluation can contribute to formulating a new policy or designing a program by finding out from diverse people in a community what their needs and concerns are. Evaluation used for accountability ensures that funds have been properly and spent to accomplish expected outcomes, like ensuring that a recycling campaign accomplishes targeted reductions in waste. Decision makers can use evaluation findings to inform a major decision about whether to continue, expand, or end a program, like whether to continue an innovative community policing project. And evaluations are used to learn lessons about what works and doesn’t work, like identifying key success factors in a campaign to get high school students to stop smoking. Evaluations can also capture and report the diverse experiences and perceptions of people with different backgrounds, those who share a particular culture, people with disabilities, and the poor and disadvantaged.  Evaluators have developed special approaches to ensure that the experiences and views of diverse groups are included in evaluation findings.

 

Evaluation’s Value and Benefits

Governments, businesses, not-for-profit agencies, philanthropic foundations, and international organizations around the world use evaluation evidence to find out what is and is not effective. This helps them make decisions about how best to allocate scarce resources, develop staff, choose quality products they need, and more effectively meet people’s needs. Independent evaluations can increase public confidence that they are getting credible information about how funds are being spent, what is being accomplished, and what is not being accomplished. Culturally-sensitive evaluations ensure that different points of view and diverse experiences are communicated and taken into account. Evaluations help funders determine if the money they’ve provided has been well spent to accomplish what they intended.  Participatory evaluations help people in programs and communities reflect together on how programs and policies affect them, and more effectively communicate their findings to improve services they receive. Ethical evaluations ensure that people are treated fairly when data are gathered and reported.  In all of these cases, the value and utility of an evaluation is increased when evidence is gathered systematically and ethically, appropriate and relevant data are collected, the analysis is genuinely fair and balanced, and the evaluation includes diverse perspectives so that the findings are credible. Credibility is essential for utility.

The benefits of evaluation extend beyond a particular project when the findings are used to expand the project to a larger number of communities. For example, positive findings from a pilot program can be used to support dissemination and expansion of the program, as when a pilot parent education program becomes a national model based on evaluation of its effectiveness. Worldwide, evaluations are used by governments, nongovernmental organizations, and international agencies to enhance the impacts of development aid.  The basic value and underlying theme of these many different kinds of evaluation, in widely diverse places, is assessing whether people’s lives are getting better.  

 

Who Does Evaluation?

Evaluators come from diverse backgrounds, bringing to the profession a wide variety of experiences, training and skills, as well as diverse cultural, ethnic, and community backgrounds. You’ll find evaluators representing the full range of disciplines and professions such as sociology, political science, economics, psychology, communications, management, information technology, health sciences, education, organizational development, and natural sciences, among others.  Evaluators draw on the methods and theories of these diverse disciplines and professions to design and conduct appropriately relevant and rigorous evaluations.

Evaluators may work within an organization (internal evaluators) or be commissioned under contract (external evaluators). Some evaluators are affiliated with a consulting firm while others are independent consultants. Some work in nonprofit or governmental organizations, and others work in academic or research settings. Some work in private industry, such as quality assurance specialists in businesses and hospitals. They often have a graduate degree, either a masters or a doctorate, but some do not.

Currently, there is no official licensing body for evaluators. Therefore, some people in the evaluation field might not necessarily have the appropriate training and experience.  Organizations are encouraged to always check the credentials of the evaluators they are planning to engage to assure that they have the appropriate methodological skills, cultural competence, specialized knowledge, and professional training to competently and credibly conduct the evaluation in accordance with the standards and principles of the evaluation profession.                                                                                                     

 

What is the American Evaluation Association?

 

Click here to learn more about the American Evaluation Association.

 

AEA offers a series of Guiding Principles for evaluators. The Principles describe agreed-upon criteria of excellence endored by AEA. See: http://www.eval.org/p/cm/ld/fid=51

 

Please offer your thoughts on this document by hitting the comment button below

 

16 Comments

Welcome

January 30, 2014 08:19 AM by Michael Quinn Patton

On behalf of the AEA Task Force that developed this statement, we want to emphasize that the purpose of this blog is to generate discussion on the important but often difficult challenge of defining and communicating what evaluation is and what evaluators do.  We see this statement as a living document.  Based on your comments, we will periodically revise the statement.


Here are some questions to generate discussion:   What aspects of the statement do you particuarly like as is?  What, if anything, is unclear or confusing?  What would you like to see added?  What deleted?  What changed?  


Overall, what are your reactions to both the task (defining evaluation) and this process (making this statement a matter of community dialogue and engagement)?


Michael Quinn Patton, Utilization-Focused Evaluation

Good start!

February 3, 2014 11:10 AM by Chad T. Green, PMP

We need a more compelling growth narrative to validate the context of our work.  Why is evaluation critical in the 21st century? What gap does it fill?  Which creative tensions does it enable?  Traditionally evaluators have worked primarily with vulnerable populations. Today we can all fit that category. A little content on the initial conditions of our field would be helpful.


What about our growth internationally along with some compelling statistics? Also AEA's guiding principles could be fleshed out more rather than merely listed at the end as a link.


Best,


Chad

What is Evaluation

February 6, 2014 10:51 AM by David Robinson

The work of this task force is very valuable, and the statement is very thorough and compelling.  I can imagine the difficulty in gaining consensus on defining evaluation for everyone.  The reasons one evaluates are so diverse, and the philosophical and theoretical approaches guiding evaluators' practice and products are complex.  


But I am especially gratefull for the decision to publish the statement in blog format for others to comment about.  Thank you for your efforts to guide and provoke us to do a better evaluation.

Definition

February 6, 2014 11:01 AM by Robert K. Walker

Compare the definition "Evaluation is a systematic process to determine merit, worth, value or significance" with Scriven (2013), "the determining or asserting of a claim about the merit (a.k.a., roughly speaking, quality), worth (a.k.a., in one sense, value), or significance (a.k.a., approximately, importance of some entity." "Worth" comes from the Anglo Saxon "weorth" (honorable), while "value" comes from Latin. Nowadays they are practically, or at least partially (in which sense, Michael?) synonymous (so I have a hard time differentiating them in Portuguese, a romance language). Why not condense them into one term, as Scriven seems to have done? What is the value (worth) of including both in the definition?


Robert K. Walker


Brasilia


 

definition of evaluation

February 6, 2014 11:48 AM by Sandra Mathison

Kudos to all for this work! I applaud the democratic, inclusive strategy the committee and AEA has taken.


I've written a longer blog post responding to this definition... http://blogs.ubc.ca/evaluation/ in which I outline two areas for further discussion. In effect, both of these are about defining terms included in the definition. One has already been mentioned... that is, clarification about what the meanings of merit, worth, value and significance are. The other is what it means to be "systematic"?

Reflecting Evaluation's scope of practice

February 6, 2014 02:41 PM by Jasper Buys

I am referring to the second paragraph under the heading "What is Evaluation?"


As an evaluator that largely practices within the Organizational Development context, I found this paragraph's examples to be limited to more traditional evaluation contexts. If the aim of this paragraph is to highlight the breadth of Evaluation contexts, I would suggest that examples outside of the public sector and not for profits be considered.

what is evaluation

February 6, 2014 07:21 PM by gene shackman

If the goal is mainly to communicate to the public what evaluation is, this part of the statement is good, and is enough for the question "what is evaluation"


"Different types of evaluation include product evaluation, program evaluation, policy evaluation, and personnel evaluation. Personnel evaluations aim to make people more effective. Product evaluations help inform consumer decisions. Policy evaluation helps policy makers judge the effectiveness and consequences of specific policies. So, there are many different types of evaluation depending on the purpose of the evaluation and what is being evaluated. Program evaluations, as illustrated earlier, can improve program effectiveness, efficiency, and results. Examples of different kinds of evaluation questions include:



  • What is the quality of program or policy implementation?

  • What outcomes are being achieved?

  • Are the real needs of people being met?

  • What works for different people in what ways and under what conditions?

  • How do cultural and diversity variations affect what is done and achieved?

  • What are the costs and benefits of a program, policy, product, or training effort for personnel evaluation?

  • What unintended consequences or negative side effects are appearing that need to be addressed?

  • What are key success factors that others can learn from and use?


These are just a few of the many kinds of evaluation questions that can be asked – and answered with evaluation information and data."

WHAT IS EVALUATION?

February 6, 2014 08:08 PM by ROBERT PICCIOTTO

A big thank you for re-opening this debate. The statement is an excellent start. Securing broad based agreement on the definition (Evaluation is a systematic process to determine merit, worth, value or significance) would be a major achievement, However in this context I agree with Robert Walker. Merit, worth and value (or significance) need to be defined perhaps in a glossary attached to the statement. In this connection I would favor a reference to the goals of the intervention and - in line with the well tested approach of development evaluation - I would define worth as the relevance of the goals; merit as the effectiveness of the intervention design and implementation arrangements in terms of the likelihood of achieving the relevant goals and value as the efficiency of the intervention relative to alternative approaches in terms of resource use. Focusing on goals strengthens accountability and emphasizes the need to engage with stakeholders. Focusing on the relevance of the goals is what most distinguishes evaluation from auditing. Highlighting efficiency relative to counterfactuals is critical especially when resources are scarce. These definitions have been shown to work for all kinds of evaluands (policies, programs, projects, organizations, etc.) and to facilitate debate among stakeholders about all pertinent dimensions of the intervention and in a wide variety of contexts.   

Is ethical evaluation a kind of evaluation?

February 6, 2014 11:05 PM by Moein

Salaam,


I think “Ethical evaluations” in this sentence: “Ethical evaluations ensure that people are treated fairly when data are gathered and reported.” is confusing.


Best


Moein


 

Great Discussion

February 7, 2014 02:25 AM by David Roberts

I applaud the committee for a very thorough job of trying to tease out what evaluation might be.   I have one question and one comment.


The question is how do we answer the question "What is evaluation?" in an elevator speech?


The comment is: I can't use the definition for my elevator speech because it is misleading to anyone who is not part of the discipline, fundamentally inaccurate and inherently arrogant.  I have a fundamental problem with the notion that we can 'determine' the 'merit, worth, value or signficance' of anything.  Scriven's masterful formulation, while it   stresses the importance of criteria to inform  judgements, recognises that 'merit' etc are value judgements and that we make assertions of those judgements.  He neverthless appears to accept the idea of determination.


I do not.  At best we offer a judgement, that is an opinion, an opinion informed by the evidence we have collected and assessed: and almost always by our experience (lets not pretend!!).  but it should always be (as in science) a provisional opinion and subject to change as further or better evidence is adduced.


Then there is the question of why our values and judgements should have any higher status than the values of others.  It is the evidence that is our strength and helps us to make judgements that can be useful and valuable.   

Definition of Evaluation

February 7, 2014 07:19 PM by Oumoul




Dear colleagues,


I would like to congratulate AEA, MQP and his team for this initiative...YES! I find it relevant to open the debate (or reopen as some have said)...my hope is that we, as a professional community, commit to achieve a commun or at least widely shared definition of Evaluation and some sort of classification so that every one who practices Evaluation could fit into it, so "Bravo Michael" for a good start.  


I have one comment and a question, for now. You wrote "Currently, there is no official licensing body for evaluators"...and I wonder if you have taken into account the Canadian (CES) credentialing programme. My understanding was that this is intended to serve as a "licensing" tool, isn't it?....I am sure Martha and Ian Davies (to name just a few) will react to clarify my understanding.


I will come back on the issue of classification, I trust it is a major element a any definition, and thus far, I am confused as I do not see a clear coherent model where all practices could fit nicely...the most confusing to me is the so-called "Development Evaluation"...the more I read and "practice" about it, the less I understand what the core difference is with, say, "Programme Evaluation"...so I wonder if we are not just saying the same thing, of course under different context, and we give it different names.


Oumou


Mauritania


 






 

a textbook definition that I've always liked

February 9, 2014 03:06 PM by Bernadette M Wright

"Evaluation research is the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social interventions. In other words, evaluation researchers (evaluators) use social research methodologies to judge and improve the ways in which human services policies and programs are conducted, from the earliest stages of defining and designing programs through their development and implementation." ("Evaluation: A Systematic Approach", Rossie, Lipsey, & Freeman, 1993, p. 5)

Feedback acknowledged and appreciated

February 16, 2014 12:18 PM by MQP

Thanks for your thoughtful entries, reactions, and feedback.  The comments illustrate the challenge.  We'll probably try a revision after the annual AEA conference in tme for 2015 and the International Year of Evaluation.  In the meantime, keep the comments and questions coming.  MQP

From a newcomer's perspective

February 18, 2014 05:01 PM by Sybil Chan

As someone who is still very new to evaluation and involved primarily as a support staffperson within a Quality Improvement department, I enjoyed reading through the definition and comments. Clearly a lot of work was put into defining this subject and making it broad, understandable and accessible to the average person. I liked seeing the examples in the narrative as well. Interesting point in the comments about sounding lofty/arrogant when determining "merit, worth, value, or significance" without further expanding on those definitions in relation to evaluation, but obviously standards must be made and people have to make judgments about the functioning of a program (/service/product/etc.) in relation to its goal. From what I understand, the value of an evaluator comes from the fact that he/she is a third party who is, ideally, being trusted to make an impartial judgment, leading to appropriate actions for maintenance or improvement.


Also thinking about the fact that evaluators come from such a wide range of backgrounds and credentials -- what makes a good and competent evaluator? All of this leading to a question I can personally reflect on.... how can I become a better evaluator and explore and expand my skills in this regard?



 







 







 


 


 


 


 

I know it when I see it - not!

April 6, 2014 02:05 PM by Hans-Martin Boehmer, Ph.D.

First of all let me applaud this effort.  I’m sure there are many of us who are being asked “so what exactly is evaluation.”  Almost inevitably, using popular examples, such as the ‘evaluation’ of the ladies figure skating performances at the recent Sochi Olympics, the response is something along the lines of “so it all depends on who likes it and who doesn’t?”


I like the opening paragraph and have only one suggestion to make.  The wording “Programs and projects of all kinds aspire to make the world a better place,” both risks underplaying the importance of the word ‘aspire’ and underplaying the importance of many programs that may not be aimed at making the world a better place.  Nothing against the private sector, but much of the private sector depends on others for the appropriate regulatory framework and private sector evaluation may have different goals. 


What is so important about the word ‘aspire’ is the consensus it requires to agree on what exactly a program or project that is to be evaluated ‘aspires’ to achieve.  This distinguishes evaluation from merely an ex post evidence-based statement of likes and dislikes. 


Perhaps the statement could bring in earlier the notion of evaluability and generally be more inclusive of the valuable work in the area of private sector evaluations.


Overall, well done!


         Hans-Martin Boehmer


 

What is Evaluation?

January 31, 2015 11:42 AM by Albert-Eneas Gakusi

I have just read the statment that I found very useful to explain what is meant by evauation. Thus, I have distributed it to colleagues.

In the phrase below, I think that the ethical aspects should be also be considered as key a determinant to hire evaluators or managers of evaluations.

“Organizations are encouraged to always check the credentials of the
evaluators they are planning to engage to assure that they have the appropriate
methodological skills, cultural competence, specialized knowledge, and
professional training to competently and credibly conduct the evaluation in
accordance with the standards and principles of the evaluation profession.”

All the best,

 

Eneas

Recent Stories

What is Evaluation?