This statement was developed by a previous AEA Task Force commissioned by the AEA Board to “define and communicate the value of evaluation to the media, the public, and other audiences as well as be used comfortably by evaluators throughout the field without regard for specialty or area of expertise.” The statement is meant to encourage dialogue -- so based on comments and responses it will be revised periodically. Thus, your reactions and comments are encouraged (see comment section below).The Task Force was comprised of both long-time evaluation professionals and AEA members newer to the profession. All have experience and expertise in communicating to others about evaluation. The task force included:
Michael Quinn Patton, Chair, Edith Asibey, Jara Dean-Coffey, Robin Kelley, Roger Miranda, Susan Parker, and Gwen Fariss Newman
What is Evaluation?
Evaluation is a systematic process to determine merit, worth, value or significance. So what does that mean in practice? Let’s use one kind of evaluation, program evaluation, to illustrate. Programs and projects of all kinds aspire to make the world a better place. Program evaluation answers questions like: To what extent does the program achieve its goals? How can it be improved? Should it continue? Are the results worth what the program costs? Program evaluators gather and analyze data about what programs are doing and accomplishing to answer these kinds of questions.
A program evaluation has to be designed to be appropriate for the specific program being evaluated. Health programs aim to make people healthier and prevent disease. School programs strive to increase student learning. Employment training programs try to help the unemployed get jobs. Homelessness initiatives work to get people off the streets and into safe housing. Chemical dependency programs help people using alcohol and drugs. Community development programs plan initiatives to increase prosperity among those in poverty. Juvenile diversion programs try to keep kids out of jail and put them on a path to becoming productive adults. For each kind of program, an evaluation would gather and analyze data about that program’s effectiveness. But program evaluation is only one kind of evaluation.
What are the Different Kinds of Evaluation?
All of us have conducted some sort of evaluation, whether formally or not. We do it almost every day when we decide what to wear or how to prioritize the various tasks that lay before us. A more specific example is when it comes to purchasing expensive items such as a car or home. We tend to weigh various criteria in order to make a decision, for example, price, location, number of rooms in the case of a house or miles per gallon and safety features in a car. That’s evaluation. The evaluation profession has developed systematic methods and approaches that can be used to inform judgments and decisions. Because making judgments and decisions is involved in everything people do, evaluation is important in every discipline, field, profession and sector, including government, businesses, and not-for-profit organizations.
Different types of evaluation include product evaluation, program evaluation, policy evaluation, and personnel evaluation. Personnel evaluations aim to make people more effective. Product evaluations help inform consumer decisions. Policy evaluation helps policy makers judge the effectiveness and consequences of specific policies. So, there are many different types of evaluation depending on the purpose of the evaluation and what is being evaluated. Program evaluations, as illustrated earlier, can improve program effectiveness, efficiency, and results. Examples of different kinds of evaluation questions include:
- What is the quality of program or policy implementation?
- What outcomes are being achieved?
- Are the real needs of people being met?
- What works for different people in what ways and under what conditions?
- How do cultural and diversity variations affect what is done and achieved?
- What are the costs and benefits of a program, policy, product, or training effort for personnel evaluation?
- What unintended consequences or negative side effects are appearing that need to be addressed?
- What are key success factors that others can learn from and use?
These are just a few of the many kinds of evaluation questions that can be asked – and answered with evaluation information and data.
How are evaluations used?
Evaluations are used in different ways depending on the primary purposes for the evaluation. Evaluations can be used to monitor how an effort is progressing, like tracking implementation of a vaccination campaign. Sometimes evaluations improve a program by getting and using feedback from participants in the program, like a professional development course or parent education program. Evaluation can contribute to formulating a new policy or designing a program by finding out from diverse people in a community what their needs and concerns are. Evaluation used for accountability ensures that funds have been properly and spent to accomplish expected outcomes, like ensuring that a recycling campaign accomplishes targeted reductions in waste. Decision makers can use evaluation findings to inform a major decision about whether to continue, expand, or end a program, like whether to continue an innovative community policing project. And evaluations are used to learn lessons about what works and doesn’t work, like identifying key success factors in a campaign to get high school students to stop smoking. Evaluations can also capture and report the diverse experiences and perceptions of people with different backgrounds, those who share a particular culture, people with disabilities, and the poor and disadvantaged. Evaluators have developed special approaches to ensure that the experiences and views of diverse groups are included in evaluation findings.
Evaluation’s Value and Benefits
Governments, businesses, not-for-profit agencies, philanthropic foundations, and international organizations around the world use evaluation evidence to find out what is and is not effective. This helps them make decisions about how best to allocate scarce resources, develop staff, choose quality products they need, and more effectively meet people’s needs. Independent evaluations can increase public confidence that they are getting credible information about how funds are being spent, what is being accomplished, and what is not being accomplished. Culturally-sensitive evaluations ensure that different points of view and diverse experiences are communicated and taken into account. Evaluations help funders determine if the money they’ve provided has been well spent to accomplish what they intended. Participatory evaluations help people in programs and communities reflect together on how programs and policies affect them, and more effectively communicate their findings to improve services they receive. Ethical evaluations ensure that people are treated fairly when data are gathered and reported. In all of these cases, the value and utility of an evaluation is increased when evidence is gathered systematically and ethically, appropriate and relevant data are collected, the analysis is genuinely fair and balanced, and the evaluation includes diverse perspectives so that the findings are credible. Credibility is essential for utility.
The benefits of evaluation extend beyond a particular project when the findings are used to expand the project to a larger number of communities. For example, positive findings from a pilot program can be used to support dissemination and expansion of the program, as when a pilot parent education program becomes a national model based on evaluation of its effectiveness. Worldwide, evaluations are used by governments, nongovernmental organizations, and international agencies to enhance the impacts of development aid. The basic value and underlying theme of these many different kinds of evaluation, in widely diverse places, is assessing whether people’s lives are getting better.
Who Does Evaluation?
Evaluators come from diverse backgrounds, bringing to the profession a wide variety of experiences, training and skills, as well as diverse cultural, ethnic, and community backgrounds. You’ll find evaluators representing the full range of disciplines and professions such as sociology, political science, economics, psychology, communications, management, information technology, health sciences, education, organizational development, and natural sciences, among others. Evaluators draw on the methods and theories of these diverse disciplines and professions to design and conduct appropriately relevant and rigorous evaluations.
Evaluators may work within an organization (internal evaluators) or be commissioned under contract (external evaluators). Some evaluators are affiliated with a consulting firm while others are independent consultants. Some work in nonprofit or governmental organizations, and others work in academic or research settings. Some work in private industry, such as quality assurance specialists in businesses and hospitals. They often have a graduate degree, either a masters or a doctorate, but some do not.
Currently, there is no official licensing body for evaluators. Therefore, some people in the evaluation field might not necessarily have the appropriate training and experience. Organizations are encouraged to always check the credentials of the evaluators they are planning to engage to assure that they have the appropriate methodological skills, cultural competence, specialized knowledge, and professional training to competently and credibly conduct the evaluation in accordance with the standards and principles of the evaluation profession.
What is the American Evaluation Association?
Click here to learn more about the American Evaluation Association.
AEA offers a series of Guiding Principles for evaluators. The Principles describe agreed-upon criteria of excellence endored by AEA. See: http://www.eval.org/p/cm/ld/fid=51
Please offer your thoughts on this document by hitting the comment button below