EXTERNAL CITATIONS and USE OF EPTF WORK

AEA published it's most recent version of An Evaluation Roadmap for a More Effective Government in September 2010. AEA and the Evaluation Policy Task Force have used it in commenting on emerging national policies such as maternal, infant, and children's home-visiting programs, health care reform, and foreign assistance. A broader goal was to use it as AEA's "place at the table" when evaluation policies were being considered or would be appropriate, even if an AEA representative couldn't be there in person. The latter goal is now being realized.

The Roadmap has been cited in congressional testimony, Government Accountability Office (GAO) reports, and other settings. Below are examples we know about. Do you know of other explicit citations of the work of the Roadmap? We would welcome hearing from you. Please send them to info@eval.org. Thank you for your help.

NOVEMBER 2012: United States Department of Health and Human Services Administration for Children and Families - Evaluation Policy
Newly issued policy cites the Roadmap, in particular around its assertion that "Independence and objectivity are core principles of evaluation."

]MAY 2012: United States Government Accountability Office – President's Emergency Plan for AIDS Relief: Agencies Can Enhance Evaluation Quality, Planning, and Dissemination
Multiple references to the Roadmap, with the Roadmap used as a primary guiding document and the only guiding document from outside the US Federal Government. Examples include: "To address these objectives, we reviewed the American Evaluation Association’s (AEA) An Evaluation Roadmap for a More Effective Government (AEA Roadmap) 5 as well as policies and guidance developed by the Department of State (State), State’s Office of the U.S.Global AIDS Coordinator (OGAC), the Department of Health and Human Services’ (HHS) Centers for Disease Control and Prevention (CDC), and the U.S. Agency for International Development (USAID)." [page 2] "...we assessed State, OGAC, CDC, and USAID policies and practices against selected general principles of evaluation defined in the AEA Roadmap." [page 3]
http://www.gao.gov/assets/600/591284.pdf

FEBRUARY 2012: Congressional Research Service - Changes to the Government Performance and Results Act (GPRA): Overview of the New Framework of Products and Processes
"This tendency [to focus on goal-setting and performance measurement versus evaluation] led the evaluation community to argue for elevating the salience of program evaluation in federal officials and organizations. See American Evaluation Association, An Evaluation Roadmap for a More Effective Government..." [Footnote 33 "GPRAMA continues with GPRA 1993’s emphasis on goal-setting and performance measurement, with little explicit emphasis on producing or presenting program evaluations. Under GPRA 1993, this sometimes could leave unclear how well programs themselves were performing, why, and what might be done in response." [page 33 which then cites the following footnote] "American Evaluation Association, An Evaluation Roadmap for a More Effective Government... In practice, however, it may be noted that evaluation and measurement often are complementary..." [footnote 37]
http://www.fas.org/sgp/crs/misc/R42379.pdf

OCTOBER 2011: Independent Evaluation Group - World Bank - Evaluation Capacity Development: Monitoring and Evaluation in the United States Government - an Overview
"Managers need to be trained in the design and use of evaluations and to learn the importance of incorporating plans for their conduct at the outset of program operations to guide program and managerial priorities and information design, as well as to assess program impacts after implementation. Expert, independent professional associations like the American Evaluation Association in the United States, which presented the incoming Obama Administration with ―An Evaluation Roadmap for a More Effective Government, (AEA 2009), are a valuable resource that the government could use to help develop solid strategies for implementing such efforts. [report page 35]
http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2011/11/15/
000333037_20111115000822/Rendered/PDF/654110NWP0260U0C0disclosed011040110.pdf

MAY 2011: United States Government Accountability Office – International School Feeding: USDA's Oversight of the McGovern-Dole Food for Education Program Needs Improvement
"The American Evaluation Association's An Evaluation Roadmap for a More Effective Government recommends that agencies develop policies and procedures to guide evaluation and assess the strengths and weaknesses of programs to improve their effectiveness." [from summary] Goes on to discuss USDA's efforts in this area. The Roadmap is referenced in the text, summary, and opening page of findings. "Finally, we compared USDA’s oversight and internal control practices to our Standards for Internal Control in the Federal Government and the American Evaluation Association’s An Evaluation Roadmap for a More Effective Government." [report page 2]
http://www.gao.gov/products/GAO-11-544  

APRIL 2011: The Economist - Economist Intelligence Unit – Creating Value in the Public Sector: Intelligent Project Selection in the US Federal Government
"Programme evaluation would profit from an influx of trained and dedicated people. Most of the agency planning offices that carry out the project evaluation and analysis now rely on a very small workforce, which will be stretched even further with the demands from the OMB, the new GPRA and Congress. The American Evaluation Association made the same point in its Evaluation Roadmap. The units formed by agencies to conduct evaluations are too often under-resourced. Training and capacity building for evaluation have been inconsistent across agencies 'and, in many cases, insufficient to achieve the needed evaluation capacity and to sustain it over time', the association reports." [page 12]
http://www.businessresearch.eiu.com/sites/businessresearch.eiu.com/files/downloads/
Oracle_PublicSector_WEB.pdf 

MARCH 2011: United States Government Accountability Office – Employment and Training Administration: More Actions Needed to Improve Transparency and Accountability of Its Research Program
“While there is no single or ideal way for government agencies to conduct research, several leading national organizations have developed guidelines that identify key elements that promote a sound research program. These guidelines identify five elements as key: agency resources, professional competence, independence, evaluation policies and procedures, and evaluation plans.” [page 6] Cites Roadmap in accompanying footnote.
http://www.gao.gov/new.items/d11285.pdf

FEBRUARY 2011: Grantmakers for Effective Organizations – Scaling What Works - Briefing Paper – What Do We Mean by Scale?
"Choosing from among the extensive set of analytic approaches and methods developed by the evaluation field over many years of practice— including logic models, case studies, surveys, quasiexperimental designs and needs assessments— depends on what is being scaled, its implementation stage, how results will be used and the kinds of decisions an evaluation is meant to facilitate." [page 5] Roadmap cited as basis for this quote.
http://www.geofunders.org/document.aspx?oid=a0660000005uSOAAA2

JANUARY 2011: United States Government Accountability Office – Program Evaluation: Experienced Agencies Follow a Similar Model for Prioritizing Research
"These [Department of Education] plans align well with the American Evaluation Association’s (AEA) recommendation, made in a recent policy paper on federal government evaluation, that federal agencies prepare annual and multiyear evaluation plans to guide program decision-making and consult with the Congress and nonfederal stakeholders in defining program and policy objectives, critical operations, and definitions of success." [page 15] Cites Roadmap in accompanying footnote.
http://www.gao.gov/new.items/d11176.pdf

JANUARY 2011: IBM Center for the Business of Government – Jonathan Breul in a blog post on Producing an Effective Program Evaluation Agenda
Highlights GAO's Program Evaluation: Experienced Agencies report, adds as a final aside "In addition, you might check out the American Evaluation Association’s Evaluation Roadmap for a More Effective Government.”
http://www.businessofgovernment.org/blog/business-government/producing-effective-program-evaluation-agenda  

DECEMBER 2010: NASA – NASA Informal Education: Final Report
"The call for improved evaluation practices is strongly conveyed in the American Evaluation Association’s (AEA) paper, An Evaluation Roadmap for a More Effective Government (2009). In it, the AEA Evaluation Policy Task Force outlines a framework and set of guiding principles for each federal agency to inform the development of its own evaluation program. Moreover, it advocates that evaluation be used to inform a variety of decisions, all of which are particularly relevant to NASA’s Informal Education Program...Although our discussions with NASA OE personnel suggest that their vision of the role that evaluation would serve is in sync with the purposes outlined by the AEA Roadmap, it is clear this vision is not being implemented in a way that allows for effective practice." [page 50]
http://www.nasa.gov/pdf/511273main_2010_InformalEvaluation.pdf  

SEPTEMBER 2010: Brookings Institution Center on Children and Families and the National Institute for Early Education Research - Investing in Young Children: New Directions in Federal Preschool and Early Childhood Policy, Chapter on Strengthening Home-Visiting Intervention Policy: Expanding Reach, Building Knowledge
"Defining the evidentiary base necessary for estimating the potential impacts of a given intervention is complex and particularly challenging when the reform involves multiple strategies. Randomized control trials are often the best and most reliable method for determining whether changes observed in program participants over time are due to the intervention rather than to other factors. Maximizing the utility of program evaluation efforts, however, requires more than just randomized clinical trials. As noted by the American Evaluation Association in a February 2009 memo to Peter Orszag, the Director of the Office of Management and Budget: 'There are no simple answers to questions about how well programs work and there is no single analytic approach or method that can decipher the complexities that are inherent within the program environment and assess the ultimate value of public programs.'" [page 85]
http://www.brookings.edu/reports/2010/1013_investing_in_young_children_haskins.aspx  

MARCH 2010: United States Government Accountability Office – Afghanistan Drug Control: Strategy Evolving and Progress Reported, but Interim Performance Targets and Evaluation of Justice Reform Efforts Needed
"To assess U.S. monitoring and evaluation of counternarcotics programs, we first utilized, as a framework, the Government Performance and Results Act, which outlines good management practices such as establishing strategic, long-term goals and planning and reporting progress toward these goals on an annual basis. We also referenced good management practices outlined within previous GAO reports and considered monitoring and evaluation principles established by the American Evaluation Association." [page 42] Cites Roadmap in accompanying footnote.
http://www.gao.gov/new.items/d10291.pdf

SEPTEMBER 2009: United States Government Accountability Office – International Food Assistance: USAID Is Taking Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid, but Weaknesses in Planning Could Impede Efforts
Roadmap is cited explicitly throughout as a core framing document for the GAO's review including providing principles for evaluation policies and procedures [page 26], for independence [page 27], for scope and coverage [page 28], for dissemination of results and for professional competence [page 30], for resources [page 33], for evaluation plans [page 34].
http://www.gao.gov/new.items/d09980.pdf 

JUNE 2009: US House of Representatives – Hearing Before the US House of Representatives Ways and Means Committee Subcommittee on Income Security and Family Support on Proposals to Provide Federal Funding for Early Childhood Home Visitation Programs
“We considered established principles for monitoring and evaluation—especially the “Roadmap” principles of the American Evaluation Association (AEA), which have been developed to integrate monitoring and evaluation with program management…” [page 4]
http://www.gpo.gov/fdsys/pkg/CHRG-111hhrg52502/pdf/CHRG-111hhrg52502.pdf  

2009: Wellcome Trust Project – Developing Metrics and Measures for Dual-Use Education
"It is apparent that metrics, measures and evaluation have become increasingly salient in the US where there is evidence of the emergence of a shift towards revising and enhancing the process of evaluation and assessment of government projects. The American Evaluation Association’s (AEA) Evaluation Policy Task Force is worth quoting at length in this regard and in the 2009 report, An Evaluation Roadmap for a More Effective Government the AEA state: [extended Roadmap quote on the effectiveness of federal programs]"  [page 8]
http://www.brad.ac.uk/bioethics/media/SSIS/Bioethics/docs/
Education_metrics_and_measurs_2nd_DRAFT.pdf 
 

American Evaluation Association│16 Sconticut Neck Rd #290 │Fairhaven MA 02719
1-888-232-2275 │1-508-748-3326 │info@eval.org