AEA Newsletter: October 2018

Message from the Executive Director

Looking Forward - 2019 Perspective for AEA

Anisha Lewis.jpg
From Anisha Lewis, AEA Executive Director 

What a great time to be a member of AEA! This week, evaluators from around the world are assembled in Cleveland, Ohio, at AEA’s 32nd Annual conference. As we convene to learn and share, members are utilizing this event as an opportunity to establish a community with colleagues that will promote continual development and networking.

The annual conference is also a time to celebrate our accomplishments as an association, some of which include:

  • Approval and acceptance of updated Guiding Principles
  • Approval of the Competencies for Evaluators
  • Conducted seven (7) Town Hall meetings to inform members on issues affecting the association
  • Conducted 14 coffee breaks
  • Conducted nine (9) eStudies
  • Launched the Issues and Ideas Portal via the AEA website for members to communicate ideas and issues to the Board of Directors.
  • Three (3) task forces completed their work (Guiding Principles, Competencies and Membership Engagement, Leadership and Diversity)
  • Established two new Topical Interest Groups (TIGs): Graduate Education Diversity Internship (GEDI) and Health Professions Educations Evaluation and Research (HPEER). 

The conference also gives members the opportunity to participate in the Annual Business Meeting, where they can receive updates from AEA leaders about strategic initiatives and most important, share their voice to inform planning of the association.

Unfortunately, not everyone is able to attend the conference due to budgetary and timing constraints. However, we are continually working to provide additional opportunities for members to have a voice as we plan strategic initiatives that will affect the future of AEA.

If you are not able to attend the Annual Business Meeting, please join us in November for our Virtual Town Hall Meeting (date and time TBA), which will focus on strategic initiatives for the upcoming year. 

Below are some key initiatives for 2019:

  • Recruitment and development of students and entry level professionals – the future of the evaluation industry and the association
  • Increasing revenue to support the growth of the association
  • Consolidating member offerings to maximize the value of AEA membership
  • Development of a communications plan to help message our value to the world
  • Laying the framework for a new website
  • Launching a new and improved online career center
  • Formation of new working groups to address membership, leadership and diversity initiatives as recommended by task forces. 

Overall, AEA is doing very well! We have a healthy investment reserve, and our membership numbers are steady. As we continue to grow and increase programs and initiatives, we must ensure that our revenue streams support our growth and the work of the association. As we plan on ways to accommodate rising costs, we want to do so informed by ideas for increasing member value. Please be sure to attend our November Town Hall Meeting so that you can weigh in on these important issues. 

Many thanks to you for choosing to be a member of AEA. It is through your dedicated support that we continue to grow and champion the profession of evaluation! 

 

The Face of AEA

Meet Chantel DePaepe

Chantel DePaepe.jpgThe Face of AEA spotlights our members and their backstories - why they joined the profession, what drives them and memorable lessons they've learned along the way. Know someone who should be featured? Email the AEA editor, Kristin Fields, at kfields@eval.org

Name: Chantel DePaepe
Affiliation: American Association of Nurse Practitioners (AANP)
Degrees: MPH
Years in the Evaluation Field: 2
Joined AEA: 2018

Why do you belong to AEA?

I belong to AEA to improve my evaluation skills and to share my knowledge with others.

What is the most memorable or meaningful evaluation you have been a part of? 

The most meaningful evaluation that I have been a part of was for the Transforming Clinical Practices Initiative (TCPI). AANP worked with the National Nurse-Led Care Consortium under a grant from the Centers for Medicare and Medicaid Services (CMS) as part of the TCPI. I helped to evaluate a small part of the program but I learned a tremendous amount about planning, implementing and evaluating a nationwide program, as well as the importance of connecting everything we did back to our goals.

What advice would you give to those new to the field?

It is easy to go off track from your goals. We had stakeholders present wonderful ideas, but their ideas did not align with our goals. Another issue was stakeholders did not have an evaluation plan in mind for their ideas. Therefore, I recommend that throughout every step of the program you keep reminding yourself and your team about what you are trying to accomplish and how you will evaluate what you implement.

 

An Update from the AEA Health Professions Education Evaluation and Research TIG

From Anne Vo, Jessica Harlan, Patrick VBarlow and Asma Ali, HPEER TIG leads 

Approved by the AEA Board in January 2018, the mission of the Health Professions Education Evaluation and Research (HPEER) Topical Interest Group (TIG) is to develop a network of program evaluators working across different areas of health professions education (HPE) to promote greater evaluation quality, relevance and use. HPE is concerned with the training of healthcare professionals across fields such as allied health, dentistry, medicine, nursing, pharmacy and veterinary medicine. These professions share regulatory, cultural and educational objectives that influence the education delivery life cycle, from pre-health professional training through continuing health professional education. Furthermore, HPE programs occur in a variety of learning environments, including traditional classrooms, labs, clinics, hospitals and other field-based settings. Regulatory and attainment of educational objectives also influence the ways in which HPE is delivered and evaluated. These increasingly complex and changing contexts and environments, in turn, contribute to the diversity of the evaluator community within HPE, as well as shape HPE practice, outcomes and evaluation.  

HPEER Graphic.jpg

The TIG aims to provide ongoing support of HPEER evaluators by supporting developing partnerships, scholarly opportunities and discourse among practitioners in the HPE field. The TIG also strives to promote inter-professional approaches, identification of best practices in HPE and provide opportunities for partnerships. Toward this end, the HPEER TIG will focus on the following priorities:

  • Creating an AEA space for discourse among health professions evaluators regarding best practices and partnership opportunities;
  • Building a network and collaborative to support sharing, networking, and development of HPEER evaluators;
  • Supporting the development of scholarly presentations and publication opportunities among HPEER scholars.  

Interested in getting involved? The HPEER TIG will have its first TIG Business Meeting at Evaluation 2018 in Cleveland. The meeting will take place on Thursday, November 1 from 7:30 PM – 9:00 PM in Room CC-25A. If you are not able to make this meeting, please reach out to a member of the HPEER TIG leadership team to learn more.

Chair: 
Jessica M. Harlan, PhD
The Johns Hopkins University School of Medicine
E: jessica.harlan@jhmi.edu

 

Co-Chair:
Asma M. Ali, PhD
AA & Associates, LLC
Educational Development Center [EDC]
E: asma@aa-associates.org

Co-Program Chair:
Anne Vo, PhD
Keck School of Medicine of USC
University of Southern California
E: annevo@usc.edu

Co-Program Chair:
Patrick Barlow, PhD
Carver College of Medicine
The University of Iowa
E: patrick-barlow@uiowa.edu

 

Call for Articles from Topical Interest Groups (TIGs)

Submit an Article or Update for an Upcoming AEA Newsletter 

Do you lead or participate in one of AEA's Topical Interest Groups (TIGs)? We want to hear from you and spotlight your work. Send an email to the AEA editor, Kristin Fields (kfields@eval.org) to share news, updates and articles for consideration in an upcoming AEA newsletter.

Policy Watch

Sessions on Evaluation Policy at Evaluation 2018

From Nick Hart, chair of the Evaluation Policy Task Force (EPTF)

NickHart_withborder.jpgThe last year has been a very active one for evaluation policy in the United States. During the fall conference, Evaluation 2018, AEA’s Evaluation Policy Task Force (EPTF) is sponsoring several opportunities for members to learn about major activities and opportunities to provide feedback. Here are a few of the highlights for the conference:

  • Annual Update on AEA’s Evaluation Policy Task Force, Nov 1 at 10:30 a.m.: EPTF members will offer a brief overview of EPTF’s activities over the past year to influence evaluation policy and will provide updates on upcoming activities of the task force over the next year, with targeted opportunities for member feedback.
  • Innovative Efforts in State-Level Evaluation, Nov. 1 at 11:30 a.m.: Officials from multiple states will offer perspectives about innovative efforts to develop, implement, and maintain strong state-level evaluation policies. Through the course of this discussion, the task force will explore future opportunities to engage with AEA members on state evaluation policy issues. 
  • Executive and Legislative Perspectives on Federal Evaluation Policy, Nov. 1 at 3:45 p.m.: Following the recommendations of the U.S. Commission on Evidence-Based Policymaking, federal agencies and Congress have been working to consider a range of changes in national evaluation policy. The executive branch proposed reforms to institutionalize evaluation practice and Congress is considering pending legislation that would require agencies to have chief evaluation officers and written evaluation policies. ETPF members will offer members updates and an opportunity to provide feedback on various legislative and administrative actions. 
  • Roundtable Discussion on Updating AEA’s Evaluation Roadmap, Nov. 2 at 8:00 a.m.: Much has changed since AEA members approved the “Evaluation Roadmap for More Effective Government” in 2009. Given the changing landscape of state and federal evaluation policy, the task force is considering updates to the Roadmap to ensure its continued relevance. This discussion formally initiates EPTF’s efforts to encourage member feedback about potential revisions to be considered moving forward. 
  • Expert Lecture on Evaluation Policy by George Grob, Nov. 2 at 10:30 a.m.: Navigating the web of procedural and process issues to connect evaluation and public policy can be complicated. Former EPTF Chair George Grob will offer an opportunity for AEA members to learn more about how the policy process works, along with practical advice on opportunities to engage in policymaking.

Future opportunities will also be available to learn more about the EPTF and opportunities for feedback. AEA members can also always submit suggestions through AEA’s Issues and Ideas Portal.

 

Potent Presentations Initiative

Be Present with Your Presenter!

From Sheila B. Robinson, Potent Presentations Initiative Coordinator 

Sheila Robinson-RS 2.png

Evaluation 2018 is in full swing! As you attend the conference in Cleveland, or perhaps attend virtually from anywhere else in the world, consider making it a priority to study the presenters whose sessions you choose. While it can be challenging to focus on the presenter’s message, design, and delivery all at the same time, you can employ simple strategies to capture your key takeaways. 

Good Old-Fashioned Note-Taking

Consider how you track your learning from a typical conference session – Do you take notes? If so, do you use your phone, a tablet, laptop, or paper and pen? 

Depending on the location, the furniture setup and my mood, I use a laptop or a paper notebook. I use one of three configurations during any given session to capture my notes on the session content, and on the presenter characteristics. These configurations are:

  1. Notebook with two different pages assigned to content, and to presenter
  2. Laptop with two Google docs open assigned to content and presenter
  3. One paper notebook page and one electronic device page 

I’ve learned to toggle between my two pages quite easily and it generally doesn’t detract from my experience. If going back and forth is problematic, format just one page with two columns. 

In my “presenter characteristics” notes, I capture how the presenter successfully:

  • interacts with the audience
  • uses interactive strategies
  • organizes content
  • uses humor
  • responds to questions

Of course, I do have some notes on what goes wrong in presentations – my “what not to do” list – but I tend to focus more on what presenters do that I want to emulate and incorporate into my own practice. 

Organize, Review, Reflect

At a later time, organize your “presenter characteristics” notes in one place. If you decide to use paper and pen, you can scan or retype these handwritten notes (this works as an effective review/reflection strategy for me) and store in an electronic folder. 

Every now and then, or when you’re preparing for a presentation, review your notes to look for patterns – what do you tend to pay attention to? How can you incorporate those strategies into your own presentations? 

Pro Tip

If you can manage your attention and not lose out on the content of the presentation, take note of how others in the audience are reacting to the presenter. What makes them laugh? When are they more attentive? When do they tune out? Are their reactions to the presentation and presenter aligned with yours? 

What Do the Rock Stars Do?

In 3 Traits of the World's Most Powerful Speakers author Danny Iny identifies three commonalities from three rock star presenters – Tony Robbins, Mitch Joel, and Nancy Duarte – and offers brief insights from each.

  1. They have exceptional depth of knowledge and insight.
  2. They're genuinely passionate and excited about their content.
  3. They respect the audience. 

The article concludes with video clips of each presenter in action, a great opportunity to practice taking notes on presenter characteristics. And if not these videos, try note-taking on some of your favorite TED talks.

p2i Needs Your Help!

P2i October Image.jpgHave you successfully used p2i tools or p2i principles in your presentations?
Do you have “before” and “after” slide examples you would be willing to share?
Do you have ideas for, or are you interested in writing a blog article on Potent Presentations?
Do you have an interest in sharing your tips for Potent Presentations through a brief video or webinar?

Please contact me at p2i@eval.org and let’s talk! I’m happy to help, offer guidance, or collaborate on any of these. 

 

International Evaluation Update

Reflections from the European Evaluation Society (EES)

In early October, the European Evaluation Society (EES) held its 13th biennial conference in Thessaloniki, Greece. The theme of the conference was “evaluation for more resilient societies” – which the organizers noted was particularly apt given the magnitude of political, economic, social and cultural turbulence experienced in Europe, and globally, over the last decade. In the face of climate change, migrant crises, and the rise of nationalism, the European evaluation community is considering its role in advancing a more just society.

Of particular interest to those working on international evaluation were several sessions – including a plenary – on the Organization for Economic Co-operation and Development’s (OECD) Criteria for Evaluating Development Assistance. As discussed in the August 2018 newsletter, a movement is underway to revisit and evolve the criteria – and this movement was palpable at the EES conference.  

The push for this is fairly straightforward: There’s recognition that the existing Criteria, though helpful, were created by a small number of powerful stakeholders and designed primarily to guide evaluation of projects or programs. Today, evaluation of complex interventions that take place within complex systems must be inclusive, culturally and contextually-sensitive, and complexity-aware. The conversations at EES were an important reminder that the next iteration of the DAC Criteria will need to be developed with input from a broad cross-section of actors, and will need to be responsive to new development paradigms if they are to remain relevant and influential in the future.

For more information on the conference or EES, visit https://www.europeanevaluation.org/.

And the 2018 AEA International Travel Awardees Are…

Each year, AEA provides a number of awards to evaluators living and working in developing countries to subsidize their participation in the association’s annual conference. This year, the International and Cross Cultural Evaluation (ICCE) Topical Interest Group (TIG) received over 70 applications for just eight awards. We’re proud to share that the awards were distributed equitably – to four female and four male awardees, representing seven countries: Benin, India, Kenya, Nepal, Peru, the Philippines and Tanzania.

Each application was reviewed by a minimum of three individuals. This year we were very fortunate to have 30 volunteer reviewers, and we’re starting to collect names for next year. If you are interested in supporting the 2019 travel award application review process, please reach out to Shawna Hoffman at SAHoffman@rockfound.org.

Finally, the money raised at the annual silent auction goes directly to support the international travel awards. If you will be in Cleveland at Evaluation 2018 this year, be sure to come out on Friday night to bid on some great books and other items. For more information about the silent auction, please contact Hubert Paulmer at hubertpaulmer@gmail.com.

 

AEA Town Hall Recordings Now Available

Access Past Meetings Online 

From the AEA Board of Directors

Did you miss the AEA Town Hall meetings? You can now access them on AEA’s website here. Be sure to watch the most recent conversation featuring Jean King and AEA Board member Susan Tucker discussing AEA’s Evaluator Competencies.

 

AEA Professional Development Corner 

On-Demand Resources Available, Plus Live eStudies Through October

From the AEA Education Team 

The Digital Knowledge Hub is an online platform featuring professional development opportunities for evaluators, by evaluators. View past eStudies, like this one: 

eStudy 091: Designing Useful Surveys

Surveys for program evaluation, performance measurement, or needs assessment can provide excellent information for evaluators. However, designing effective surveys requires an eye for both unbiased question design and the best methods for data administration. Michelle Kobayashi shares guidelines and methods for survey development that will increase response rates and create reliable and valid questionnaires.

Save the date for these upcoming live eStudy courses.


eStudy 094: Engaging the Whole System in Evidence Gathering, Advocacy and Action | October 23, 25, November 6, 8, 12-1:30 p.m. EDT

Presented by Kanti Gopal Kovvali, Author, Organizational Unlearning Specialist, Visiting Faculty TISS and NMIMS

The eStudy will expose participants to a radically different evaluation process that is rapid, transformational and sustainable. The eStudy will help participants to change their paradigms about evaluation, introduce new methodologies and tools and transfrom their orientation from doing good research to facilitating transformational research.

eStudy 095: Using the Cultural Consensus Method to Evaluate Program Impacts | November 27, 29, 12-1:30 p.m. EDT

Presented by Peggy Ochandarena, Chief of Party- Enhancing Palestinian Justice Program, Chemonics International Inc; Co-Director, Global Impact Collaboratory and Roseanne Schuster, Assistant Research Scientist, School of Human Evolution and Social Change, Arizona State University; Director of MEL, Global Impact Collaboratory.

Culture is the shared beliefs of a particular group of people, and strongly shapes what is socially acceptable and thus shapes action. Understanding the culture of an intervention's beneficiaries is critically important in designing interventions for effectiveness. In this webinar, leaders of the Global Impact Collaboratory, a partnership between Arizona State University and Chemonics International, give learners hands-on interaction with the theory, instrument design, and analysis for the CCM, with demonstration of its use in international development projects and application to a case study. 

Recent Stories
AEA Newsletter: October 2018

AEA Newsletter: September 2018

AEA Newsletter: August 2018