|
Session Title: The Advantages of a Mixed Methods Approach
|
|
Panel Session 232 to be held in the Granite Room Section C on Thursday, Nov 6, 9:15 AM to 10:45 AM
|
|
Sponsored by the AEA Conference Committee
|
| Chair(s): |
| Carrie Markovitz,
Abt Associates Inc,
carrie_markovitz@abtassoc.com
|
| Abstract:
Mixed methods research designs provide evaluators with powerful ways of aligning real-world data needs with useful evaluation techniques that can yield valuable results for clients. This panel presentation will illustrate the mixed methods design approaches employed by evaluators on three very different evaluation studies. Each presentation will not only present different mixed methods designs, but also explore how the use of dual methods allowed the evaluators to develop, identify, and/or measure unique outcomes and perspectives.
|
|
First Identifying Then Investigating Indigenous Program Impacts Using Mixed Methods
|
| Mary McNabb,
Learning Gauge Inc,
mlmcnabb@msn.com
|
|
Dr. Mary McNabb, EdD, will discuss the design of a five-year evaluation study in education which began with an Exploratory Design and ended with an Explanatory Design. The evaluation goal was to discover the variables pertinent to successful implementation of a professional development program. It began with QUAL data collection involving interviews with project staff about those who had participated in their professional development training. Telephone interviews with key informants resulted in the generation of themes and codes leading to the discovery of program variables appropriate for a taxonomy of program impacts. Dr. McNabb then used these variables to design a program impact survey. Later in the evaluation, she used the taxonomy to code and to quantify QUAL answers from open-ended survey questions. At this point in the evaluation, the Explanatory Design was employed to shed light on the survey's other QUAN results. In her presentation, Dr. McNabb will reflection upon the benefits of mixed methods designs in her work with educational programs. She will provide specific examples of how the distinctions between the two designs helped her clarify the purpose, appropriate timing, and weighting of various types of data to answer different types of evaluation questions during the phases of a long-term evaluation.
|
|
|
Improving Lives and Communities: Perspectives on 40 Years of VISTA Service
|
| Carrie Markovitz,
Abt Associates Inc,
carrie_markovitz@abtassoc.com
|
|
Dr. Carrie Markovitz will present the findings from a study on the experiences of participants in the AmeriCorps*VISTA program starting with the program's inception in 1965 until 1993. In addition to participants' experiences, the study focused particular attention on the long-term effects of VISTA on members' civic engagement, education, employment, and the intergenerational transfer of values. Outcomes were assessed using a mixed methods approach: 1) a telephone survey designed to provide information on the breadth of VISTA's effects; and 2) a series of in-depth personal interviews providing highly detailed insights into the experiences of a much smaller sample. Both data collection components were structured to gather feedback from three distinct generations of members defined by major program and policy shifts that have shaped the evolution of VISTA over its 40-year history. To provide a point of reference, the experiences and outcomes of VISTA members were compared to a comparison group of similar individuals who applied for VISTA, completed a portion of the training, but ultimately did not serve. In particular, Dr. Carrie Markovitz will discuss the advantages of using a mixed methods approach for answering the key research questions of the study.
| |
|
Giving Voice to the Client Perspective in Data Collection and Data Analysis Using Mixed Methods
|
| Carrie Petrucci,
Evaluation, Management and Training Associates Inc,
cpetrucci@emt.org
|
|
Dr. Petrucci will discuss how mixed methods were used in data collection and data analysis to incorporate the client perspective within a survey design. A nominal group technique was used in a focus group to identify what residents had learned about HIV transmission and how their behaviors had changed. A qualitative approach was used to brainstorm ideas with clients, followed by a quantitative ranking of all of the gathered responses. The analysis proceeded with an emerging themes latent content analysis by the researcher. These responses resulted in five main themes. It was suggested that from a client perspective, the immediate areas of concern were how to deal with their health situations in the present time, and learning how to deal with their own emotions connected to HIV/AIDS, particularly their fears. Secondary were the basic behavioral changes that must occur to maintain their health. The rankings showed how clients prioritized each of the themes relative to one another, suggesting practice and research applications. These qualitative findings will be discussed within the context of HIV risk behaviors from a longitudinal survey given to the same clients. An additional advantage to focus group interviews is that they can be more amenable to working in multicultural settings, due to the depth of knowledge that can be gathered, the voice that is given to clients, and the trust that can be achieved.
| |