| Session Title: Using Rigorous Qualitative Data Analysis to Enhance Random Assignment Designs: Lessons From an Evaluation of a Teacher Professional Development Program |
| Multipaper Session 251 to be held in Panzacola Section H1 on Thursday, Nov 12, 10:55 AM to 12:25 PM |
| Sponsored by the Qualitative Methods TIG |
| Chair(s): |
| Savitha Moorthy, Berkeley Policy Associates, savitha@bpacal.com |
| Abstract: Qualitative data analysis can provide an invaluable complement to experimental evaluations of educational programs. This presentation responds to the call for the public disclosure of methods in naturalistic inquiry and addresses the need for greater transparency and clarity, especially in the qualitative analysis process. We draw from our experiences as a research team working on a longitudinal random assignment study of teacher professional development, to summarize the iterative steps of our analysis approach and outline the strategies we employ to articulate the relationship between qualitative data collection, analysis, and research findings. Specific strategies to be discussed include team-based exploratory coding and the use of computer-assisted qualitative data analysis. |
| Policy Relevance of Increased Rigor in Qualitative Methods: The Case of Experimental Evaluations in Education |
| Hannah Betesh, Berkeley Policy Associates, hannah@bpacal.com |
| In an era of educational research that emphasizes empirical knowledge and evidence, it has become important to incorporate qualitative analyses into studies of what works in education. Within the policy framework of No Child Left Behind, this paper will identify important facets of rigorous qualitative analysis and explore the potential of such work, specifically naturalistic inquiry, for addressing policy relevant issues that cannot be fully explored through quantitative work alone, such as program and policy implementation; characterizing change in organizational cultures; and understanding values, preferences, and perspectives, particularly of vulnerable or marginalized populations. |
| Team-Based Qualitative Analysis: Lessons From School-Based Data Collection |
| Jacklyn Altuna, Berkeley Policy Associates, jacklyn@bpacal.com |
| Through the lens of a multi-year evaluation of a teacher professional development program geared towards improving the education of English Learners, we examine our team-based analysis approach in our qualitative research process as one piece that contributes to increased transparency in qualitative research. As one component of a larger study primarily focused on quantitative impacts, the systematic implementation of rigor in qualitative analyses has become increasingly crucial. Through the team-based analysis of field notes from direct observations of after-school lesson design meetings, transcribed interviews from the meeting facilitators, written records provided by program coaches, and transcribed interviews from interviews with program coaches, we provide a detailed account of how our process unfolded in the co-construction of codebooks, coming to consensus about the salience of various codes, restructuring of coding hierarchies, and establishing interrater reliability among multiple coders. |
| Computer-Assisted Qualitative Data Analysis: Applications to Educational Research |
| Savitha Moorthy, Berkeley Policy Associates, savitha@bpacal.com |
| For some, the use qualitative data analysis software automatically signals increased rigor in the analytic process, while for others, it raises concerns that the dominance of queries and auto-retrieve functions mechanizes analysis, and discourages researchers from pursuing other ways of scrutinizing the data. In our team-based coding approach, we examine the use of software as an invaluable and time-saving analysis tool that in combination with careful inquiry, can contribute to sustaining increased transparency in the qualitative analysis process. We provide an in-depth look into our coding processes using data from an implementation study of a teacher professional development program. Using field notes from direct observations, transcribed interviews with program coaches, and written records provided by the program, we explore the various applications of Nvivo8 in our data sets and how it can be utilized to improve rigor in qualitative research. |