| Session Title: Generalizability Theory in Analysis of Survey Data |
| Multipaper Session 305 to be held in Centennial Section B on Thursday, Nov 6, 1:40 PM to 3:10 PM |
| Sponsored by the Quantitative Methods: Theory and Design TIG |
| Chair(s): |
| Lee Sechrest, University of Arizona, sechrest@u.arizona.edu |
| Discussant(s): |
| Patrick McKnight, George Mason University, pem725@gmail.com |
| Abstract: Generalizability theory, although not yet much used in evaluation, has considerable potential utility. Its use is illustrated by analysis of survey data, for which generalizability analyses can provide insights to guide further analyses directed at answering substantive questions. Generalizability analyses can also help analysts to make decisions about approaches that will yield dependable results and help in planning future surveys. Generalizability analysis is an analysis of variance that permits allocation of variance in responses to different sources. A large survey of Americans concerning making of medical decisions provides the material for four papers. One focuses on generalizability of responses across survey items and the implications of the findings. A second focuses on generalizaiblity across respondents. A third paper will show how generalizability may guide the development of specific further analyses, and a fourth paper will show how the results of generalizability analyses can inform the development of further surveys. |
| Generalizability Theory and Person Variance |
| Lee Sechrest, University of Arizona, sechrest@u.arizona.edu |
| Generalizability theory may also be used to study the variance attributable to persons responding to various tasks such as surveys. Variance attributable to persons means that the means of individual persons across a set of items are not generalizable to a larger set. At some level, researchers will almost always want the proportion of variance attributable to persons to shrink to zero, i.e., it will be most useful and meaningful to know that certain subsets of persons are very much alike in their responses. If variance is attributable to persons, further analyses should show what characteristics of persons are associated with differences among them. Main effects of persons may be interesting in their own right, but interactions between persons and other design factors are also usually of interest, as is shown by responses to the national survey. |
| Generalizabilty D Study in Planning Productive Analyses |
| Mei-Kuang Chen, University of Arizona, kuang@u.arizona.edu |
| Patrick McKnight, George Mason University, pem725@gmail.com |
| Lee Sechrest, University of Arizona, sechrest@u.arizona.edu |
| In Generalizability theory, D study refers to the determination of the extent to which results for one observation or some subset of observations is generalizable to other similar observations. The determination then makes it possible to estimate how many such observations would be required in order to achieve a predetermined level of reliability. Data from large scale studies, e.g., surveys, can often be broken down in many different ways for the purpose of conducting sub analyses. Such analyses may be disappointing, however, and even misleading, if the number of observations in the subset is not sufficient to produce dependable estimates. Illustrative uses of D studies to estimate required number of observations for items, topics, and persons for the national data set on medical decision-making are instructive. Such analyses can show, among other things, whether optimal use of the data available may make replications possible. |
| Generalizability Theory D Studies in Planning for Future Research |
| Sally Olderbak, University of Arizona, sallyo@email.arizona.edu |
| Patrick McKnight, George Mason University, pem725@gmail.com |
| Lee Sechrest, University of Arizona, sechrest@u.arizona.edu |
| Generalizability D studies can be used in the planning of future research in order to maximize statistical power and efficiency achieved by minimizing data to be collected. D studies can show how many respondents are needed to achieve a specified level of reliability of the data, which could make it possible to minimize response burden by not requiring that every item be asked of every respondent. D studies can also show how many items might be required, or how many topics might need to be addressed in order to get dependable results. Data from the National Survey of Medical Decision Making can be used to determine in any follow-on surveys the specific characteristics of the survey design that would be required in order to obtain dependable data. Follow-on surveys, if required, can be conducted efficiently so as to conserve resources of all kinds. |