Return to search form  

Session Title: Cross-validation of Empirical Findings From Evaluations

Panel Session 856 to be held in Calhoun Room on Saturday, November 10, 3:30 PM to 5:00 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Chair(s):

Barbara payday loan  Brumbach payday loan ,  Northern Arizona University,  barb.brumbach@gmail.com

Abstract: Data analyses are frequently insufficiently guided by theory so that ultimate decisions and interpretations are subject to the phenomenon of "capitalizing on chance." That occurs because, inescapably, data are subject to error. Aside from theory, the primary check on the possibility of capitalizing on chance is cross-validation. Remarkably little, beyond statistical procedures, in the way of specification for the methodology of cross-validation has been written. Cross-validation is not so simple since even the criteria are for deciding whether a finding has been cross-validated have not been specified. The common method of cross-validation based on splitting a large sample into halves is useful only for narrow purposes. Better methods need to be developed. One possibility is the method we call "identifying quasi-populations." The problems involved in cross-validation will be discussed and illustrated and potential improvements in methods will be demonstrated in relation to cross-validation of parameter estimates and of models.

Rationale for Cross-validation Lee payday loan  Sechrest payday loan ,  University of Arizona,  sechrest@u.arizona.edu Cross-validation is an attempt to estimate (or demonstrate) the dependability of research findings by showing that they are replicable. There is no general agreement on just what aspects of a research effort are required to be dependable in order for cross-validation to be considered a success. Cross-validation depends on the assumption that errors are independent across sets of observations involved in the cross-validation. Theory and empirical data may be said to involve such independence and, hence, cross-validation if they are truly independently derived. In the absence of strong theory, conclusions drawn solely from a single set of observations are suspect. A frequent method of cross-validating is to split a large sample and show that findings from one half are replicated in the other half. That method of cross-validation has very limited usefulness. Replicability of findings in independent samples is by far the preferable method.

The Quasi-population Approach as a Tool for Cross-validation Mei-kuang payday loan  Chen payday loan ,  University of Arizona,  kuang@email.arizona.edu Replicabilitiy is the hallmark of science and the focus of cross-validation efforts, but replicability is not often undertaken in the social and behavioral sciences. Obstacles to replication are great. If one considers how replication is actually carried out when it does occur, one will find that many large data sets include incidental variables that permit partitioning of the data set into two or more subsamples that may be said to represent "quasi-populations," i.e., subsamples that emulate differences between samples that might be found if different investigators were undertaking the work. Socio-demographic variables, e.g., sex, age, place of residence, marital status, religious affiliation, may be used to partition the original sample. As long as the partitioning is not along lines that would be confounded with the variables of theoretical interest, two or more analyses could be considered to constitute a cross-validation design. Examples of such data sets are informative.

Cross-validation of Parameter Estimates Michele payday loan  Walsh payday loan ,  University of Arizona,  mwalsh@u.arizona.edu Probably the most common cross-validation effort involves assessing the dependability of parameter estimates. Parameters most commonly of interest with respect to cross-validation are means and measures of association (correlation). Cross-validation by split-sample methods can result in high, but misleading estimates of dependability because the random (error) component in one half of a random sample should be virtually the same as in the other half of the sample. Thus, what can be determined by split-sample cross-validation are simply the magnitudes of standard errors. That can easily be illustrated with a variety of data sets. An alternative, more demanding but more telling, approach to cross-validation is to try to define quasi-populations within the data set, i.e., subsamples that, in theory should not differ in ways related to the parameter(s) of interest and then determine the closeness of estimates obtained from the quasi-populations. Again, this process is easily demonstrated with actual data sets.

Cross-validation of Models Barbara payday loan  Brumbach payday loan ,  Northern Arizona University,  barb.brumbach@gmail.com Despite the widespread popularity of the idea of "confirmatory" analyses, most published accounts of multivariate models will be found to have been "modified" in some ways based on empirical data. That makes such models at least somewhat suspect of the possibility of "capitalizing on chance." Hence, even though it does not happen often, multivariate models should be cross-validated. It is not easy to specify just what would constitute cross-validation, e.g., the general structure of the overall model, the proportion of variance in the dependent variable that is accounted for, the robustness of specified paths, or specific path coefficients. When it is possible to do so, quasi-population approaches to cross-validation are of great interest. The use of such approaches is illustrated with large data sets.