Draper, David and Krnjajic, Milovan
12/31/2007 09:00 AM
Applied Mathematics & Statistics
A standard (data-analytic) approach to statistical model specification, practiced with equal vigor in both Bayesian and non-Bayesian approaches to model-building, involves the initial choice, for the structure of the model, of one or another of a variety of standard parametric families, followed by modification of this initial choice---once data begin to arrive---if the data suggest deficiencies in the original specification. In this paper (a) we argue that this approach is formally incoherent, because it amounts to using the data both to specify the prior distribution on structure space and to update using this data-determined prior; (b) we identify two approaches to avoiding (at least in principle, and with a fair amount of data) the incoherence in (a): (1) Bayesian semi-parametric modeling and (2) three-way out-of-sample predictive validation; (c) we provide details on implementing (2); (d) we argue that to make progress in coherent Bayesian model specification in complicated problems You (the modeler) have to either implicitly or explicitly choose a utility structure which defines, for You, when the model currently being examined is ``good enough"; (e) we argue that it is best to make this choice explicitly on the basis of real-world considerations regarding the use to which the model will be put; and (f) we contrast model selection methods based on the log score and deviance information criteria (DIC) as two examples of (e) with utilities governed by predictive accuracy.
