On the asymptotic distribution of Pearson’s X ² in cross-validation samples

Abstract

In categorical data analysis, two-sample cross-validation is used not only for model selection but also to obtain a realistic impression of the overall predictive effectiveness of the model. The latter is of particular importance in the case of highly parametrized models capable of capturing every idiosyncracy of the calibrating sample. We show that for maximum likelihood estimators or other asymptotically efficient estimators Pearson’s X ² is not asymptotically chi-square in the two-sample cross-validation framework due to extra variability induced by using different samples for estimation and goodness-of-fit testing. We propose an alternative test statistic, X _xval ², obtained as a modification of X ² which is asymptotically chi-square with C - 1 degrees of freedom in cross-validation samples. Stochastically, X _xval ² ≤ X ². Furthermore, the use of X ² instead of X _xval ² with a χ _C - 1 ² reference distribution may provide an unduly poor impression of fit of the model in the cross-validation sample.