Choosing Between Two Classification Learning Algorithms Based on Calibrated Balanced $5\times 2$ Cross-Validated F-Test

Abstract

$5\times 2$ cross-validated F-test based on independent five replications of 2-fold cross-validation is recommended in choosing between two classification learning algorithms. However, the reusing of the same data in a $5\times 2$ cross-validation causes the real degree of freedom (DOF) of the test to be lower than the F(10, 5) distribution given by (Neural Comput 11:1885–1892, [1]). This easily leads the test to suffer from high type I and type II errors. Random partitions for $5\times 2$ cross-validation result in difficulty in analyzing the DOF for the test. In particular, Wang et al. (Neural Comput 26(1):208–235, [2]) proposed a new blocked $3 \times 2$ cross-validation, that considered the correlation between any two 2-fold cross-validations. Based on this, a calibrated balanced $5\times 2$ cross-validated F-test following F(7, 5) distribution is put forward in this study by calibrating the DOF for the F(10, 5) distribution. Simulated and real data studies demonstrate that the calibrated balanced $5\times 2$ cross-validated F-test has lower type I and type II errors than the $5\times 2$ cross-validated F-test following F(10, 5) in most cases.

Appendix

Proof of proposition

Denoting $U=(\hat{\mu }_{B_1}^{(1)}, \hat{\mu }_{B_2}^{(1)}, \hat{\mu }_{B_1}^{(2)}, \hat{\mu }_{B_2}^{(2)}, \ldots , \hat{\mu }_{B_1}^{(5)}, \hat{\mu }_{B_2}^{(5)})^{T}$, we have $U\sim N(0, \sigma ^{2}\Sigma )$ from the assumption of Proposition, where

$$\begin{aligned} \Sigma = \left( \begin{array}{cccccc} 1 &{} \rho _{1} &{}\rho _{2} &{} \cdots &{}\rho _{2} &{}\rho _{2}\\ \rho _{1} &{}1 &{}\rho _{2} &{}\cdots &{}\rho _{2} &{}\rho _{2}\\ \vdots &{} \vdots &{} \ddots &{} \vdots &{} \vdots &{} \vdots \\ \rho _{2} &{}\rho _{2} &{}\rho _{2} &{} \cdots &{}1 &{}\rho _{1}\\ \rho _{2} &{} \rho _{2} &{}\rho _{2} &{} \cdots &{}\rho _{1} &{}1 \\ \end{array}\right) _{10\times 10} \end{aligned}$$

The eigenvalues of $\Sigma$ are obtained easily from $|\lambda I-\Sigma |=0$: $\lambda _{1}=1-\rho _{1}$ with multiplicity 5, $\lambda _{6}=-2\rho _{2}+\rho _{1}+1$ with multiplicity 4, and $\lambda _{10}=\rho _{1}+8\rho _{2}+1$. Thus, we can conclude that real symmetric matrix $\Sigma$ represents a positive definite matrix when $0\le \rho _{1}\le \rho _{2}<0.50$. We can also conclude that $\Sigma ^{\frac{1}{2}}$ is a positive definite matrix from the decomposition $\Sigma =\Sigma ^{\frac{1}{2}} \Sigma ^{\frac{1}{2}}$.

Let $U_{1}=U/\sigma , Z=\Sigma ^{-\frac{1}{2}}U_{1}$, then we have  $U_{1}\sim N(0,\Sigma ), Z\sim N(0, I_{10})$ and obviously $U_{1}^{T}U_{1}=Z^{T}\Sigma Z$.

An orthogonal matrix exists for each n order real symmetric matrix such that the matrix can be diagonalized. Thus, an orthogonal matrix T exists such that $T\Sigma T^{T}=\Lambda$, i.e., $\Sigma =T^{T}\Lambda T,$ where $\Lambda$ is a diagonal matrix, and its element is the eigenvalue of $\Sigma$.

We know that $TZ\sim N(0, I_{10})$ from the properties of the orthogonal matrix, then $U_{1}^{T}U_{1}=Z^{T}\Sigma Z=Z^{T}T^{T} \Lambda TZ=\sum _{i=1}^{10}\lambda _{i}\eta _{i}^{2}$. Thus, $U_{1}^{T}U_{1}$ approximately follows an  $C \chi ^{2}(f)$ distribution because $\sum _{i=1}^{10}\lambda _{i}\eta _{i}^{2}$ approximately follows  $C \chi ^{2}(f)$ distribution, where $\lambda _{i}$ denotes the eigenvalue of $\Sigma$, $\eta _{i}$ is the i-th element of matrix TZ, and

$$\begin{aligned} C=\frac{\sum _{i=1}^{10}\lambda _{i}^{2}}{\sum _{i=1}^{10}\lambda _{i}} =1+\rho _{1}^{2}+8\rho _{2}^{2}, f=\frac{(\sum _{i=1}^{10}\lambda _{i})^{2}}{\sum _{i=1}^{10}\lambda _{i}^{2}}=\frac{10}{1+\rho _{1}^{2}+8\rho _{2}^{2}} \end{aligned}$$

(see [18, 19]).

Note $\sum _{i=1}^{10}\lambda _{i}=10, \sum _{i=1}^{10}\lambda _{i}^{2}=10(1+\rho _{1}^{2}+8\rho _{2}^{2}), fC=\sum _{i=1}^{10}\lambda _{i}=10.$

Moreover, we have $Var(\hat{\mu }_{B_1}^{(i)}-\hat{\mu }_{B_2}^{(i)})=2(1-\rho _{1})\sigma ^{2}, Cov(\hat{\mu }_{B_1}^{(i)}-\hat{\mu }_{B_2}^{(i)},\hat{\mu }_{B_1}^{(i')}-\hat{\mu }_{B_2}^{(i')}) = (\rho _{2}-\rho _{2}-\rho _{2}+\rho _{2})\sigma ^{2}=0$ for $i\ne i'$. If letting $U_{2}=((\hat{\mu }_{B_1}^{(1)}-\hat{\mu }_{B_2}^{(1)})/\sigma , \ldots , (\hat{\mu }_{B_1}^{(5)}-\hat{\mu }_{B_2}^{(5)})/\sigma )$, then $U_{2} \sim N(0,\Sigma _{2}),$ and $U_{2}^{T}U_{2} \sim 2(1-\rho _{1}) \chi ^{2}(5)$, where

$$\begin{aligned} \Sigma _{2}= \left( \begin{array}{cccc} 2(1-\rho _{1}) &{}0 &{} \cdots &{}0\\ 0 &{}2(1- \rho _{1}) &{}\cdots &{}0\\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ 0 &{}0 &{}2(1- \rho _{1}) &{} \cdots \\ 0 &{} 0 &{} \cdots &{}2(1-\rho _{1}) \\ \end{array}\right) _{5\times 5} \end{aligned}$$

These result in

$$\begin{aligned} \frac{\sum _{i=1}^{5}\sum _{k=1}^{2}(\hat{\mu }_{B_k}^{(i)})^{2}/C / f}{\sum _{i=1}^{5}(\hat{\mu }_{B_1}^{(i)}-\hat{\mu }_{B_2}^{(i)})^{2}/(2(1-\rho _{1}))/ 5} =\frac{U_{1}^{T}U_{1}}{U_{2}^{T}U_{2}} \frac{10(1-\rho _{1})}{fC} =(1-\rho _{1})\frac{U_{1}^{T}U_{1}}{U_{2}^{T}U_{2}}\sim F(f,5) . \end{aligned}$$

We know that

$$\begin{aligned} {S_B}_{i}^2= & {} \sum _{k=1}^{2}(\hat{\mu }_{B_k}^{(i)}-\hat{\mu }_B^{(i)})^{2} =(\hat{\mu }_{B_1}^{(i)}-(\frac{\hat{\mu }_{B_1}^{(i)}+\hat{\mu }_{B_2}^{(i)}}{2}))^{2} +(\hat{\mu }_{B_2}^{(i)}-(\frac{\hat{\mu }_{B_1}^{(i)}+\hat{\mu }_{B_2}^{(i)}}{2}))^{2}\\= & {} \frac{(\hat{\mu }_{B_2}^{(i)}-\hat{\mu }_{B_1}^{(i)})^{2}}{2}, \end{aligned}$$

where $\hat{\mu }_B^{(i)}=\frac{\hat{\mu }_{B_1}^{(i)}+\hat{\mu }_{B_2}^{(i)}}{2}$.

Therefore,

$$\begin{aligned} F=(1-\rho _{1})\frac{\sum _{i=1}^{5}\sum _{k=1}^{2}(\hat{\mu }_{B_k}^{(i)})^{2}}{\sum _{i=1}^{5}(\hat{\mu }_{B_1}^{(i)}-\hat{\mu }_{B_2}^{(i)})^{2}} =\frac{1-\rho _{1}}{2}\frac{\sum _{i=1}^{5}\sum _{k=1}^{2} (\hat{\mu }_{B_k}^{(i)})^{2}}{\sum _{i=1}^{5}{S_B}_{i}^{2}}\sim F(f,5), \end{aligned}$$

where $f=\frac{10}{1+\rho _{1}^{2}+8\rho _{2}^{2}}.$ $\square$