On Some Test Statistics for Testing the Regression Coefficients in Presence of Multicollinearity: A Simulation Study

Abstract

Ridge regression is a popular method to solve the multicollinearity problem for both linear and non-linear regression models. This paper studied forty different ridge regression t-type tests of the individual coefficients of a linear regression model. A simulation study was conducted to evaluate the performance of the proposed tests with respect to their empirical sizes and powers under different settings. Our simulation results demonstrated that many of the proposed tests have type I error rates close to the 5% nominal level and, among those, all tests except one have considerable gain in powers over the standard ordinary least squares (OLS) t-type test. It was observed from our simulation results that seven tests based on some ridge estimators performed better than the rest in terms of achieving higher power gains while maintaining a 5% nominal size.

1. Introduction

Multicollinearity is the occurrence of high inter-correlations among independent variables in a multiple regression model. When this condition is present, it can result in unstable and unreliable regression coefficient estimates if the method of ordinary least squares is used. One of the proposed solutions to the problem of multicollinearity is the concept of ridge regression as pioneered by Hoerl and Kennard [1] They found that there is a nonzero value of k (ridge or shrinkage parameter) for which mean square error (MSE) for the ridge regression estimator is smaller than the variance of the ordinary least squares (OLS) estimator.

Estimating the shrinkage parameter (k) is a vital issue in the ridge regression model. Several researchers at different period of times have worked in this area of research and proposed different estimators for k. To mention a few, Hoerl and Kennard [1], Hoerl, Kennard and Baldwin [2], Lawless and Wang [3], Gibbons [4], Nomura [5], Kibria [6], Khalaf [7], Khalaf and Shukur [8], Alkhamisi and Shukur [9], Muniz and Kibria [10], Feras and Gore [11], Gruber [12], Muniz et al. [13], Mansson et al. [14], Hefnawy and Farag [15], Roozbeh and Arashi [16], Arashi and Valizadeh [17], Aslam [18], Asar and Karaibrahimoğlu [19], Saleh et al. [20], Asar and Erişoğlu [21], Goktas and Sevinc [22], Fallah et al. [23], Norouzirad and Arashi [24], and very recently Saleh et al. [25], among others

It is well known that, to make inference about an unknown population parameter, one may consider both confidence interval and hypothesis testing methods. However, the literature on the test statistics for testing the regression coefficients under the ridge regression model is very limited. First, Halawa and Bassiouni [26] proposed non exact t-tests for the regression coefficients under ridge regression estimation and compared empirical sizes and powers of only two tests based on the estimator of k proposed by Hoerl and Kennard [1] and Hoerl, Kennard, and Baldwin [2]. Their results evidenced that, for models with large standard errors, the ridge based t-tests have correct sizes with considerable gain in powers over those of the least squares t-test. For models with small standard errors, tests are found to be slightly exceeding the nominal level in few cases. Cule et al. [27] evaluated the performance of tests proposed by Hoerl and Kennard [1], Hoerl, Kennard, and Baldwin [2], and Lawless and Wang [3] based on linear ridge and logistic ridge regression models. Gokpinar and Ebegil [28] evaluated the performance of the t-tests based on 22 different estimators of the ridge parameter k collected from the published literature. Finally, Kibria and Banik [29] analyzed the performance of the t-tests based on 16 popular estimators of the ridge parameter.

Since different ridge regression estimators are considered by several researchers at different times and under different simulation methods and conditions, testing regression coefficients based on the basis of size (Type I error) and power properties under the ridge regression model are not comparable as a whole. Therefore, the important contribution of this paper is to make a more comprehensive comparison of a much larger ensemble of available t test statistics for testing regression coefficients. We consider in our analysis most of the ones analyzed in Gokpinar and Ebegil [28] and Kibria and Banik [29] as well as other test statistics based on other ridge estimators not included in the aforementioned studies at the same time. In total, our paper compares forty different t-tests statistics. The test statistics were compared based on the empirical type I error and the power properties following the testing procedures that are detailed in Halawa and Bassiouni [26]. These results are of interest for statistical practitioners using ridge regression in different fields of application as a guide to which test statistics to use when testing the significance of variables in their ridge regression models.

This paper is organized as follows. The proposed test statistics for the linear regression model are described in Section 2. To compare the performance of the test statistics, a simulation study is conducted in Section 3. An application is discussed in Section 4. Finally, some concluding remarks are given in Section 5.

2. Test Statistics for Regression Coefficients

Let us consider the following multiple linear regression model:

$$Y=X\beta +\epsilon , \epsilon ~ N(0, {\sigma }^2{I}_n) , rank(X_{n\times q})=q \le n$$

(1)

Y is an (n × 1) dimensional vector of dependent variables centered about their mean, X is an (n × q) dimensional observed matrix of the regressors centered and scaled such that $X^{T}X$ is in correlation form, β is (q × 1) dimensional unknown coefficient vector, and ε is (n × 1) error vector distributed as multivariate normal with mean 0 and variance–covariance matrix ${\sigma }^2{I}_n$, where I_n is an (n × n) identity matrix.

The ordinary least square estimator (OLS) of the parameter vector β is:

$$\widehat{\beta }={(X^{T}X)}^{-1}{X}^{T}Y$$

(2)

To test whether the i-th component of the parameter vector β is equal to zero, the following test is used based on the OLS estimator:

$$\begin{gathered} H_0: {\beta }_i=0 versus H_1: {\beta }_i\ne 0 \\ t=\frac{\widehat{{\beta }_i}}{S(\widehat{{\beta }_i})} \end{gathered}$$

(3)

where $\widehat{{\beta }_i}$ is the ith component of $\widehat{\beta }$, and $S(\widehat{{\beta }_i})$ is the square root of the ith diagonal element of $Var (\widehat{\beta })= \widehat{{\sigma }^2}{(X^{T}X)}^{-1}$ with:

$$\widehat{{\sigma }^2}={(Y-X\widehat{\beta })}^T(Y-X\widehat{\beta })/(n-q-1)$$

The test statistic in Equation (3) is the least square test statistic. Under the null hypothesis, it is distributed as Student t- distribution with n−q−1 degrees of freedom. However, when $X^{T}X$ is ill conditioned due to multicollinearity, the least square estimator in (2) produces unstable estimators with unduly large sampling variance. Adding a constant k to the diagonal elements of $X^{T}X$ improves the ill conditioned situation. This is called ridge regression. The ridge estimator of the parameter vector β is then:

$${\widehat{\beta }}_{(k)}={(X^{T}X+k{I}_n)}^{-1}{X}^{T}Y$$

(4)

where k > 0 is the ridge or shrinkage parameter.

The bias, the variance matrix, and the MSE expression of ${\widehat{\beta }}_{(k)}$ are respectively given as follows:

$$\begin{gathered} Bias=E({\widehat{\beta }}_{(k)})-\beta =-k{(X^{T}X+k{I}_p)}^{-1}\beta \\ Var (\widehat{{\beta }_{(k)}})= {\sigma }^2{(X^{T}X+k{I}_n)}^{-1}{X}^{T}X{(X^{T}X+k{I}_n)}^{-1} \\ MSE (\widehat{{\beta }_{(k)}})= {\sigma }^2{(X^{T}X+k{I}_n)}^{-1}{X}^{T}X{(X^{T}X+k{I}_n)}^{-1}+k^2\beta {(X^{T}X+k{I}_p)}^{-2}\beta \end{gathered}$$

(5)

and ${\sigma }^2$ is estimated as follows:

$$\widehat{{\sigma }_k{}^2}=\frac{{(Y-X{\widehat{\beta }}_{(k)})}^T(Y-X{\widehat{\beta }}_{(k)})}{n-q-1}.$$

(6)

To test whether the i-th component of the parameter vector β is equal to zero, Halawa and Bassouni [26] proposed the following t-test statistic based on the ridge estimator of the parameter vector:

$$t_k=\frac{{\widehat{\beta }}_{i(k)}}{S({\widehat{\beta }}_{i(k)})}$$

(7)

where ${\widehat{\beta }}_{i(k)}$ is the ith element of ${\widehat{\beta }}_{(k)}$, and $S({\widehat{\beta }}_{i(k)})$ is the square root of the ith diagonal element of $Var (\widehat{{\beta }_{(k)}}).$

Under the null hypothesis, the test statistic (7) was shown to be approximately distributed as a Student t-distribution with $n-q-1$ degrees of freedom. For more details on this topic, see Halawa and Bassiouni [26], among others.

Values of the Ridge Estimator k Considered for the Test Statistic t_k

Since the ridge parameter k is unknown, it needs to be estimated from observed data. This section gives the formulas for the forty different ridge regression estimators considered in our simulation study for the test statistic defined in (7).

Table 1. Some ridge regression estimators.

Authors	Ridge Estimator Formula (k)
Hocking, Speed and Lynn [30]	$k_{HSL}=\widehat{{\sigma }^2}{\displaystyle {\displaystyle \sum }_{j=1}^q}{\widehat{\alpha }}_j^2{\lambda }_j^2/{({\displaystyle {\displaystyle \sum }_{j=1}^q}{\widehat{\alpha }}_j^2{\lambda }_j)}^2$
Hoerl and Kennard [1]	$k_{HK70}=\raisebox{1ex}{$\widehat{{\sigma }^2}$}\!\left/ \!\raisebox{-1ex}{$\max ({\widehat{\alpha }}^2)$}\right.$
Thisted [31]	$k_{TH}=(q-2)\widehat{{\sigma }^2}/{\displaystyle {\displaystyle \sum }_{j=1}^q}{\widehat{\beta }}_j^2$
Vennables and Rippley [32]	$k_{VR}=n(q-2)\widehat{{\sigma }^2}/({\widehat{\beta }}^T{X}^{T}X\widehat{\beta })$
Lawless and Wang [3]	$k_{LW}=q\widehat{{\sigma }^2}/{\displaystyle {\displaystyle \sum }_{j=1}^q}{\widehat{\alpha }}_j^2{\lambda }_j$
Hoerl, Kennard and Baldwin [2]	$k_{HKB}=q\widehat{{\sigma }^2}/{\displaystyle {\displaystyle \sum }_{j=1}^q}{\widehat{\alpha }}_j^2$
Kibria [6]	$k_{KibAM}=Arithmetic Mean (\widehat{{\sigma }^2}/{\widehat{\alpha }}_j^2)$
Kibria [6]	$k_{KibGM}=Geometric Mean (\widehat{{\sigma }^2}/{\widehat{\alpha }}_j^2)$
Kibria [6]	$k_{KibMED}=Median(\widehat{{\sigma }^2}/{\widehat{\alpha }}_j^2)$
Muniz and Kibria [10]	$k_{M2}=\max (1/{(\widehat{{\sigma }^2}/{\widehat{\alpha }}_j^2)}^{\frac{1}{2}})$
Muniz and Kibria [10]	$k_{M3}=\max ({(\widehat{{\sigma }^2}/{\widehat{\alpha }}_j^2)}^{\frac{1}{2}})$
Muniz and Kibria [10]	$k_{M4}=Geometric Mean(1/{(\widehat{{\sigma }^2}/{\widehat{\alpha }}_j^2)}^{\frac{1}{2}})$
Muniz and Kibria [10]	$k_{M5}=Geometric Mean({(\widehat{{\sigma }^2}/{\widehat{\alpha }}_j^2)}^{\frac{1}{2}})$
Muniz and Kibria [10]	$k_{M6}=Median(1/{(\widehat{{\sigma }^2}/{\widehat{\alpha }}_j^2)}^{\frac{1}{2}})$
Muniz et al. [13]	$k_{M8}=\max (1/g_j), where g_j=({\lambda }_{max}\widehat{{\sigma }^2)}/((n-q)\widehat{{\sigma }^2}+{\lambda }_{max}{\widehat{\alpha }}_j^2)$
Muniz et al. [13]	$k_{M9}=\max (g_j)$
Muniz et al. [13]	$k_{M10}=Geometric Mean(1/g_j)$
Muniz et al. [13]	$k_{M11}=Geometric Mean(g_j)$
Muniz et al. [13]	$k_{M12}=Median(1/g_j)$
Dorugade and Kashid [33]	$k_D=\max (0, k_{HKB}-1/(n\ast VI{F}_{max})$
Khalaf and Shukur [8]	$k_{KS}=({\lambda }_{max}\widehat{{\sigma }^2})/((n-q)\widehat{{\sigma }^2}+{\lambda }_{max}{\widehat{\alpha }}_{max}^2)$
Khalaf and Shukur [8]	$k_{K12}=k_{HK70}+2/({\lambda }_{max}+{\lambda }_{min})$
Alkhamisi and Shukur [9]	$k_{KSAM}=Arithmetic Mean(m_j)$ $m_j= ({\lambda }_j\widehat{{\sigma }^2})/((n-q)\widehat{{\sigma }_j^2}+{\lambda }_j{\widehat{\alpha }}_j^2)$
Alkhamisi and Shukur [9]	$k_{KSMAX}=\max (m_j)$
Alkhamisi and Shukur [9]	$k_{KSMED}=Median(m_j)$
Alkhamisi and Shukur [9]	$k_{ASH}=\max (\widehat{{\sigma }^2}/{\widehat{\alpha }}_j^2+1/{\lambda }_j)$
Schaffer et al. [34]	$k_{SC}=1/\max ({\widehat{\alpha }}^2$)
Asar et al. [21]	$k_{A1}=(q^2\widehat{{\sigma }^2})/({\lambda }_{max}^2{\displaystyle {\displaystyle \sum }_{j=1}^q}{\widehat{\alpha }}_j^2)$
Asar et al. [21]	$k_{A2}=(q^3\widehat{{\sigma }^2})/({\lambda }_{max}^3{\displaystyle {\displaystyle \sum }_{j=1}^q}{\widehat{\alpha }}_j^2)$
Asar et al. [21]	$k_{A3}=(q\widehat{{\sigma }^2})/({\lambda }_{max}^{1/3}{\displaystyle {\displaystyle \sum }_{j=1}^q}{\widehat{\alpha }}_j^2)$
Asar et al. [21]	$k_{A4}=(q^2\widehat{{\sigma }^2})/\{{({\displaystyle {\displaystyle \sum }_{j=1}^q}\sqrt{{\lambda }_j})}^{\frac{1}{3}}{\displaystyle {\displaystyle \sum }_{j=1}^q}{\widehat{\alpha }}_j^2\}$
Asar et al. [21]	$k_{A5}=(2q\widehat{{\sigma }^2})/({\lambda }_{max}^{1/2}{\displaystyle {\displaystyle \sum }_{j=1}^q}{\widehat{\alpha }}_j^2)$
Nomura [22]	$k_{NOM}=q\widehat{{\sigma }^2}/{\displaystyle {\displaystyle \sum }_{j=1}^q}\left[{\widehat{\alpha }}_j^2/\left\{1+(1+{\lambda }_{j }{(({\widehat{\alpha }}_j^2/\widehat{{\sigma }^2}))}^{\frac{1}{2}})\right\}\right]$
Goktas and Sevinc [22]	$k_{GS1}=\sqrt{Median({(\widehat{{\sigma }^2}/{\widehat{\alpha }}_j^2)}^{\frac{1}{2}})}$
Goktas and Sevinc [22]	$k_{GS2}=\widehat{{\sigma }^2}/{(Median({(\widehat{{\sigma }^2}/{\widehat{\alpha }}_j^2)}^{\frac{1}{2}}))}^2$
Dorugade [35]	$k_{D1}=ArithmeticMean((2\widehat{{\sigma }^2})/({\lambda }_{max}{\widehat{\alpha }}_j^2))$
Dorugade [35]	$k_{D2}=Median ((2\widehat{{\sigma }^2})/({\lambda }_{max}{\widehat{\alpha }}_j^2))$
Dorugade [35]	$k_{D3}=Harmonic Mean((2\widehat{{\sigma }^2})/({\lambda }_{max}{\widehat{\alpha }}_j^2))$
Dorugade [35]	$k_{D4}=Geometric Mean((2\widehat{{\sigma }^2})/({\lambda }_{max}{\widehat{\alpha }}_j^2))$
Feras and Gore [11]	$k_{FG}=q\widehat{{\sigma }^2}/{\displaystyle {\displaystyle \sum }_{j=1}^q}\left[{\widehat{\alpha }}_j^2/\left\{{({\widehat{\alpha }}_j^4{\lambda }_j^2/4\widehat{{\sigma }^4}+6{\widehat{\alpha }}_j^2{\lambda }_j/\widehat{{\sigma }^2})}^{\frac{1}{2}}-{\widehat{\alpha }}_j^2{\lambda }_j/2\widehat{{\sigma }^2}\right\}\right]$

below shows the estimators. For details on how the estimators were derived, we refer the readers to the corresponding original papers that are available in the list of references [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36].

Here, $\widehat{\alpha }$ is defined as:

$$\widehat{\alpha }=P^T\widehat{\beta }$$

where P is an orthonormal matrix that satisfies $P^T{X}^{T}XP= \mathsf{\Lambda }$, and Λ is a diagonal matrix of eigenvalues (λ_j j = 1, 2,…, q) of $X^{T}X$. To calculate the value of test statistics, we considered each of the k values in Table 1 in Equations (4)–(7) and thus obtained 40 different values of the test statistics. Since a theoretical assessment among the test statistics was not possible, a simulation study was conducted to evaluate the performances of the suggested tests in the following section.

3. Simulation Study

Our simulation study has two parts. First, we analyzed the empirical Type I error of the tests. The test statistics that achieved the nominal size of 5% were kept, and the ones that deviated significantly from the 5% size were discarded. Then, the second part of the simulation study compared the tests statistics that achieved 5% nominal size in regards to statistical power.

3.1. Type I Error Rates Simulation Procedure

R Studio was used for all calculations of this paper. The R package lmridge was used to fit the ridge regression models. For the empirical Type I error simulation and the power of the test, we considered sample sizes n = 30, 50, 80, and 100, the number of regressors q = 4, 6, 8, 10, and 25, and the standard deviation of the error term was chosen as σ = 1. To see the effects of multicollinearity by stating the correlation matrix among the regressors, we assumed ρ = 0.80 and 0.95. An n × p matrix X was created as H Λ^0.5 G^T, where H is any (n × p) matrix whose columns are orthogonal, Λ is the diagonal matrix of eigenvalues of the correlation matrix, and G is the matrix of normalized eigenvectors of the correlation matrix, respectively. Following Halawa and Bassiouni [26], our study was based on the most favorable (MF) direction of β for model (1). The MF orientation of β corresponds to the largest normalized eigenvector of the matrix $X^{T}X$, which is a vector of the form $(1/\sqrt{q})1_q$. We chose not to use the least favorable orientation (LF) of β in our simulation, since all the literature available shows that both orientations give similar results in terms of Type I error and power. For a detailed explanation of MF and LF directions of β and other details of the simulation procedure, please see the paper by Halawa and Bassiouni [26].

To estimate the 5% nominal size (α = 0.05) for testing $H_0: {\beta }_i=0 versus H_1: {\beta }_i\ne 0$ under different conditions, 5000 pseudo random vectors from N(0, σ²) were created to compute the error term in (1). Without loss of any generality, we let zero intercept for (1). Under the null model, substituting the i-th element of the considered MF β by zero, model (1) was used to find 5000 simulated vectors of Y. The estimated sizes were computed as the percentage of times the absolute values of all selected test statistics were greater than the critical value of t_{0.025, (n-q-1).}

3.2. Type I Error Rates: Simulation Results

In

Table 2. Simulated Type I errors for ρ = 0.80 and $\alpha =0.05$.

Statistics	n = 30			n = 50		n = 80	n = 100
	q: 4	6	8	8	10	10	25	Average Type I Error
tOLS	0.0516	0.0553	0.0509	0.0525	0.0486	0.0490	0.0506	0.051
tKHSL	0.0537	0.0544	0.0441	0.0529	0.0445	0.0473	0.0416	0.048
tKHK70	0.0511	0.0546	0.0479	0.0532	0.0470	0.0478	0.0489	0.050
tKTH	0.0513	0.0537	0.0442	0.0522	0.0454	0.0472	0.0417	0.048
tKVR	0.1139	0.1745	0.2069	0.3280	0.3967	0.4864	0.8803	0.370
tKLW76	0.0473	0.0493	0.0349	0.0485	0.0418	0.0489	0.0346	0.044
tKHKB	0.0548	0.0526	0.0434	0.0525	0.0447	0.0473	0.0412	0.048
tKKibAM	0.0828	0.1152	0.1327	0.1856	0.2249	0.2612	0.5162	0.217
tKKibMED	0.0597	0.0588	0.0444	0.0589	0.0520	0.0543	0.0369	0.052
tKKibGM	0.0639	0.0669	0.0487	0.0686	0.0598	0.0656	0.0394	0.059
tKM2	0.0955	0.1333	0.1387	0.1879	0.2063	0.2319	0.2572	0.179
tKM3	0.0917	0.1277	0.1488	0.2037	0.2470	0.2877	0.5848	0.242
tKM4	0.0770	0.0795	0.0603	0.0849	0.0744	0.0859	0.0478	0.073
tKM5	0.0650	0.0685	0.0497	0.0691	0.0612	0.0659	0.0396	0.060
tKM6	0.0798	0.0861	0.0692	0.0961	0.0842	0.0970	0.0572	0.081
tKM8	0.1229	0.1736	0.1836	0.2921	0.3180	0.4119	0.4440	0.278
tKM9	0.0428	0.0581	0.0708	0.0510	0.0661	0.0436	0.1802	0.073
tKM10	0.1197	0.1451	0.1062	0.2468	0.2174	0.3620	0.1286	0.189
tKM11	0.0420	0.0346	0.0282	0.0419	0.0304	0.0450	0.0307	0.036
tKM12	0.1204	0.1514	0.1209	0.2540	0.2322	0.3665	0.1616	0.201
tKD	0.0533	0.0570	0.0511	0.0536	0.0477	0.0483	0.0440	0.051
tKKS	0.0507	0.0546	0.0479	0.0532	0.0473	0.0478	0.0493	0.050
tKKSAM	0.0499	0.0529	0.0458	0.0529	0.0475	0.0481	0.0503	0.050
tKKSMAX	0.0482	0.0495	0.0399	0.0483	0.0408	0.0459	0.0349	0.044
tKKSMED	0.0518	0.0550	0.0497	0.0531	0.0485	0.0488	0.0504	0.051
tKSC	0.0520	0.0545	0.0473	0.0530	0.0469	0.0477	0.0486	0.050
tKA1	0.0508	0.0549	0.0487	0.0532	0.0480	0.0479	0.0504	0.051
tKA2	0.0511	0.0548	0.0487	0.0531	0.0476	0.0480	0.0503	0.051
tKA3	0.0527	0.0536	0.0455	0.0519	0.0466	0.0468	0.0468	0.049
tKA4	0.0528	0.0534	0.0453	0.0521	0.0463	0.0469	0.0462	0.049
tKA5	0.0522	0.0543	0.0482	0.0530	0.0474	0.0481	0.0504	0.051
tKASH	0.1223	0.1848	0.2238	0.3237	0.3951	0.4564	0.8705	0.368
tKNOM	0.0798	0.0693	0.0454	0.0652	0.0503	0.0568	0.0364	0.058
tKSG1	0.0643	0.0611	0.0410	0.0635	0.0525	0.0596	0.0379	0.054
tKSG2	0.0862	0.1125	0.1225	0.1767	0.2037	0.2432	0.4428	0.198
tKK12	0.0576	0.0499	0.0350	0.0482	0.0401	0.0439	0.0425	0.045
tKD1	0.0740	0.0920	0.0933	0.1250	0.1444	0.1583	0.2042	0.127
tKD2	0.0542	0.0525	0.0435	0.0512	0.0456	0.0476	0.0481	0.049
tKD3	0.0522	0.0543	0.0482	0.0530	0.0474	0.0481	0.0504	0.051
tKD4	0.0564	0.0542	0.0432	0.0527	0.0454	0.0482	0.0472	0.050
tKFG	0.0644	0.0566	0.0391	0.0550	0.0446	0.0500	0.0357	0.049

and

Table 3. Simulated Type I errors for ρ = 0.95 and $\alpha =0.05$.

Statistics	n = 30			n = 50		n = 80	n = 100
	q: 4	6	8	8	10	10	25	Average Type I Error
tOLS	0.0480	0.0489	0.0508	0.0522	0.0524	0.0510	0.0507	0.051
tKHSL	0.0582	0.0467	0.0374	0.0499	0.0426	0.0507	0.0327	0.045
tKHK70	0.0480	0.0464	0.0487	0.0505	0.0508	0.0499	0.0495	0.049
tKTH	0.0479	0.0442	0.0439	0.0482	0.0465	0.0477	0.0424	0.046
tKVR	0.1913	0.3010	0.3954	0.5188	0.6153	0.6867	0.9799	0.527
tKLW76	0.0624	0.0537	0.0429	0.0620	0.0589	0.0704	0.0375	0.055
tKHKB	0.0497	0.0433	0.0412	0.0467	0.0453	0.0470	0.0416	0.045
tKKibAM	0.0898	0.1321	0.1615	0.2074	0.2474	0.2783	0.4582	0.225
tKKibMED	0.0614	0.0510	0.0403	0.0499	0.0448	0.0510	0.0366	0.048
tKKibGM	0.0630	0.0505	0.0393	0.0518	0.0467	0.0520	0.0331	0.048
tKM2	0.1903	0.2978	0.3915	0.4868	0.5710	0.6307	0.9065	0.496
tKM3	0.1359	0.1961	0.2473	0.3083	0.3705	0.4085	0.6832	0.336
tKM4	0.1694	0.2243	0.2548	0.3093	0.3243	0.3603	0.2893	0.276
tKM5	0.0634	0.0515	0.0395	0.0523	0.0472	0.0523	0.0331	0.048
tKM6	0.1730	0.2364	0.2793	0.3378	0.3683	0.4111	0.3880	0.313
tKM8	0.1914	0.3016	0.3962	0.5100	0.5917	0.6685	0.9287	0.513
tKM9	0.0659	0.1081	0.1673	0.1393	0.1956	0.1311	0.4297	0.177
tKM10	0.1906	0.2728	0.2940	0.4405	0.4541	0.6072	0.4127	0.382
tKM11	0.0296	0.0417	0.0388	0.0393	0.0469	0.0350	0.0314	0.038
tKM12	0.1921	0.2657	0.2635	0.4501	0.4538	0.6200	0.4060	0.379
tKD	0.0481	0.0489	0.0508	0.0522	0.0524	0.0510	0.0507	0.051
tKKS	0.0480	0.0464	0.0488	0.0508	0.0509	0.0502	0.0495	0.049
tKKSAM	0.0443	0.0397	0.0361	0.0477	0.0451	0.0494	0.0472	0.044
tKKSMAX	0.0452	0.0543	0.0854	0.0432	0.0480	0.0435	0.0376	0.051
tKKSMED	0.0479	0.0482	0.0502	0.0526	0.0526	0.0507	0.0506	0.050
tKSC	0.0483	0.0456	0.0481	0.0502	0.0505	0.0499	0.0494	0.049
tKA1	0.0483	0.0473	0.0494	0.0520	0.0526	0.0506	0.0504	0.050
tKA2	0.0484	0.0473	0.0494	0.0519	0.0525	0.0506	0.0504	0.050
tKA3	0.0483	0.0449	0.0459	0.0492	0.0487	0.0495	0.0483	0.048
tKA4	0.0490	0.0442	0.0451	0.0489	0.0474	0.0490	0.0469	0.047
tKA5	0.0479	0.0465	0.0484	0.0508	0.0513	0.0502	0.0503	0.049
tKASH	0.1859	0.2986	0.3967	0.5238	0.6191	0.6850	0.9770	0.527
tKNOM	0.0631	0.0445	0.0345	0.0445	0.0405	0.0454	0.0352	0.044
tKSG1	0.1040	0.0908	0.0676	0.0902	0.0786	0.0881	0.0418	0.080
tKSG2	0.1100	0.1484	0.1721	0.2247	0.2554	0.2853	0.3872	0.226
tKK12	0.0897	0.0508	0.0309	0.0457	0.0393	0.0458	0.0351	0.048
tKD1	0.0712	0.0892	0.0988	0.1241	0.1380	0.1530	0.1683	0.120
tKD2	0.0525	0.0425	0.0426	0.0484	0.0480	0.0479	0.0490	0.047
tKD3	0.0479	0.0465	0.0484	0.0508	0.0513	0.0502	0.0503	0.049
tKD4	0.0526	0.0435	0.0411	0.0470	0.0463	0.0470	0.0484	0.047
tKFG	0.0483	0.0381	0.0318	0.0429	0.0390	0.0440	0.0346	0.040

, we recorded the empirical sizes of the tests for the MF orientation for correlation levels of 0.80 and 0.95, respectively

If the true Type I error rate is 5%, then, for a simulation based in 5000 runs, the observed Type I error will be in the following interval 95% of the times $0.05\pm 2\sqrt{\frac{0.05\times 0.95}{5000}}\approx (4.4\%, 5.6\%)$. We did not consider those tests for comparison’s purpose whose observed average Type I error was not in the above range.

Based on the above tables, we observed the following:

(i)

The tests based on the following ridge estimators, K_VR, K_KibAM, K_M2, K_M3, K_M4, K_M6, K_M8, K_M9, K_M10, K_M12, K_ASH, K_SG1, K_SG2, and K_D1, have Type I errors very well above the 5% nominal size and therefore cannot be recommended.

(ii)

The tests based on the following ridge estimators, K_M11, K_NOM, and K_FG, did not surpass the 5% nominal size but stayed below it—around 3% to 4%—and therefore cannot be recommended.

(iii)

The rest of the tests (including the test based on the ordinary least squares estimator) were, on average, very close to the nominal size of 5% for different sample sizes, number of variables, and levels of correlation analyzed. These tests were the ones that were compared in terms of statistical power.

We also carried out simulations for nominal sizes of 10% and 1%, and the behavior of the tests was consistent with what was observed for a nominal size of 5%. Those results are available upon request. However, we are including a table of simulated Type I errors for nominal size 1% and correlation level 0.95 in

Table 4. Simulated Type I errors for ρ = 0.95 and $\alpha =0.01$.

Statistics	n = 30			n = 50		n = 80	n = 100
	p: 4	6	8	8	10	10	25	Average Type I Error Probability
tOLS	0.0130	0.0138	0.0066	0.0098	0.0090	0.0106	0.0088	0.010
tKHSL	0.0112	0.0092	0.0030	0.0062	0.0054	0.0080	0.0042	0.007
tKHK70	0.0126	0.0128	0.0062	0.0096	0.0090	0.0104	0.0084	0.010
tKTH	0.0122	0.0126	0.0056	0.0092	0.0080	0.0094	0.0070	0.009
tKVR	0.0514	0.0956	0.1228	0.2274	0.2946	0.3896	0.9000	0.297
tKLW	0.0092	0.0070	0.0010	0.0102	0.0068	0.0118	0.0050	0.007
tKHKB	0.0106	0.0118	0.0050	0.0092	0.0076	0.0092	0.0070	0.009
tKKibAM	0.0198	0.0324	0.0360	0.0748	0.1018	0.1290	0.3414	0.105
tKKibMED	0.0130	0.0110	0.0042	0.0086	0.0058	0.0082	0.0044	0.008
tKKibGM	0.0122	0.0098	0.0046	0.0066	0.0056	0.0094	0.0044	0.008
tKM2	0.0482	0.0928	0.1222	0.2134	0.2784	0.3388	0.7274	0.260
tKM3	0.0330	0.0564	0.0612	0.1182	0.1544	0.1898	0.5078	0.160
tKM4	0.0474	0.0740	0.0750	0.1222	0.1272	0.1560	0.1094	0.102
tKM5	0.0120	0.0100	0.0046	0.0066	0.0056	0.0096	0.0044	0.008
tKM6	0.0478	0.0760	0.0836	0.1376	0.1488	0.1834	0.1706	0.121
tKM8	0.0454	0.0904	0.1222	0.2270	0.2916	0.3768	0.7718	0.275
tKM9	0.0144	0.0278	0.0406	0.0458	0.0688	0.0496	0.2704	0.074
tKM10	0.0457	0.0820	0.0850	0.1804	0.1977	0.3089	0.1871	0.155
tKM11	0.0062	0.0059	0.0015	0.0043	0.0042	0.0065	0.0028	0.004
tKM12	0.0494	0.0830	0.0744	0.1964	0.2058	0.3322	0.1914	0.162
tKD	0.0130	0.0138	0.0066	0.0098	0.0090	0.0106	0.0088	0.010
tKKS	0.0126	0.0128	0.0062	0.0096	0.0090	0.0104	0.0086	0.010
tKKSAM	0.0092	0.0092	0.0058	0.0086	0.0068	0.0096	0.0072	0.008
tKKSMAX	0.0082	0.0138	0.0240	0.0056	0.0062	0.0074	0.0056	0.010
tKKSMED	0.0130	0.0132	0.0064	0.0098	0.0090	0.0108	0.0086	0.010
tKSC	0.0122	0.0126	0.0062	0.0096	0.0088	0.0102	0.0082	0.010
tKA1	0.0128	0.0132	0.0064	0.0100	0.0090	0.0106	0.0088	0.010
tKA2	0.0128	0.0132	0.0064	0.0098	0.0090	0.0106	0.0088	0.010
tKA3	0.0114	0.0126	0.0062	0.0094	0.0084	0.0098	0.0080	0.009
tKA4	0.0110	0.0126	0.0058	0.0094	0.0080	0.0096	0.0078	0.009
tKA5	0.0122	0.0128	0.0062	0.0096	0.0090	0.0106	0.0088	0.010
tKASH	0.0398	0.0748	0.0996	0.2200	0.2900	0.3884	0.8944	0.287
tKNOM	0.0136	0.0080	0.0038	0.0062	0.0048	0.0078	0.0044	0.007
tKSG1	0.0250	0.0176	0.0064	0.0196	0.0124	0.0224	0.0068	0.016
tKSG2	0.0254	0.0380	0.0368	0.0736	0.1010	0.1254	0.2874	0.098
tKK12	0.0206	0.0086	0.0010	0.0070	0.0036	0.0072	0.0046	0.008
tKD1	0.0152	0.0186	0.0188	0.0408	0.0518	0.0634	0.1054	0.045
tKD2	0.0114	0.0112	0.0052	0.0080	0.0080	0.0090	0.0084	0.009
tKD3	0.0122	0.0128	0.0062	0.0096	0.0090	0.0106	0.0088	0.010
tKD4	0.0106	0.0110	0.0046	0.0082	0.0074	0.0086	0.0080	0.008
tKFG	0.0098	0.0080	0.0026	0.0054	0.0044	0.0076	0.0044	0.006

so that one can verify that the behavior of the tests was consistent with the results for 5% nominal size shown before.

3.3. Statistical Power Simulation Procedure

After calculating the empirical type I error rates of the tests based on our initial forty ridge estimators, we discarded seventeen that did not have a nominal size between 4.4% and 5.6%. The remaining twenty-three test statistics were compared in terms of power. Following the paper by Gokpinar and Ebegil [28], we replaced the i-th component of the β vector by $J w(0)\sigma {\beta }_i$, where J is a whole positive number, and $w^2(0)=(1+(q-2)\rho )/\left[(1-\rho )(1+(q-1)\rho )\right]$. We picked J = 6 since that value achieved approximately a power of 80% for the OLS test when q = 4, and having a sizeable power for the OLS test allowed for a better comparison with the other ones.

Based on 5000 simulation runs, the powers of the tests were computed by the proportion of times the absolute value of the test statistic exceeded the critical value t_{0.025, (n-q-1).} All combinations of sample sizes of n = 30, 50, 100 and number of regressors q = 4, 6, 10 were considered under correlation levels of 0.80 and 0.95, respectively.

3.4. Statistical Power: Simulation Results

We recorded the empirical statistical power of the tests for the MF orientation for correlation levels of 0.8 and 0.95 in

Table 5. Powers of tests for ρ = 0.80 and α = 0.05.

Statistics	q = 4			q = 6			q = 10
	n = 30	n = 50	n = 100	n = 30	n = 50	n = 100	n = 30	n = 50	n = 100
tOLS	0.8074	0.8254	0.8496	0.6362	0.6532	0.6774	0.4318	0.4496	0.449
tKHSL	0.8728	0.8912	0.907	0.7198	0.728	0.7634	0.4678	0.5092	0.524
tKHK70	0.8746	0.8912	0.9062	0.7034	0.7122	0.7478	0.453	0.4804	0.4856
tKTH	0.8790	0.8976	0.9154	0.7346	0.7478	0.7878	0.4766	0.5214	0.5406
tKLW76	0.8840	0.901	0.9136	0.7398	0.7506	0.7878	0.485	0.5396	0.5624
tKHKB	0.9332	0.9468	0.956	0.7806	0.8034	0.8402	0.4888	0.5430	0.566
tKKibMED	0.9636	0.975	0.9804	0.8558	0.8798	0.906	0.5666	0.6454	0.6726
tKKibGM	0.9706	0.9796	0.9858	0.8952	0.916	0.9328	0.6472	0.7282	0.7546
tKM5	0.9718	0.98	0.986	0.9004	0.9184	0.9336	0.6568	0.7322	0.7566
tKD	0.8390	0.9012	0.9424	0.6732	0.7332	0.808	0.4444	0.4966	0.5398
tKKS	0.8534	0.861	0.8684	0.6856	0.6882	0.7056	0.4502	0.4728	0.4724
tKKSAM	0.8532	0.8518	0.858	0.6788	0.6812	0.69	0.3578	0.4718	0.4596
tKKSMAX	0.9362	0.9026	0.8822	0.8664	0.766	0.7342	0.7006	0.6278	0.5148
tKKSMED	0.8158	0.8298	0.85	0.6452	0.6574	0.681	0.4376	0.4528	0.451
tKSC	0.8720	0.8888	0.905	0.7010	0.7114	0.7474	0.4542	0.4810	0.4856
tKA1	0.8612	0.8774	0.8952	0.6782	0.689	0.7216	0.445	0.4632	0.4692
tKA2	0.8698	0.8868	0.904	0.6864	0.6972	0.7308	0.4472	0.4666	0.4734
tKA3	0.9042	0.9176	0.933	0.7272	0.737	0.7748	0.463	0.4950	0.5094
tKA4	0.9064	0.9198	0.9346	0.7304	0.7406	0.7788	0.465	0.4978	0.5118
tKA5	0.8936	0.9094	0.9266	0.7016	0.7106	0.7458	0.4508	0.4728	0.4788
tKK12	0.9934	0.9974	0.998	0.8958	0.9158	0.9342	0.5158	0.5902	0.6074
tKD2	0.9384	0.957	0.9642	0.7634	0.7794	0.8126	0.4694	0.4974	0.5086
tKD3	0.8936	0.9094	0.9266	0.7016	0.7106	0.7458	0.4508	0.4728	0.4788
tKD4	0.9494	0.9636	0.9716	0.7966	0.8152	0.8422	0.4826	0.5248	0.5394

and

Table 6. Powers of tests for ρ = 0.95 and α = 0.05.

Statistics	q = 4			q = 6			q = 10
	n = 30	n = 50	n = 100	n = 30	n = 50	n = 100	n = 30	n = 50	n = 100
tOLS	0.8076	0.8254	0.8416	0.6284	0.6528	0.6738	0.4156	0.4418	0.4592
tKHSL	0.891	0.9046	0.9164	0.7478	0.7702	0.792	0.4768	0.5505	0.58
tKHK70	0.8812	0.8986	0.911	0.6916	0.7146	0.7378	0.428	0.465	0.4844
tKTH	0.8846	0.906	0.9168	0.7264	0.7616	0.7864	0.4468	0.5018	0.5314
tKLW76	0.8926	0.9064	0.9198	0.7528	0.7776	0.7992	0.4834	0.546	0.5964
tKHKB	0.9412	0.9582	0.9664	0.785	0.8168	0.8442	0.4546	0.5212	0.5526
tKKibMED	0.9664	0.9796	0.9846	0.8554	0.892	0.905	0.5358	0.6202	0.6554
tKKibGM	0.9754	0.9868	0.9894	0.901	0.9318	0.9436	0.6094	0.7022	0.7344
tKM5	0.9776	0.9868	0.9898	0.907	0.934	0.9442	0.6194	0.708	0.7374
tKD	0.8076	0.8254	0.8416	0.6284	0.6528	0.6738	0.4156	0.4418	0.4592
tKKS	0.8712	0.8844	0.8892	0.6848	0.7044	0.7208	0.428	0.4628	0.4826
tKKSAM	0.7096	0.9358	0.8868	0.4198	0.7978	0.7218	0.1308	0.5156	0.4874
tKKSMAX	0.953	0.9918	0.9598	0.7946	0.9734	0.884	0.4756	0.8742	0.7224
tKKSMED	0.816	0.8286	0.844	0.6348	0.657	0.675	0.4178	0.4442	0.4598
tKSC	0.8768	0.8978	0.9096	0.689	0.7136	0.7384	0.4294	0.4636	0.4856
tKA1	0.849	0.8702	0.8842	0.6554	0.6782	0.7034	0.422	0.4508	0.4676
tKA2	0.8508	0.8722	0.8854	0.6568	0.6794	0.7044	0.4222	0.4512	0.4678
tKA3	0.9088	0.9268	0.9356	0.7116	0.7418	0.7666	0.4352	0.4782	0.4984
tKA4	0.9216	0.9376	0.9438	0.7284	0.7626	0.7862	0.4378	0.4856	0.5094
tKA5	0.8902	0.909	0.9192	0.6798	0.704	0.731	0.426	0.4574	0.4768
tKK12	1	1	1	0.9988	0.9992	0.9996	0.6938	0.7774	0.8222
tKD2	0.937	0.9548	0.9632	0.7384	0.768	0.7832	0.4358	0.4802	0.4986
tKD3	0.8902	0.909	0.9192	0.6798	0.704	0.731	0.426	0.4574	0.4768
tKD4	0.9492	0.9622	0.9696	0.7722	0.7998	0.8204	0.4484	0.5002	0.5206

, respectively.

For a better visualization of the power of the ridge tests vs. the OLS test, we provided the power of the test for α = 0.05 and ρ = 0.80 and q = 4, 6, and 10 in Figure 1, Figure 2 and Figure 3, respectively.

For a better visualization of the power of the ridge tests vs. the OLS test, we provided the power of the test for α = 0.05 and ρ = 0.90 and q = 4, 6, and 10 in Figure 4, Figure 5 and Figure 6, respectively.

The following

Table 7. Average gain in power over the ordinary least squares (OLS) test for α = 0.05 and q = 0.06.

Statistics	Average Gain in Power Over OLS Test
	ρ = 0.80	ρ = 0.95
tKHSL	7%	10%
tKHK70	5%	5%
tKTH	8%	8%
tKLW76	9%	10%
tKHKB	12%	12%
tKKibMED	19%	18%
tKKibGM	23%	23%
tKM5	23%	23%
tKD	7%	0%
tKKS	3%	4%
tKKSAM	1%	-2%
tKKSMAX	13%	21%
tKKSMED	0%	0%
tKSC	5%	5%
tKA1	4%	3%
tKA2	4%	3%
tKA3	8%	7%
tKA4	8%	9%
tKA5	6%	5%
tKK12	19%	28%
tKD2	10%	9%
tKD3	6%	5%
tKD4	12%	11%

provides the average gains in power for the tests with respect to the OLS test for both levels of correlations, namely 0.80 and 0.95.

Based on the above tables, we observed the following:

(i)

All the considered tests (with the exception of the one based on K_KSAM when n = 30) achieved higher statistical power than the OLS test.

(ii)

Keeping the number of variables in the model fixed, if the sample size increased, the power of the tests also increased, as was expected.

(iii)

Keeping the sample size fixed, increasing the number of variables in the model decreased the power of the tests.

(iv)

Among the tests considered, the ones with the highest gain in power over the OLS test across different values of q, n, and ρ were the ones based on the following ridge estimators: K_HKB, K_KibMED, K_KibGM, K_M5, K_KSMAX, K_K12, and K_D4. The observed gains over the OLS test were between 12% to 28% (see Table 7). Therefore, we recommend these seven tests to data analysis practitioners since they achieve the highest power among the ones considered while maintaining a 5% probability of Type I error.

4. Application Example

The following car consumption dataset available in the webpage http://data-mining-tutorials.blogspot.com/2010/05/solutions-for-multicollinearity-in.html (See Appendix A) was used to illustrate the finding of the paper.

The goal was to create a linear regression model to predict the consumption of cars from various characteristics such as: price, engine size, horsepower, and weight. There were n = 27 observations in the dataset. We made use of the mctest and the lmridge R packages in our computations. For more info on the functionality of the aforementioned packages, see Ullah, Aslam, and Altaf [36].

There was strong evidence of multicollinearity in the data, as evidenced by all of the VIFs (variance inflation factors) being greater than 10 (See

Table 8. Variance inflation factors (VIFs) of the regressors.

Variable	VIF
Price	19.7919
Engine Size	12.8689
Horsepower	14.8922
Weight	10.2260

below).

Also, the condition number (CN), which is defined as $CN={(\frac{largest eigenvalue (X^{T}X)}{smallest eigenvalue(X^{T}X)})}^2=38.3660$, was greater than 30, indicating high dependency between the explanatory variables. Since multicollinearity existed, ridge regression estimation was preferable to OLS estimation for this model. We contrasted the results of the OLS method with ridge regression using two of the ridge estimators that showed higher power, namely K_KibMED and K_KibGM, and the analyses are given in the following

Table 9. Regression analysis.

Variable	OLS (k = 0)			KKibMED (k = 0.34766)			KKibGM (k = 0.38798)
	coef	t-stat	p-value	coef	t-stat	p-value	coef	t-stat	p-value
Price	3 × 10−5	0.7697	0.4497	4 × 10−5	7.8052	<0.0001	4 × 10−5	8.2386	<0.0001
Engine Size	0.0012	1.7103	0.1013	0.0007	5.9079	<0.0001	0.0007	6.2434	<0.0001
Horsepower	−0.0037	−0.2546	0.8014	0.0108	4.5418	0.0001	0.0111	4.9219	<0.0001
Weight	0.0037	2.9330	0.0077	0.0022	7.6687	<0.0001	0.0021	7.9105	<0.0001
Intercept	1.8380	−0.9442	0.3553	3.1625	−8.6380	<0.0001	3.2371	−9.1118	<0.0001
$MSE(\widehat{\beta })$	23.4343			15.4460			16.0363

.

From Table 9, we observed that no variable except for weight was a significant predictor of car consumption under the OLS estimation. When ridge regression was applied, all variables (price, engine size, horsepower, and weight of the car) became significant predictors of car consumption, and the MSE [computed using Equation (5) of the coefficient vector also decreased compared to the OLS estimate, as is expected when a ridge regression approach is appropriate. Also, we could see that the sign of the coefficient of horsepower reversed from negative and not significant under the OLS estimation to positive and significant under ridge regression estimation. Change of sign in the coefficients is one of the signals that the ridge regression approach is a good fit for this particular problem according to what is explained in the foundational paper by Hoerl and Kennard [1]. Also, it makes physical sense that higher horsepower of the car leads to higher gas consumption, thus a positive sign for the coefficient would be the right choice.

5. Some Concluding Remarks

In this paper, we investigated forty different ridge regression estimators in order to find some good test statistics for testing the regression coefficients of the linear regression model in case of multicollinearity. A simulation study under different conditions was constructed to make the empirical comparison among the ridge regression estimators. We compared the performance of the test statistics based on the empirical size and the power of the test. It was observed from our simulations that the tests based on ridge estimators K_HKB, K_KibMED, K_KibGM, K_M5, K_KSMAX, K_K12, and K_D4 were the best in terms of achieving higher power gains with respect to the OLS test while maintaining a 5% nominal size.

Our results are consistent with Kibria and Banik [29], although they did not conclude which tests were the best ones. While Gokpinar and Ebergil [28] concluded that the best tests in terms of power were the ones based on K_HSL and K_HKB, we found that the gains in power over the OLS test for K_HSL are somewhat smaller than the gains in power for the tests based on the seven estimators we mentioned above, and therefore we did not include K_HSL in our final list.

All in all, based on our simulation results, we recommend the tests based on K_HKB, K_KibMED, K_KibGM, K_M5, K_KSMAX, K_K12, and K_D4 to statistical practitioners for the purpose of testing linear regression coefficients when multicollinearity is present.