Variable selection and identification of high-dimensional nonparametric nonlinear systems by directional regression

Abstract

The importance of discovering significant variables from a large candidate pool is now widely recognized in many fields. There exist a number of algorithms for variable selection in the literature. Some are computationally efficient but only provide a necessary condition for, not a sufficient and necessary condition for, testing if a variable contributes or not to the system output. The others are computationally expensive. The goal of the paper is to develop a directional variable selection algorithm that performs similar to or better than the leading algorithms for variable selection, but under weaker technical assumptions and with a much reduced computational complexity. It provides a necessary and sufficient condition for testing if a variable contributes or not to the system. In addition, since indicators for redundant variables aren’t exact zero’s, it is difficult to decide variables whether are redundant or not when the indicators are small. This is critical in the variable selection problem because the variable is either selected or unselected. To solve this problem, a penalty optimization algorithm is proposed to ensure the convergence of the set. Simulation and experimental research verify the effectiveness of the directional variable selection method proposed in this paper.

Appendices

Appendix A

Following the definition of $G$,

$$ \begin{aligned} G & = E\left[ {2\Sigma - E\left[ {({\textbf{x}} - {\tilde{\textbf{x}}})(\textbf{x} - {\tilde{\textbf{x}}})^{{\text{T}}} |y,\tilde{y}} \right]} \right]^{2} \\ & = 4\Sigma^{2} - 4\Sigma E\left[ {E\left[ {({\textbf{x}} - {\tilde{\textbf{x}}})(\textbf{x}- {\tilde{\textbf{x}}})^{{\text{T}}} |y,\tilde{y}} \right]} \right] \\ & \quad + E\left[ {\left( {E\left[ {({\textbf{x}} - {\tilde{\textbf{x}}})(\textbf{x} - {\tilde{\textbf{x}})}^{T} |y,\tilde{y}} \right]} \right)^{2} } \right]. \\ \end{aligned} $$

Further define

$$ \begin{aligned} & E\left[ {{{(\textbf{x} - \tilde{\textbf{x}})(\textbf{x} - \tilde{\textbf{x}})}}^{T} |y,\tilde{y}} \right] \\ & \quad = \underbrace {{E\left[ {{\mathbf{xx}}^{{\text{T}}} |y} \right] - E[{\mathbf{x}}|y]E\left[ {{\tilde{\textbf{x}}}^{{\text{T}}} |\tilde{y}} \right]}}_{{Q(y,\tilde{y})}} \\ & & \quad \quad \underbrace {{ - E[{\tilde{\textbf{x}}}|\tilde{y}]E\left[ {{\textbf{x}}^{{\text{T}}} |y} \right] + E\left[ {{\tilde{\textbf{x}\tilde{x}}}^{{\text{T}}} |\tilde{y}} \right]}}_{{Q(\tilde{y},y)}}. \\ \end{aligned} $$

$$ \begin{aligned} & E\left[ {\left( {E\left[ {{{(\textbf{x}- \tilde{\textbf{x}})(\textbf{x} - \tilde{\textbf{x}})}}^{{\text{T}}} |y,\tilde{y}} \right]} \right)^{2} } \right] \\ & \quad = 2E\left[ {Q^{2} (y,\tilde{y})} \right] + 2E[Q(y,\tilde{y})Q(\tilde{y},y)] \\ E\left[ {Q^{2} (y,\tilde{y})} \right] \\ & \quad = E\left[ {E^{2} \left[ {{\mathbf{xx}}^{{\text{T}}} |y} \right]} \right] + E^{2} \left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]} \right] \\ & E[Q(y,\tilde{y})Q(\tilde{y},y)] \\ & \quad = \Sigma^{2} + E\left[ {E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]E[{\mathbf{x}}|y]} \right]E\left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]} \right] \\ \end{aligned} $$

It follows that

$$ \begin{aligned} G & = 2E\left[ {E^{2} \left[ {{\mathbf{xx}}^{{\text{T}}} - \Sigma |y} \right]} \right] + 2E^{2} \left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]} \right] \\ & \quad + 2E\left[ {E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]E[{\mathbf{x}}|y]} \right]E\left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]} \right] \\ \end{aligned} $$

Thus, $G$ can be expressed as Eq. (6).

Appendix B

Proof of Theorem 1

Since $y$ is independent of $x_{i} ,i = d + 1, \ldots ,q$, and $x_{i} \in S_{c} ,x_{j} \in S_{r}$ are independent. $E\left[ {{\mathbf{xx}}^{{\text{T}}} - \Sigma |y} \right]$ can be represented as

$$ E\left[ {{\mathbf{xx}}^{{\text{T}}} - \Sigma |y} \right] = \left( {\begin{array}{*{20}c} {K_{11} } & 0 \\ 0 & 0 \\ \end{array} } \right),\quad K_{11} \in R^{d \times d} . $$

If $E\left[ {x_{i}^{2} |y} \right],\;\;i = 1, \ldots d$ is non-degenerated, then we have

$$ {\text{diag}} \left( {K_{11} } \right) = \left[ {\sigma_{1} , \ldots ,\sigma_{d} } \right] $$

where $\left| {\sigma_{i} } \right| > 0,\;\;i = 1, \ldots ,d$.

$$ \begin{array}{*{20}c} {E^{2} \left( {{\mathbf{xx}}^{{\text{T}}} - \Sigma |y} \right) = \left( {\begin{array}{*{20}c} {K_{11} } & 0 \\ 0 & 0 \\ \end{array} } \right)\left( {\begin{array}{*{20}c} {K_{11} } & 0 \\ 0 & 0 \\ \end{array} } \right) = \left( {\begin{array}{*{20}c} {K_{11}^{2} } & 0 \\ 0 & 0 \\ \end{array} } \right)} \\ {{\text{diag}} \left( {K_{11}^{2} } \right) \ge \left[ {\sigma_{1}^{2} , \ldots ,\sigma_{d}^{2} } \right] > 0} \\ \end{array} $$

$$ \begin{aligned} & \Rightarrow e_{i}^{{\text{T}}} 2E\left[ {E^{2} \left[ {{\mathbf{xx}}^{{\text{T}}} - {\Sigma |}y} \right]} \right]e_{i} \\ & \quad = 2E\left[ {e_{i}^{{\text{T}}} E^{2} \left[ {{\mathbf{xx}}^{{\text{T}}} - {\Sigma |}y} \right]e_{i} } \right] > 0,\quad i = 1, \ldots ,d \\ \end{aligned} $$

$$ \begin{aligned} & e_{i}^{{\text{T}}} 2E\left[ {E^{2} \left[ {{\mathbf{xx}}^{{\text{T}}} - \Sigma |y} \right]} \right]e_{i} \\ & \quad = 2E\left[ {e_{i}^{T} E^{2} \left[ {{\mathbf{xx}}^{{\text{T}}} - \Sigma |y} \right]e_{i} } \right] = 0,\;\;\;i = d + 1, \ldots ,q \\ \end{aligned} $$

Else if $E\left[ {x_{i} |y} \right],\;\;i = 1, \ldots d$ is non-degenerated, i.e., it is random and is almost surely not a constant, we have,

$$ E\left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]} \right] = \left( {\begin{array}{*{20}c} {U_{11} } & 0 \\ 0 & 0 \\ \end{array} } \right),\quad U_{11} \in R^{d \times d} $$

where ${\text{diag}} \left( {U_{11} } \right) = \left[ {\eta_{1} , \ldots ,\eta_{d} } \right] > 0$. Then

$$ E^{2} \left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]} \right] = \left( {\begin{array}{*{20}c} {U_{11} } & 0 \\ 0 & 0 \\ \end{array} } \right)\left( {\begin{array}{*{20}c} {U_{11} } & 0 \\ 0 & 0 \\ \end{array} } \right) $$

and ${\text{diag}} \left( {E^{2} \left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]} \right]} \right) \ge \left[ {\eta_{1}^{2} , \ldots ,\eta_{d}^{2} } \right] > 0$.

In addition, the coefficient $E\left[ {E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]E[{\mathbf{x}}|y]} \right]$ of $E\left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]} \right]$ holds, $E\left[ {E\left[ {{\mathbf{x}}^{{\text{T}}} |y} \right]E[{\mathbf{x}}|y]} \right] > 0$. Under Assumptions 1 and 2, we have

$$ \begin{array}{*{20}c} {e_{i}^{T} \left\{ {2E^{2} \left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{T} |y} \right]} \right] + 2E\left[ {E\left[ {{\mathbf{x}}^{T} |y} \right]E[{\mathbf{x}}|y]} \right].} \right.} \\ {\left. {E\left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{T} |y} \right]} \right]} \right\}e_{i} > 0,i = 1, \ldots ,d} \\ {e_{i}^{T} \left\{ {2E^{2} \left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{T} |y} \right]} \right] + 2E\left[ {E\left[ {{\mathbf{x}}^{T} |y} \right]E[{\mathbf{x}}|y]} \right].} \right.} \\ {\left. {E\left[ {E[{\mathbf{x}}|y]E\left[ {{\mathbf{x}}^{T} |y} \right]} \right]} \right\}e_{i} = 0,i = d + 1, \ldots ,q.} \\ \end{array} $$

The above equality and inequality imply that $e_{i}^{{\text{T}}} Ge_{i} > 0\;\;{\text{for}}\;\;i = 1, \ldots ,d$and

$$ e_{i}^{T} Ge_{i} = 0 \quad {\text{for}} \quad \, i = d + 1, \ldots ,q. $$

This completes the proof.□