A Novel Angle-Based Learning Framework on Semi-supervised Dimensionality Reduction in High-Dimensional Data with Application to Action Recognition

Abstract

The existing outliers in high-dimensional data create various challenges to classify datasets such as the exact classification with imbalanced scatters. In this paper, we propose an angle-based framework as Angle Global and Local Discriminant Analysis (AGLDA) to consider imbalanced scatters. AGLDA chooses an optimal subspace by using angle cosine to achieve appropriate scatter balance in the dataset. The privilege of this method is to classify datasets with the effect of outliers by finding optimal subspace in high-dimensional data. Generally, this method is more effective and more reliable than other methods to classify data when there are outliers. Besides, human posture classification has been used as an application of the balanced semi-supervised dimensionality reduction to assist human factor experts and designers of industrial systems for diagnosing the type of maintenance crew postures. The experimental results show the efficiency of the proposed method via two real case studies, and the results have also been verified by comparing it with other approaches.

Appendices

Appendix A

To explain how a linear combination of features can be achieved, the following assumptions are made. Given a data matrix X, $X = \{ x_{i} |x_{1} ,x_{2} , \ldots ,x_{l} \} \in R^{n \times l}$ where $l$ and $n$ are, respectively, the number of samples and the dimension of data, LDA can model the supervised data distribution by mapping the input data $x_{i} \in R^{n}$ in $n$-dimensional space into a vector $y_{i} \in R^{r}$ in the lower $r$-dimensional [34, 35]. Then, a scalar y can be obtained by projecting the samples $x$ onto a line

$$y_{i} = a^{T} x_{i} ,\;a \in R,\;r \le n.$$

(A-1)

The optimization procedure $a$ can be applied in the following form

$$a_{\text{opt}}^{\text{LDA}} = \mathop {\arg \limits_{a}\hbox{max} } \frac{{{\text{tr}}\left( {a^{T} S_{\text{b}} a} \right)}}{{{\text{tr}}(a^{T} S_{\text{w}} a)}},$$

(A-2)

where $a^{T} a = I$ and the within-class scatter matrix $S_{\text{w}}$ and the between-class matrix $S_{\text{b}}$ are defined as

$$S_{\text{w}} = \frac{1}{N}\mathop \sum \limits_{i = 1}^{c} \mathop \sum \limits_{{x \in X_{j} }} \left( {x - \mu_{i} } \right)\left( {x - \mu_{i} } \right)^{T} = X^{T} L_{F} X$$

(A-3)

$$S_{\text{b}} = \frac{1}{N}\mathop \sum \limits_{i = 1}^{c} N_{i} \left( {\mu_{i} - \mu } \right)\left( {\mu_{i} - \mu } \right)^{T} .$$

(A-4)

Here, $c$ is the number of classes, $L_{F} = I - F$ is a graph Laplacian matrix, and F is an adjacency matrix. If any pair of samples belongs to the same class, elements of F will be considered $\frac{1}{{N_{i} }}$ and 0 otherwise. Also, $\mu_{i}$ and $\mu$ are the mean of ith class and the mean of the labeled points, respectively.

Appendix B

SDA has been formulated as follows:

$$J_{S} \left( a \right) = \frac{{a^{T} S_{\text{b}} a}}{{a^{T} S_{\text{w}} a + \gamma a^{T} S_{L} a}} = \frac{{a^{T} S_{\text{b}} a}}{{a^{T} S_{\text{w}} a + J\left( a \right)}}, ,$$

(B-1)

where $a$ is an $n$-dimensional projection vector, $S_{\text{b}}$ and $S_{\text{w}}$ are the between–class and within-class scatter matrices as shown in Eqs. (A-3) and (A-4). Furthermore, $J\left( a \right)$ is a regularization term which is learned from the labeled and unlabeled data and $S_{L} = X^{T} LX$ is a graph scatter matrix [30, 36, 37]. $L = D - H$ is a Laplacian matrix and $D$ is a diagonal matrix whose entries are the column (or row, because $H$ is symmetric) sum of $H.$ SDA minimizes both the within-class scatter of labeled data and the local scatter of the labeled and unlabeled data and simultaneously maximizes the between-class scatter of labeled samples.

A p-nearest neighbor graph $G = \left( {V,E} \right)$ can be used to model scatter matrices. The vertex set $V = \left\{ {1,2, \ldots ,n} \right\}$ corresponding to the data points in $X$ and the edge $E \subseteq V \times V$ represents the relationships between data points [38, 39]. Let H denote the similarity matrix between entire data points and $H_{ij}$ is $\left( {i,j} \right)$ element of $H$ matrix. Hence, each edge of graph is assigned $H_{ij}$ which put the edge between nodes i and j if $x_{i}$ and $x_{j}$ are close to each other among p-nearest neighbors. Thus, $H_{ij}$ can be defined as follows:

$$H_{ij} = \left\{ {\begin{array}{*{20}l} {1,} \hfill & {{\text{if}}\;x_{i} \in N_{p} \left( {x_{j} } \right)\;{\text{or}}\;x_{j} \in N_{p} \left( {x_{i} } \right) } \hfill \\ {0,} \hfill & {\text{otherwise}} \hfill \\ \end{array} } \right.$$

(B-2)

where $N_{p} \left( {x_{i} } \right)$ represents the set of p-nearest neighbors of $x_{i}$. $A$ Laplacian style matrix is defined as follows:

$$\begin{aligned} S_{L} & = \frac{1}{2}\mathop \sum \limits_{i = 1}^{N} \mathop \sum \limits_{j = 1}^{N} H_{ij} (x_{i} - x_{j} )^{2} \\ & = X^{T} LX \\ \end{aligned}$$

(B-3)

According to [16], solving optimization SDA problem is equivalent to solving the following generalized eigenvalue problem (GEP) [40]:

$$S_{\text{b}} a = \lambda \left( {S_{\text{w}} + \gamma XLX^{T} } \right)a.$$

(B-4)

Appendix C

The RGLDA optimization problem has been defined as follows [41, 42]:

$$\begin{aligned} & \hbox{min} \frac{1}{2} a^{T} S_{D} a + \nu \sum \varepsilon_{i} \\ & \quad {\text{s}}.{\text{t}}\; \left| {a^{T} H_{i} } \right| + \varepsilon_{i} \ge 1,\;\varepsilon_{i} \ge 0 \\ \end{aligned}$$

(C-1)

where $S_{D} = S_{\text{w}} + \gamma S_{L}$, $\varepsilon_{i}$ is a loss function which relaxes the hard constraints and $H = [ \left( {x_{1} - \mu } \right),\left( {x_{2} - \mu } \right), \ldots \left( {x_{l} - \mu } \right),\gamma \left( {x_{1} - m} \right),\gamma \left( {x_{2} - \mu } \right), \ldots ,\gamma \left( {x_{l + M} - m} \right)].$ Here, $M$ is the number of unlabeled data. Gao et al. also constructed the following semi-supervised problem:

$$J_{\text{RG}} \left( a \right) = \frac{{a^{T} S_{\text{b}} a + \gamma a^{T} S_{N} a}}{{a^{T} S_{\text{w}} a + \gamma a^{T} S_{L} a}},$$

(C-2)

where $\gamma$ denotes a regularization parameter. Equation (C-2) can be rewritten with the following semi-supervised problem:

$$J_{\text{RG}} \left( a \right) = \frac{{a^{T} S_{\text{w}} a + \gamma a^{T} S_{L} a}}{{a^{T} S_{T}^{{\prime }} a + \gamma a^{T} S_{T} a}},$$

(C-3)

see relation between Eqs. (C-2) and (C-3) in Ref. [16]. After that, the following optimization problem can be solved

$$\begin{aligned} & \hbox{min} \frac{1}{2}a^{T} S_{D} a + \nu e^{T} \varepsilon \\ & \quad {\text{s}}.{\text{t}}\;Fa + \varepsilon \ge e,\;\varepsilon \ge 0 \\ \end{aligned}$$

(C-4)

where $F = \{ {\text{sign}}\left( {a_{t}^{T} \left( {H_{1} } \right)} \right)\left( {H_{1} } \right),{\text{sign}}\left( {a_{t}^{T} \left( {H_{2} } \right)} \right)\left( {H_{2} } \right), \ldots ,{\text{sign}}\left( {a_{t}^{T} \left( {H_{l + M} } \right)} \right)\left( {H_{l + M} } \right)\}^{T}$ and $e$ is a column vector of ones with $l + M$ dimensions. According to the Concave–Convex Procedure (CCP) [31, 43], to solve Eq. (C-4), first of all $a_{t}$ is obtained by minimizing (C-1), and then, it is replaced with $a_{t + 1}$. The solution (C-1) can be conveniently obtained by solving the following Wolf dual mathematical optimization formulation:

$$\begin{aligned} & \hbox{min} \frac{1}{2}\tau^{T} F(S_{D} )^{ - 1} F^{T} \tau - e^{T} \tau \\ & \quad {\text{s}}.{\text{t}}\;0 \le \tau \le \upsilon e. \\ \end{aligned}$$

(C-5)

It is easy to check, and the optimization (C-4) can be emulated by a regularized SVM without threshold.

For emulation, a new training set consisting of $l + M$ training samples $\left( {k_{1} ,y_{1} } \right),\left( {k_{2} ,y_{2} } \right), \ldots$,$\left( {k_{l + M} ,y_{l + M} } \right)$ is needed, where $k_{i} = H_{i}$, $y_{i} \in \left\{ { - 1, + 1} \right\}$ and $i = 1,2, \ldots ,l + M$ are the class labels estimated by computing ${\text{sign}}\left( {a_{t}^{T} \left( {H_{i} } \right)} \right)$.

Appendix D

Proof of Eq. (5).

The scatter $S_{L}$ within a local neighborhood from two samples $x_{i}$ and $x_{j}$ based on angle can be obtained as follows:

$$\begin{aligned} & \frac{1}{2}\mathop \sum \limits_{i,j = 1}^{n} H_{ij} \parallel x_{i} - x_{j} \parallel_{2}^{2} \\ & \quad = \frac{1}{2}\mathop \sum \limits_{i,j = 1}^{n} 2H_{ij} \left( {1 - \cos \left( {x_{i} ,x_{j} } \right)} \right) \\ & \quad = \mathop \sum \limits_{i,j = 1}^{n} H_{ij} \left( {1 - \frac{{x_{i}^{T} x_{j} }}{{\sqrt {x_{i}^{T} x_{j} } \sqrt {x_{j}^{T} x_{i} } }}} \right) \\ & \quad = \mathop \sum \limits_{i,j = 1}^{n} H_{ij} \left( {1 - \frac{{(a^{T} x_{i} )^{T} (a^{T} x_{j} )}}{{\sqrt {(a^{T} x_{i} )^{T} (a^{T} x_{i} )} \sqrt {(a^{T} x_{j} )^{T} (a^{T} x_{j} )} }}} \right) \\ & \quad = \mathop \sum \limits_{i,j = 1}^{n} H_{ij} \left( {1 - \frac{{(a^{T} x_{i} )^{T} }}{{\parallel a^{T} x_{i} \parallel }} \cdot \frac{{(a^{T} x_{j} )}}{{\parallel a^{T} x_{j} \parallel }}} \right) \\ & \quad = \mathop \sum \limits_{i,j = 1}^{n} H_{ij} \left( {1 - (a^{T} x_{i} )_{e} (a^{T} x_{j} )_{e} } \right) \\ & \quad = \mathop \sum \limits_{i,j = 1}^{n} H_{ij} \left( {1 - \cos \left( {(a^{T} x_{i} )_{e} ,(a^{T} x_{j} )_{e} } \right)} \right) \\ & \quad = a^{T} \frac{1}{2}\mathop \sum \limits_{i,j = 1}^{n} H_{ij} \left( {x_{i} - x_{j} } \right)_{e}^{T} \left( {x_{i} - x_{j} } \right)_{e} a \\ & \quad = {\text{tr}}\left[ {a^{T} S_{L}^{{\prime }} a} \right]. \\ \end{aligned}$$

Meanwhile, the above equation has been transformed based on graph to:

$$\begin{aligned} & = {\text{tr}}\left[ {X^{T} \left( {I - D^{{ - \frac{1}{2}}} HD^{{ - \frac{1}{2}}} } \right)X} \right] \\ & = {\text{tr}}\left[ {X^{T} HX} \right] - {\text{tr}}\left[ {X^{T} DX} \right] \\ & = {\text{tr}}\left[ {X^{T} L_{H} X} \right], \\ \end{aligned}$$

where $D = {\text{diag}}\left( {d_{i} } \right)$, $d_{i} = \sum H_{ij}$, $L_{H} = H - D.$