Redirected transfer learning for robust multi-layer subspace learning

Abstract

Unsupervised transfer learning methods usually exploit the labeled source data to learn a classifier for unlabeled target data with a different but related distribution. However, most of the existing transfer learning methods leverage 0-1 matrix as labels which greatly narrows the flexibility of transfer learning. Another major limitation is that these methods are influenced by the redundant features and noises residing in cross-domain data. To cope with these two issues simultaneously, this paper proposes a redirected transfer learning (RTL) approach for unsupervised transfer learning with a multi-layer subspace learning structure. Specifically, in the first layer, we first learn a robust subspace where data from different domains can be well interlaced. This is made by reconstructing each target sample with the lowest-rank representation of source samples. Besides, imposing the $L_{2,1}$-norm sparsity on the regression term and regularization term brings robustness against noise and works for selecting informative features, respectively. In the second layer, we further introduce a redirected label strategy in which the strict binary labels are relaxed into continuous values for each datum. To handle effectively unknown labels of the target domain, we construct the pseudo-labels iteratively for unlabeled target samples to improve the discriminative ability in classification. The superiority of our method in classification tasks is confirmed on several cross-domain datasets.

Optimization solutions of RTL

Update W: W can be solved by fixing other irrelevant variables and optimized by the following problem:

$$\begin{aligned} \begin{aligned} {W^*} = \arg \mathop {\min }\limits _W&{\lambda _2}{\left\| W \right\| _{2,1}} + \frac{\mu }{2}\left\| {{W^t}{X_T} - {W^t}{X_S}Z - E + \frac{{{M_1}}}{\mu }} \right\| _F^2 \\ {}&+ \frac{\mu }{2}\left\| {A - P{W^t} + \frac{{{M_3}}}{\mu }} \right\| _F^2. \end{aligned} \end{aligned}$$

(19)

Then, we take partial derivative of (19) with respect to W equal to zero, a closed-form solution $W^*$ can be obtained as

$$\begin{aligned} \begin{array}{@{}l} {W^*} = {\left( {2{\lambda _2}G + \mu {K_1}K_1^{t} + \mu I} \right) ^{ - 1}}(\mu {K_1}K_2^{t} + \mu {K_3}^{t}P), \end{array} \end{aligned}$$

(20)

where $K_1=X_T-X_SZ,K_2=E-\frac{M_1}{\mu },K_3=A-\frac{M_3}{\mu }$. $G \in \mathbb {R}^{m \times m}$ is a diagonal matrix and its each diagonal element ${\left( G \right) _{ii}} = \frac{1}{{2{{\left\| {{{\left( W \right) }^i}} \right\| }_2}}}$, where ${\left( \cdot \right) ^{i}}$ denotes the ith column of a matrix.

Update P: P can be solved by fixing other irrelevant variables and optimized by the following problem:

$$\begin{aligned} \begin{gathered} {P^*} = \arg \mathop {\min }\limits _P \frac{\mu }{2}\left\| {P{W^t} - A + \frac{{{M_3}}}{\mu }} \right\| _F^2 \qquad {\text {subject to }}{P^t}P = I. \end{gathered} \end{aligned}$$

(21)

We can convert (21) to the following maximization problem:

$$\begin{aligned} \mathop {\max }\limits _P \frac{\mu }{2} tr\left( {\mu {K_3}W{P^{t}}} \right) \qquad \text {subject\,to\,}{P^{t}}P = I. \end{aligned}$$

(22)

Let the SVD of ${\mu {K_3}W}$. Then, according to [35], the optimal solution to the problem above is

$$\begin{aligned} \begin{array}{@{}l} {P^*} = U{V^{t}}\end{array}, \end{aligned}$$

(23)

where U, V is the SVD decomposition value of ${\mu {K_3}W}$.

Update A: A can be solved by fixing other irrelevant variables and optimized by the following convex problem:

$$\begin{aligned} {A^*} = \arg \mathop {\min }\limits _A \frac{1}{2}\left\| {T - AX} \right\| _F^2 + \frac{\mu }{2}\left\| {P{W^t} - A + \frac{{{M_3}}}{\mu }} \right\| _F^2. \end{aligned}$$

(24)

Then, we take partial derivative of (24) with respect to A equal to zero, a closed-form solution $A^*$ can be obtained as

$$\begin{aligned} {A^*} = \left( {T{X^t} + \mu \left( {P{W^t} + \frac{{{M_3}}}{\mu }} \right) } \right) {\left( {X{X^t} + \mu I} \right) ^{ - 1}}. \end{aligned}$$

(25)

UpdateZ: Z can be solved by fixing other irrelevant variables and optimized by the following convex problem:

$$\begin{aligned} {Z^*} = \arg \mathop {\min }\limits _Z \frac{\mu }{2}\left\| {{W^t}{X_T} - {W^t}{X_S}Z - E + \frac{{{M_1}}}{\mu }} \right\| _F^2 + \frac{\mu }{2}\left\| {Z - J + \frac{{{M_2}}}{\mu }} \right\| _F^2. \end{aligned}$$

(26)

Then, we take partial derivative of (26) with respect to Z equal to zero, a closed-form solution $Z^*$ can be obtained as

$$\begin{aligned} \begin{array}{*{20}{c}} {{Z^ * } = {{\left( {X_S^tW{W^t}{X_S} + {I_{{n_s}}}} \right) }^{ - 1}}\left[ {X_S^tW{{\left( {{W^t}{X_T} - E + \frac{{{M_1}}}{\mu }} \right) }^t} + J - \frac{{{M_2}}}{\mu }} \right] }. \end{array} \end{aligned}$$

(27)

Update E: E can be solved by fixing other irrelevant variables and optimized by the following convex problem:

$$\begin{aligned} {E^*} = \arg \mathop {\min }\limits _E {\left\| E \right\| _{2,1}} + \frac{\mu }{2}\left\| {E - {W^t}{X_T} + {W^t}{X_S}Z - \frac{{{M_1}}}{\mu }} \right\| _F^2. \end{aligned}$$

(28)

According to [36], the optimal $E^*$ can be computed as

$$\begin{aligned} {\left( {{E^*}} \right) _{:,i}} = \left\{ \begin{gathered} \frac{{{{\left\| {{K_{:,i}}} \right\| }_2} - 1}}{{{{\left\| {{K_{:,i}}} \right\| }_2}}}{K_{:,i}},{\left\| {{K_{:,i}}} \right\| _2} > \frac{1}{\mu }; \\ 0,\qquad \qquad {\text {otherwise}}, \end{gathered} \right. \end{aligned}$$

(29)

where $K = {W^t}{X_T} - {W^t}{X_S}Z + \frac{{{M_1}}}{\mu }$.

Update J: J can be solved by fixing other irrelevant variables and optimized by the following convex problem:

$$\begin{aligned} \begin{array}{*{20}{c}} {{J^*} = \arg \mathop {\min }\limits _J {\lambda _1}{{\left\| J \right\| }_*} + \frac{\mu }{2}\left\| {Z - J + \frac{{{M_2}}}{\mu }} \right\| _F^2}. \end{array} \end{aligned}$$

(30)

The optimal $J^*$ can be computed by utilizing singular value thresholding (SVT) algorithm [37] as

$$\begin{aligned} \begin{array}{*{20}{c}} {{J^*} = {\Omega _{\frac{{{\lambda _1}}}{\mu }}}\left( {Z + \frac{{{M_2}}}{\mu }} \right) } \end{array} \end{aligned}$$

(31)

where $\Omega$ is the singular value shrinkage operator.

Update T: T can be solved by fixing other irrelevant variables and optimized by the following convex problem:

$$\begin{aligned} {T^*} = \arg \mathop {\min }\limits _T \frac{1}{2}\left\| {T - AX} \right\| _F^2 \qquad {\text {subject to }}{t_{i,{l_i}}} - \mathop {\max }\limits _{j \ne {l_i}} {t_{i,j}} \ge 1. \end{aligned}$$

(32)

It is obvious that problem (32) can be decomposed into the following sub-problems

$$\begin{aligned} \mathop {\min }\limits _{{t_{i,{l_i}}} - \mathop {\max }\limits _{j \ne {l_i}} {t_{i,j}} \geqslant 1} \left\| {{T_{i,:}} - {R_{i,:}}{\text { }}} \right\| _F^2, \end{aligned}$$

(33)

where $R=AX$. According to Theorem 2 in [21], after solving problem (33) by solving for all columns of T separately, we can obtain the optimal solution ${{T_{i,:}}}$ of problem (32).