Nonlinear Dimension Reduction by Local Multidimensional Scaling

Abstract

We propose a neighbourhood-preserving method called LMB for generating a low-dimensional representation of the data points scattered on a nonlinear manifold embedded in high-dimensional Euclidean space. Starting from an exemplary data point, LMB locally applies the classical Multidimensional Scaling (MDS) algorithm on small patches of the manifold and iteratively spreads the dimension reduction process. Differs to most dimension reduction methods, LMB does not require an input for the reduced dimension, as LMB could determine a well-fit dimension for reduction in terms of the pairwise distances of the data points. We thoroughly compare the performance of LMB with state-of-the-art linear and nonlinear dimension reduction algorithms on both synthetic data and real-world data. Numerical experiments show that LMB efficiently and effectively preserves the neighbourhood and uncovers the latent embedded structure of the manifold. LMB also has a low complexity of $O(n^2)$ for n data points.

Appendix: A Brief Introduction of the MDS Algorithm

Let n be the size of the data and D be the matrix of Euclidean pairwise distances. MDS generates the coordinates X for all the data points with their center at the origin. Each column of X represents a data point. MDS first calculates the matrix $X^TX$ by Eq. (9). The diagonal entries of H are $1-\frac{1}{n}$ and the rests are $-\frac{1}{n}$.

$$\begin{aligned} X^TX=-\frac{1}{2}HD^2H. \end{aligned}$$

(9)

$X^TX$ is positive semi-definite, and can decomposed as Eq. (10), which leads to Eq. (11).

$$\begin{aligned} X^TX = V\sigma ^2V^T = V\sigma ^T\sigma V^T = (\sigma V^T)^T\sigma V^T. \end{aligned}$$

(10)

$$\begin{aligned} X = \sigma V^T. \end{aligned}$$

(11)

If the data points are distributed in an m-dimensional space, the first m diagonal entries of $\sigma$ are non-zero. Extracting the first d entries of $\sigma$ and the corresponding eigenvectors generates an approximation of the original data in a lower d-dimensional space. The approximation is very accurate if the first d entries are the most significant ones and the rests are close to zero. If MDS is applied on the k-nearest neighbourhoods on a d-dimensional manifold, $\sigma$ is a k+1 by k+1 matrix and the first d entries are significant while the rests are almost zero. However, If MDS is applied on all the data points of a manifold, $\sigma$ is a n by n matrix, and due to the global geometry of the manifold, the first m $(m>d)$ entries of $\sigma _{n\times n}$ are significant. Therefore, only the first d dimensions are not enough to well represent the data points, and MDS on the whole nonlinear manifold may result in an inaccurate approximation in the d-dimensional space.