Discriminant neighborhood embedding for classification

doi:10.1016/j.patcog.2006.05.011

Pattern Recognition

Volume 39, Issue 11, November 2006, Pages 2240-2243

https://doi.org/10.1016/j.patcog.2006.05.011 Get rights and content

Abstract

In this paper a novel subspace learning method called discriminant neighborhood embedding (DNE) is proposed for pattern classification. We suppose that multi-class data points in high-dimensional space tend to move due to local intra-class attraction or inter-class repulsion and the optimal embedding from the point of view of classification is discovered consequently. After being embedded into a low-dimensional subspace, data points in the same class form compact submanifod whereas the gaps between submanifolds corresponding to different classes become wider than before. Experiments on the UMIST and MNIST databases demonstrate the effectiveness of our method.

Introduction

Manifold learning plays an important role in many applications such as pattern representation and classification. Several nonlinear techniques [1], [2], [3] have recently been introduced to learn the intrinsic manifold embedded in the ambient space of high dimensionality. However, these nonlinear techniques yield maps that are defined only on the training data points and it remains unclear how to evaluate the maps on novel testing points. To address this problem, locality preserving projection (LPP) is proposed [4]. Specifically, LPP finds an embedding that preserves local information and may be simply extended to any new sample. Since LPP does not make use of class label information, it cannot perform well in classification. Local discriminant embedding (LDE) [5] incorporates the class information into the construction of embedding and derives the embedding for nearest-neighbor classification in a low-dimensional space. Nevertheless, distant points are not deemphasized efficiently by LDE and it may degrade the performance of classification. Furthermore, both LPP and LDE suffer from singularity of matrix when the number of training samples is much smaller than the dimensionality of each sample. To address this problem, LPP and LDE first project the data to a PCA subspace, which may lose some discriminative information for classification.

In this paper, a novel method called discriminant neighborhood embedding (DNE) is proposed. DNE is inspired by an intuition of dynamics theory. Multi-class data points in high-dimensional space are supposed to be pulled or pushed by discriminant neighbors to form an optimum embedding of low dimensionality for classification. There is no need to calculate the inverse matrix in DNE method, so the complication due to singularity of matrix can be avoided.

Section snippets

Discriminant neighborhood embedding

Suppose $N$ multi-class data ${x_{1}, x_{2}, \dots, x_{N}}$ are sampled from underlying manifold $M$ embedded in high-dimensional ambient space $R^{n}$ and any subset of data points in the same class is assumed to lie in a submanifold of $M$ . We seek an embedding characterized by intra-class compactness and inter-class separability. Assume that there is an interaction between each pair of data points in the ambient space. The larger the distance between two points is, the weaker the interaction becomes. Therefore each

Comparisons with LPP and LDE

LPP finds an embedding of low dimensionality that preserves locality regardless of the class of neighbors. That is, for multi-class points, LPP regards repulsions between inter-class neighbors as attractions by mistake, which incurs degradation of classification performance. As an example, insular point $C$ (see Fig. 1) and its inter-class neighbors will be pulled even more nearby via LPP. The same is true for the marginal points, e.g. Point $H$ in Fig. 1. To avoid this drawback of LPP, DNE divides

Experimental results

In this section two databases, namely the UMIST database of faces and the MNIST database of handwritten digits, are tested to compare the proposed DNE method with the state-of-art approaches: PCA, LPP, and LDE. All experiments employ the nearest-neighbor classifier.

The UMIST face database, available at http://images.ee.umist.ac.uk/danny/database.html, contains 575 images of 20 individuals. Each image was cropped and downsampled into $56 \times 46$ size. Fig. 2 shows several samples of one individual in

Acknowledgements

This work is supported in part by the National Science Foundation of China under Grant No. 60373020, 60402007, 60533100, and MoE R&D Foundation 104075.

References (5)

J.B. Tenenbaum et al.
A global geometric framework for nonlinear dimensionality reduction
Science
(2000)
S. Roweis et al.
Nonlinear dimensionality reduction by locally linear embedding
Science
(2000)

There are more references available in the full text version of this article.

Cited by (58)

Local dual-graph discriminant classifier for binary classification
2024, Neurocomputing
Graph-based methods mine the potential structural information of data by constructing various graphs that positively affect the classifiers when dealing with classification problems. However, traditional graph-based classifiers are the most common single-graph classifiers and minimize only intra-class compactness, where inter-class separability is replaced by other factors. To consider real inter-class separability, we introduce a novel local dual-graph structure that can fully mine the geometric distribution of data by simultaneously maximizing the inter-class separability and minimizing the intra-class compactness. This local dual-graph structure reflects the relationship between samples and their neighbors and hence avoids the negative impact of outliers on the construction of graphs. Furthermore, a novel classifier called the local dual-graph discriminant classifier (LDGDC) is proposed using a local dual-graph structure. Originally, LDGDC is designed to perform the following optimization: minimization of the 2-norm regularization of model coefficients and intra-class compactness, and maximization of the inter-class separability, which is a non-convex optimization problem. To facilitate the solution, we transform the original non-convex problem of LDGDC into a convex problem. Finally, experiments were conducted on several public datasets, and the results demonstrate the effectiveness and robustness of the proposed LDGDC.
Theoretical framework in graph embedding-based discriminant dimensionality reduction
2021, Signal Processing
Graph embedding-based discriminative dimensionality reduction has remained to be a popular research topic over the past few decades. The weight functions in adjacent graphs are the key to the performance of methods. In practice, the weight functions are always achieved experimentally. Thus far, the selection of effective weight functions has no any theoretical guidance. A theoretical framework considering hypothesis-margin is derived in this study to guide the selection of weight functions, whose truth is verified in a popular algorithm and a more effective supervised discriminant graph embedding-based dimensionality reduction method is introduced. Many experimental results demonstrate the truth of the proposed theoretical framework and the effectiveness of the introduced method. Importantly, the proposed framework can provide theoretical support for the selection of weight functions in the graph embedding-based dimensionality reduction.
Automated interpretation of biopsy images for the detection of celiac disease using a machine learning approach
2021, Computer Methods and Programs in Biomedicine
Celiac disease is an autoimmune disease occurring in about 1 in 100 people worldwide. Early diagnosis and efficient treatment are crucial in mitigating the complications that are associated with untreated celiac disease, such as intestinal lymphoma and malignancy, and the subsequent high morbidity. The current diagnostic methods using small intestinal biopsy histopathology, endoscopy, and video capsule endoscopy (VCE) involve manual interpretation of photomicrographs or images, which can be time-consuming and difficult, with inter-observer variability. In this paper, a machine learning technique was developed for the automation of biopsy image analysis to detect and classify villous atrophy based on modified Marsh scores. This is one of the first studies to employ conventional machine learning to automate the use of biopsy images for celiac disease detection and classification.
The Steerable Pyramid Transform (SPT) method was used to obtain sub bands from which various types of entropy and nonlinear features were computed. All extracted features were automatically classified into two-class and multi-class, using six classifiers.
An accuracy of 88.89%, was achieved for the classification of two-class villous abnormalities based on analysis of Hematoxylin and Eosin (H&E) stained biopsy images. Similarly, an accuracy of 82.92% was achieved for the two-class classification of red-green-blue (RGB) biopsy images. Also, an accuracy of 72% was achieved in the classification of multi-class biopsy images.
The results obtained are promising, and demonstrate the possibility of automating biopsy image interpretation using machine learning. This can assist pathologists in accelerating the diagnostic process without bias, resulting in greater accuracy, and ultimately, earlier access to treatment.
Supervised discriminative dimensionality reduction by learning multiple transformation operators
2021, Expert Systems with Applications
Citation Excerpt :
Locally Preserving Projection (LPP) is a popular example of graph embedding methods, preserving the local structure of data in the reduced subspace (He & Niyogi, 2004). While the original version of LPP is unsupervised, there are several supervised extensions of LPP (Wang, Ding, Hsu, & Yang, 2020; Zhang, Xue, Lu, & Guo, 2006). Double graphs-based discriminant projection (DGDP) is a recent similar work, which constructs two adjacent graphs instead of one.
Analyzing and learning from high dimensional data have always been challenging in machine learning tasks, causing serious computational complexities and poor learning performances. Supervised dimensionality reduction is a popular technique to address such challenges in supervised learning tasks, where data are accompanied with labels. Traditionally, such techniques mostly learn one single transformation to project data into a low-dimensional discriminative subspace. However, learning only one transformation for the whole data could be dominated by one or several classes, and the rest of classes receive less discrimination in the reduced space. That is to say, learning one transformation is insufficient to properly discriminate classes of data in the reduced space because they may have complex and completely dissimilar distributions. This insufficiency becomes even more serious if the number of classes increases, leading to poor discrimination and lessening the learning performance in the reduced space. To overcome this limitation, we propose a novel supervised dimensionality reduction method, which learns per-class transformations by optimizing a newly designed and efficient objective function. The proposed method captures more discriminative information from each single class of data compared to the case of one single transformation. Moreover, the proposed objective function enjoys several desirable properties: (1) maximizing margins between the transformed classes of data, (2) having a closed-form solution, (3) being easily kernelized in the case of nonlinear data,(4) preventing overfitting, and (5) ensuring the transformations are sparse in rows so that discriminative features are learned in the reduced space. Experimental results verify that the proposed method is superior to the related state-of-the-art methods and promising in generating discriminative embeddings.
Dimensionality reduction via preserving local information
2020, Future Generation Computer Systems
Citation Excerpt :
Many graph embedding algorithms are proposed in recent years, such as discriminant neighborhood embedding [22], exponential local discriminant embedding [23], orthogonal neighborhood preserving projections [24], locality sensitive discriminant analysis [25], marginal fisher analysis [26,27], local feature discriminant projection [28], double adjacency graphs-based discriminant neighborhood embedding [29] and discriminant neighborhood structure graphs [30]. Here, we analyze three typical algorithms, [22,26,27,29]. Zhang et al. [22] proposed the discriminant neighborhood embedding (DNE) supposing that multi-class samples in high-dimensional space tend to move due to local intra-class attraction or inter-class repulsion, and the optimal embedding from the sample of view of classification is discovered consequently.
Dimensionality reduction is an important technique for multimedia data in smart cities. As a significant type of multimedia data, images are being processed frequently using such technique. It is worth noting that finding neighbors is an integral part of graph embedding algorithms in dimensionality reduction. In this paper, we present an effective and efficient metric function, named local similarity preserving (LSP), which can preserve the similarity information of each sample to its homogeneous and heterogeneous neighbors. The proposed LSP function is helpful to enhance the discriminative capability of subsequent feature extractors. Based on LSP function, we also propose two novel algorithms, local similarity preserving discriminant (LSPD) algorithm, which can preserve the local similarity information and LSPD $^{+}$ algorithm, which can preserve the local similarity and geometric structure information, respectively. The experimental results on digit visualization and face recognition demonstrate the effectiveness and efficiency of the proposed algorithms.
Discriminative globality and locality preserving graph embedding for dimensionality reduction
2020, Expert Systems with Applications
Graph embedding in dimensionality reduction has attracted much attention in the high-dimensional data analysis. Graph construction in graph embedding plays an important role in the quality of dimensionality reduction. However, the discrimination information and the geometrical distributions of data samples are not fully exploited for discovering the essential geometrical and discriminant structures of data and strengthening the pattern discrimination in graph constructions of graph embedding. To overcome the limitations of graph constructions, in this article we propose a novel graph-based dimensionality reduction method entitled discriminative globality and locality preserving graph embedding (DGLPGE) by designing the informative globality and locality preserving graph constructions. In the constructed graphs, bidirectional weights of edges are newly defined by considering both the geometrical distributions of each point of edges and the class discrimination. Using the adjacent weights of graphs, we characterize the intra-class globality preserving scatter, the inter-class globality preserving scatter and the locality preserving scatter to formulate the objective function of DGLPGE in order to optimize the projection of dimensionality reduction. Extensive experiments demonstrate that the proposed DGLPGE often outperforms the state-of-the-art dimensionality reduction methods.

View all citing articles on Scopus

View full text