Mining massive datasets by an unsupervised parallel clustering on a GRID: Novel algorithms and case study

Abstract

This paper proposes three novel parallel clustering algorithms based on the Kohonen’s SOM aiming at preserving the topology of the original dataset for a meaningful visualization of the results and for discovering associations between features of the dataset by topological operations over the clusters. In all these algorithms the data to be clustered are subdivided among the nodes of a GRID. In the first two algorithms each node executes an on-line SOM, whereas in the third algorithm the nodes execute a quasi-batch SOM called MANTRA. The algorithms differ on how the weights computed by the slave nodes are recombined by a master to launch the next epoch of the SOM in the nodes. A proof outline demonstrates the convergence of the proposed parallel SOMs and provides indications on how to select the learning rate to outperform both the sequential SOM and the parallel SOMs available in the literature. A case study dealing with bioinformatics is presented to illustrate that by our parallel SOM we may obtain meaningful clusters in massive data mining applications at a fraction of the time needed by the sequential SOM, and that the obtained classification supports a fruitful knowledge extraction from massive datasets.

Introduction

The main concern of clustering algorithms is to group in a reasonable time large amounts of data in meaningful classes that can be disjoint, overlapping or organized in some hierarchical fashion [1]. A “meaningful class” is a class characterized by a maximum similarity among its items and minimum similarity between its items and the ones belonging to other classes; the similarity metrics may vary with the application domain. Clustering analysis has been used in numerous fields ranging from machine learning, artificial intelligence, pattern recognition, to web mining, spatial database analysis, text mining and image segmentation [2]. Recently, clustering algorithms have been used extensively in life sciences such as genomics and proteomics (e.g., [3], [4]).

All the mentioned application fields have in common the exponentially increasing size of the number of records to be analyzed, a high number of features for each record of the dataset, and a need for interactivity to allow the users to visualize and analyze the dataset from different perspectives.

Currently, if one needs only to cluster a dataset efficiently, many scalable clustering techniques are available, such as BIRCH [5] or bEMADS [6], which are able to process massive datasets efficiently even on a normal PC. Alternatively, to save time, one could resort to the parallelization of the clustering technique more suitable for the problem at hand. For example, if one is interested in clustering large high density data, a parallelization of the original DBSCAN [7] or its recent improved version [8] could be used.

However, often the need is to cluster a massive dataset by preserving its original topology. This would allow an efficient visualization of the clusters and the extraction, by using topological operations, of cluster properties which can be referred to the original dataset too. In this case a parallelization of either the Self Organizing feature Maps (SOM) by Kohonen et al. [9], Kaski et al. [10] and Oja et al. [11], or the Generative Topographic Mapping (GTM) by Bishop and Svensen [12] would be appropriate, as pointed out in the study carried out by [13] on topographic maps and data visualization, where a thorough comparison between the SOM, the GTM based methods and some variants of the $K$-Means is presented. Another method that is able to cluster a dataset by preserving its topology is the Growing Neural Gas (GNG), that is able also to match the temporal distribution of the original dataset [14], but it is too computationally demanding.

Although the GTM and its extensions, as well as the extended $K$-Means, show a very low topographic error,¹ the SOM is significantly faster [15], thus it is a suitable target of parallelization efforts for dealing with massive datasets.

The Parallel SOMs (PSOMs) that have been proposed in the literature fall in two classes: (1) parallelization of the original SOM, known as on-line SOM, mainly implemented on ad-hoc hardware (e.g., [16]), and (2) parallelization of an approximation of the original SOM, i.e., the so called batch SOM, implemented on loosely distributed computing systems that less preserves the original dataset topology [17].

The paper presents a novel approach to the efficient parallelization of the on-line SOM on a loosely distributed computing system. This approach can be used not only to improve the on-line SOM processing time, but also to improve the performance of other SOM based techniques, namely: (1) the ESOM, that is an Extended version of the SOM [18], [19], to better preserve the topology of the original dataset and to solve complex optimization problems; (2) the Growing SOM (GSOM) to achieve the same topological accuracy of GTM methods, widely studied in this decade (e.g., [20], [21], [22]); and (3) the Relative Density SOM (ReDSOM) proposed recently by [23] to model high density datasets with variable topology, which may achieve the same performance of the GNG.

The proposed PSOMs will be compared with the original SOM, but not with the techniques that do not preserve the dataset’s topology (e.g., DBSCAN), neither with techniques that preserve topology but are much slower than the SOM (e.g., GTM). Moreover, we will show that the moderate differences between the classifications obtained by the original SOM and our PSOM influence little the associations between features one can derive by topological operations from the obtained clusters. This makes the method particularly suitable for data mining purposes.

In particular, the SOM algorithm to be parallelized is briefly described in Section 2. Section 3 reviews the literature on the parallel SOMs to point out the limits of the existing algorithms. A novel parallel clustering method that overcomes these limits is proposed with three variants in Section 4, where its suitability to parallelize other SOM based techniques featured by a very low topographic error (i.e., GSOM, and ReDSOM) is also pointed out. In all these variants the dataset is partitioned among the nodes of a GRID, or any loosely coupled computing system. Section 5 outlines the proof of the convergence of the proposed clustering method and gives some measures of its topological properties. The time performance is analyzed in Section 6, to demonstrate that all the three variants are scalable and converge with a precision of about 85%–90% to the same clusters obtainable by the sequential SOM. Also, Section 6 shows a knowledge discovery methodology based on the proposed PSOMs that is able to discover about 85% of the associations one can extract from the same methodology based on the sequential SOM. Finally, in Section 7, we first describe how the GRID environment is used to execute the proposed SOMs, then a bioinformatics case study is discussed to demonstrate the effectiveness of the PSOMs to discover features associations in a realistic case.