TeraScope: distributed visual data mining of terascale data sets over photonic networks

Abstract

TeraScope is a framework and a suite of tools for interactively browsing and visualizing large terascale data sets. Unique to TeraScope is its utilization of the Optiputer paradigm to treat distributed computer clusters as a single giant computer, where the dedicated optical networks that connect the clusters serve as the computer’s system bus. TeraScope explores one aspect of the Optiputer architecture by employing a distributed pool of memory, called LambdaRAM, that serves as a massive data cache for supporting parallel data mining and visualization algorithms.

Introduction

“Where the telescope ends, the microscope begins. Which of the two has the grander view?”—Victor Hugo

Areas of research such as Geoscience, Astronomy, and High Energy Physics are routinely producing terabytes, and soon, petabytes of data from direct data gathering, data post processing, and simulations. Algorithmic detection of hidden patterns within these large data sets has been the focus of data mining [6]. Visualization used in this context (often referred to as Visual Data Mining) has been valuable as a way to verify the detected patterns; and in particular, for when algorithmic specifications of the patterns are difficult to derive [2], [3], [8], [9], [10], [15]. In the latter case user-interfaces that allow one to interactively browse, query and visualize enormous data sets need to be developed [17].

The work described in this paper is motivated by several emerging trends. Firstly, scientific databases are becoming highly distributed. Secondly, the cost of high speed networking is increasing at a rate far exceeding Moore’s Law, network bandwidth is doubling every 8 months, whereas processors are doubling in speed every 18–24 months. This means that computers, rather than the networks are the bottleneck. Thirdly, there is an increasing need and potential, facilitated by these high speed networks, for scientists to publish terabyte data sets on the Web in a manner similar to the way most netizens can create Web pages, so that researchers can make new discoveries by combining data from previously disparate disciplines. For example by correlating data from the World Health Organization with data from the National Center for Atmospheric Research, one could potentially understand how weather patterns influence the spread of diseases.

The Optiputer is a National Science Foundation funded project intended to exploit these trends by interconnecting distributed storage, computation, and visualization resources using extremely high speed photonic networks [13]. The important difference between this and classical Grid computing is that in this new model, the optical networks serve as the system bus for a potentially planetary-scale computer; and compute clusters taken as a whole, serve as the peripherals in the computer. For example, a cluster of computers with high performance graphics cards would be thought of as a single giant graphics card in this context. In the Optiputer concept, we refer to compute clusters as LambdaNodes to denote the fact that they are connected by multiples of light paths (often referred to as Lambdas) in an optical network. Each computer in a LambdaNode is referred to as a nodule, and collections of LambdaNodes form a LambdaGrid.

TeraScope is an experimental visual data mining toolkit intended to take advantage of the Optiputer paradigm. This paper describes the prototype that was developed and demonstrated at the IGrid 2002 conference in Amsterdam (http://www.igrid2002.org). Furthermore, this paper describes LambdaRAM, a high performance cache, for the Optiputer.