Abstract
Storing and querying are two important issues that need to be addressed while designing an information retrieval system for a large and high-dimensional data set. In this work, we discuss about tackling such data, specifically about the nearest neighbour search and the efficient storage layout to store such data. The data set used in the current work has been taken from an online source called ZINC, a repository for drug like chemical structures. Processing a high dimensional data is a tough task hence dimensionality reduction should be employed. Here for dimensionality reduction is achieved through a filter-based feature selection method, based on correlation fractal dimension (CFD) discrimination measure, is used. The number of dimensions using the correlation fractal dimension are reduced from 58 to 7. To identify the nearest neighbours for a given chemical structure Tanimoto similarity coefficient is used with these reduced set of features. The nearest neighbours identified using the Tanimoto measure are stored in a storage layout known as modified inverted file. Nearest neighbours for a query can be retrieved back from the storage layout, with just one read operation from the data file thereby reducing the time for retrieval.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Cormen, T.H., Leiserson, C.E., Rivest, R.L.: Introduction to algorithms. The MIT Press, London (2000)
Fukunaga, K.: Introduction to Statistical Pattern Recognition. Academic Press Professional, Inc., San Diego (1990)
Pearson, K.: On lines and planes of closest t to systems of points in space. Philiosophical Magazine 2, 559–572 (1901)
Spearman, C.: General intelligence objectively determined and measured. American Journal of Psychology 15, 206–221 (1904)
Torgerson, W.S.: Multidimensional scaling I: Theory and method. Psychometrika 17, 401–419 (1952)
Nguyen, D.V., Rocke, D.M.: Tumor classication by partial least squares using microarray gene expression data. Bioinformatics 18, 39–50 (2002)
Maaten, L.V., Postma, E., Herik, J.V.: Dimensionality Reduction: A Comparative Review. Technical report: TiCC TR 2009-005 (2009)
Lejsek, H., Asmundsson, F.H., Jonsson, B.P., Amsaleg, L.: NV-Tree: An Efficient Disk-Based Index for Approximate Search in Very Large High-Dimensional Collections. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 869–883 (2009)
Lin, H.-Y., Huang, P.-W., Hsu, K.-H.: A new indexing method with high storage utilization and retrieval efficiency for large spatial databases. Information and Software Technology 49, 817–826 (2007)
ZINC- A free database for virtual screening, http://www.zinc.docking.org
Sankara Rao, A., Durga Bhavani, S., Sobha Rani, T., Bapi, R.S., Narahari Sastry, G.: Study of Diversity and Similarity of Large Chemical Databases Using Tanimoto Measure. In: Venugopal, K.R., Patnaik, L.M. (eds.) ICIP 2011. CCIS, vol. 157, pp. 40–50. Springer, Heidelberg (2011)
Bhavani, S.D., Rani, T.S., Bapi, R.S.: Feature selection using correlation fractal dimension: Issues and applications in binary classification problems. Applied Soft Computing 8, 555–563 (2008)
Garratt, A., Jackson, M., Burden, P., Wallis, J.: A survey of alternative designs for a search engine storage structure. Information and Software Technology 43, 661–677 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chandrasekhar, A.P., Rani, T.S. (2012). Storage and Retrieval of Large Data Sets: Dimensionality Reduction and Nearest Neighbour Search. In: Parashar, M., Kaushik, D., Rana, O.F., Samtaney, R., Yang, Y., Zomaya, A. (eds) Contemporary Computing. IC3 2012. Communications in Computer and Information Science, vol 306. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-32129-0_29
Download citation
DOI: https://doi.org/10.1007/978-3-642-32129-0_29
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-32128-3
Online ISBN: 978-3-642-32129-0
eBook Packages: Computer ScienceComputer Science (R0)