Abstract
This paper describes a novel approach to detect correlation from data streams in the context of MobiMine — an experimental mobile data mining system. It presents a brief description of the MobiMine and identifies the problem of detecting dependencies among stocks from incrementally observed financial data streams. This is a non-trivial problem since the stock-market data is inherently noisy and small incremental volumes of data makes the estimation process more vulnerable to noise. This paper presents EDS, a technique to estimate the correlation matrix from data streams by exploiting some properties of the distribution of eigenvalues for random matrices. It separates the “information” from the “noise” by comparing the eigen-spectrum generated from the observed data with that of random matrices. The comparison immediately leads to a decomposition of the covariance matrix into two matrices: one capturing the “noise” and the other capturing useful “information.” The paper also presents experimental results using Nasdaq 100 stock data.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
M. L. Mehta. Random Matrices. Academic Press, London, 2 edition, 1991.
H. Kargupta, H. Park, S. Pittie, L. Liu, D. Kushraj, and K. Sarkar. MobiMine: Monitoring the stock market from a PDA. ACM SIGKDD Explorations, 3:37–47, 2001.
A. Azoff. Neural Network Time Series Forecasting of Financial Markets. Wiley, New York, 1994.
N. Baba and M. Kozaki. An intelligent forecasting system of stock price using neural networks. In Proceedings IJCNN, Baltimore, Maryland, pages 652–657, Los Alamitos, 1992. IEEE Press.
S. Cheng. A neural network approach for forecasting and analyzing the pricevolume relationship in the taiwan stock market. Master’s thesis, National Jow-Tung University, Taiwan, R.O.C, 1994.
R. Kuo, L. Lee, and C. Lee. Intelligent stock market forecasting system through artificial neural networks and fuzzy delphi. In Proceedings of World Congress on Neural Networks, pages 345–350, San Deigo, 1996. INNS Press.
C. Lee. Intelligent stock market forecasting system through artificial neural networks and fuzzy delphi. Master’s thesis, Kaohsiung Polytechnic Institute, Taiwan, R.O.C, 1996.
J. Zirilli. Financial Prediction Using Neural Networks. International Thomson Computer Press, 1997.
J. Campbell, A. Lo, and A. MacKinley. The Econometrics of Financial Markets. Princeton University Press, USA, 1997.
G. Jang, F. Lsi, and T. Parng. Intelligent stock trading decision support system using dual adaptive-structure neural networks. Journal of Information Science Engineering, 9:271–297, 1993.
D. Jonsson. Some limit theorems for the eigenvalues of a sample covariance matrix. Journal of Multivariate Analysis, 12:1–38, 1982.
H. Kargupta, S. Ghosh, L. Ding, and K. Sivakumar. Random matrices and online principal component analysis from data streams. Submitted to The Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kargupta, H., Sivakumar, K., Ghosh, S. (2002). Dependency Detection in MobiMine and Random Matrices. In: Elomaa, T., Mannila, H., Toivonen, H. (eds) Principles of Data Mining and Knowledge Discovery. PKDD 2002. Lecture Notes in Computer Science, vol 2431. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45681-3_21
Download citation
DOI: https://doi.org/10.1007/3-540-45681-3_21
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44037-6
Online ISBN: 978-3-540-45681-0
eBook Packages: Springer Book Archive