Qi Zhao
Abstract
Estimation of traffic matrices, which provide critical input for network capacity planning and traffic engineering, has recently been recognized as an important research problem. Most of the previous approaches infer traffic matrix from either SNMP link loads or sampled NetFlow records. In this work, we design novel inference techniques that, by statistically correlating SNMP link loads and sampled NetFlow records, allow for much more accurate estimation of traffic matrices than obtainable from either information source alone, even when sampled NetFlow records are available at only a subset of ingress. Our techniques are practically important and useful since both SNMP and NetFlow are now widely supported by vendors and deployed in most of the operational IP networks. More importantly, this research leads us to a new insight that SNMP link loads and sampled NetFlow records can serve as "error correction codes" to each other. This insight helps us to solve a challenging open problem in traffic matrix estimation, "How to deal with dirty data (SNMP and NetFlow measurement errors due to hardware/software/transmission problems)?" We design techniques that, by comparing notes between the above two information sources, identify and remove dirty data, and therefore allow for accurate estimation of the traffic matrices with the cleaned dat.We conducted experiments on real measurement data obtained from a large tier-1 ISP backbone network. We show that, when full deployment of NetFlow is not available, our algorithm can improve estimation accuracy significantly even with a small fraction of NetFlow data. More importantly, we show that dirty data can contaminate a traffic matrix, and identifying and removing them can reduce errors in traffic matrix estimation by up to an order of magnitude. Routing changes is another a key factor that affects estimation accuracy. We show that using them as the a priori, the traffic matrices can be estimated much more accurately than those omitting the routing change. To the best of our knowledge, this work is the first to offer a comprehensive solution which fully takes advantage of using multiple readily available data sources. Our results provide valuable insights on the effectiveness of combining flow measurement and link load measurement.
- S. Bhattacharyya, C. Diot, J. Jetcheva, and N. Taft. Geographical and temporal characteristics of inter-pop flows: View from a single pop. European Transactions on Telecommunications, 2000.Google Scholar
- J. Cao, D. Davis, S. Vander Wiel, and B. Yu. Time-varying network tomography:router link data. Journal of American Statistics Association, pages 1063--1075, 2000.Google Scholar
- D.L. Donoho. For most large underdetermined systems of equations, the minimal l1-norm near solution approximates the sparsest near-solution. In http://www-stat.stanford.edu/~donoho/Reports/, 2004.Google Scholar
- N. Duffield and C. Lund. Predicting resource usage and estimation accuracy in an ip flow measurement collection infrastructure. In Proc. ACM SIGCOMM IMC, October 2003. Google ScholarDigital Library
- A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, and F. True. Deriving traffic demands for operational IP networks: Methodology and experience. IEEE transaction on Networking, June 2001. Google ScholarDigital Library
- A. Gunnar, M. Johansson, and T. Telkamp. Traffic matrix estimation on a large ip backbone-- a comparison on real data. In Proc. USENIX/ACM SIGCOMM IMC, October 2004. Google ScholarDigital Library
- S.M. Kay. Fundamentals of Statistical Signal Processing: Estimation Theory. Prentice Hall, 1993. Google ScholarDigital Library
- A. Lakhina, K. Papagiannaki, and M. Crovella. Diagnosing network-wide anomalies. In Proc. ACM SIGCOMM, August 2004. Google ScholarDigital Library
- A. Medina, N. Taft, K. Salamatian, S. Bhattacharyya, and C. Diot. Traffic matrix estimation:existing techniques and new directions. In Proc. ACM SIGCOMM, August 2002. Google ScholarDigital Library
- White paper-netflow services and applications. http://www.cisco.com/warp/public/cc/pd/iosw/ioft/ neflct/tech/napps_wp.htm.Google Scholar
- A. Nucci, R. Cruz, N. Taft, and C. Diot. Design of igp link weight changes for estimation of traffic matrices. In Proc. IEEE INFOCOM, March 2004.Google ScholarCross Ref
- K. Papagiannaki, N. Taft, and A. Lakhina. A distributed approach to measure traffic matrices. In Proc. ACM/SIGCOMM IMC, October 2004. Google ScholarDigital Library
- A. Shaikh and A. Greenberg. Ospf monitoring: Architecture, design and deployment experience. In Proc. USENIX NSDI, 2004. Google ScholarDigital Library
- A. Soule, A. Lakhina, N. Taft, and K. Papagiannaki. Traffic matrices: Balancing measurements, inference and modeling. In Proc. ACM SIGMETRICS, August 2005. Google ScholarDigital Library
- A. Soule, A. Nucci, R. Cruz, E. Leonardi, and N. Taft. How to identify and estimate the largest traffic matrix elements in a dynamic environment. In Proc. ACM SIGMETRICS, June 2004. Google ScholarDigital Library
- Y. Vardi. Internet tomography: estimating source-destination traffic intensities from link data. Journal of American Statistics Association, pages 365--377, 1996.Google Scholar
- J. Wu, Z. Mao, J. Rexford, and J. Wang. Finding a Needle in a Haystack: Pinpointing Significant BGP Routing Changes in an IP Network. In Proc. USENIX NSDI, 2005. Google ScholarDigital Library
- Y. Zhang, Z. Ge, A. Greeenberg, and M. Roughan. Network anomography. In Proc. USENIX/ACM SIGCOMM IMC, Oct 2005. Google ScholarDigital Library
- Y. Zhang, M. Roughan, N. Duffield, and A. Greenberg. Fast accurate computation of large-scale ip traffic matrices from link loads. In Proc. ACM SIGMETRICS, June 2003. Google ScholarDigital Library
- Y. Zhang, M. Roughan, C. Lund, and D. Donoho. An information-theoretic approach to traffic matrix estimation. In Proc. ACM SIGCOMM, August 2003. Google ScholarDigital Library
- Q. Zhao, Z. Ge, J. Wang, and J. Xu. Robust traffic matrix estimation with imperfect information: Making use of multiple data sources. In Technical Report, April 2006.Google ScholarDigital Library
- Q. Zhao, A. Kumar, J. Wang, and J. Xu. Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices. In Proc. ACM SIGMETRICS, June 2005. Google ScholarDigital Library
Index Terms
- Robust traffic matrix estimation with imperfect information: making use of multiple data sources
Recommendations
Robust traffic matrix estimation with imperfect information: making use of multiple data sources
SIGMETRICS '06/Performance '06: Proceedings of the joint international conference on Measurement and modeling of computer systemsEstimation of traffic matrices, which provide critical input for network capacity planning and traffic engineering, has recently been recognized as an important research problem. Most of the previous approaches infer traffic matrix from either SNMP link ...
Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices
SIGMETRICS '05: Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systemsThe traffic volume between origin/destination (OD) pairs in a network, known as traffic matrix, is essential for efficient network provisioning and traffic engineering. Existing approaches of estimating the traffic matrix, based on statistical inference ...
Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices
Performance evaluation reviewThe traffic volume between origin/destination (OD) pairs in a network, known as traffic matrix, is essential for efficient network provisioning and traffic engineering. Existing approaches of estimating the traffic matrix, based on statistical inference ...
Comments