Anukool Lakhina
Mark Crovella
ABSTRACT
Detecting and understanding anomalies in IP networks is an open and ill-defined problem. Toward this end, we have recently proposed the subspace method for anomaly diagnosis. In this paper we present the first large-scale exploration of the power of the subspace method when applied to flow traffic. An important aspect of this approach is that it fuses information from flow measurements taken throughout a network. We apply the subspace method to three different types of sampled flow traffic in a large academic network: multivariate timeseries of byte counts, packet counts, and IP-flow counts. We show that each traffic type brings into focus a different set of anomalies via the subspace method. We illustrate and classify the set of anomalies detected. We find that almost all of the anomalies detected represent events of interest to network operators. Furthermore, the anomalies span a remarkably wide spectrum of event types, including denial of service attacks (single-source and distributed), flash crowds, port scanning, downstream traffic engineering, high-rate flows, worm propagation, and network outage.
- Abilene Network Operations Center Weekly Reports. At http://www.abilene.iu.edu/routages.cgi.Google Scholar
- S. Agarwal, C.-N. Chuah, S. Bhattacharyya, and C. Diot. The Impact of BGP Dynamics on Intra-Domain Traffic. In ACM SIGMETRICS, New York, June 2004. Google ScholarDigital Library
- P. Barford, J. Kline, D. Plonka, and A. Ron. A Signal Analysis of Network Traffic Anomalies. In Internet Measurement Workshop, Marseille, November 2002. Google ScholarDigital Library
- Cisco NetFlow. At www.cisco.com/warp/public/732/Tech/netflow/.Google Scholar
- Deloader Worm Description. At http://www.f-secure.com/v-descs/deloader.shtml.Google Scholar
- N. Duffield, C. Lund, and M. Thorup. Estimating Flow Distributions from Sampled Flow Statistics. In ACM SIGCOMM, Karlsruhe, August 2003. Google ScholarDigital Library
- R. Dunia and S. J. Qin. Multi-dimensional Fault Diagnosis Using a Subspace Approach. In American Control Conference, 1997.Google Scholar
- A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, and F. True. Deriving Traffic Demands for Operational IP networks: Methodology and Experience. In IEEE/ACM Transactions on Neworking, pages 265--279, June 2001. Google ScholarDigital Library
- A. Hussain, J. Heidemann, and C. Papadopoulos. A Framework for Classifying Denial of Service Attacks. In ACM SIGCOMM, Karlsruhe, August 2003. Google ScholarDigital Library
- J. Jung and B. Krishnamurthy and M. Rabinovich. Flash Crowds and Denial of Service Attacks: Characterization and Implications for CDNs and Web Sites. In World Wide Web Conference, Hawaii, May 2002. Google ScholarDigital Library
- J. E. Jackson. A User's Guide to Principal Components. John Wiley, New York, NY, 1991.Google Scholar
- J. E. Jackson and G. S. Mudholkar. Control Procedures for Residuals Associated with Principal Component Analysis. Technometrics, pages 341--349, 1979.Google ScholarCross Ref
- Juniper Traffic Sampling. At www.juniper.net/techpubs/software/junos/junos60/swconfig60-policy/html/sampling overview.html.Google Scholar
- M.-S. Kim, H.-J. Kang, S.-C. Hung, S.-H. Chung, and J. W. Hong. A Flow-based Method for Abnormal Network Traffic Detection. In IEEE/IFIP Network Operations and Management Symposium, Seoul, April 2004.Google Scholar
- A. Lakhina, M. Crovella, and C. Diot. Characterization of Network-Wide Anomalies in Traffic Flows. Technical Report BUCS-2004-020, Boston University, 2004.Google ScholarDigital Library
- A. Lakhina, M. Crovella, and C. Diot. Diagnosing Network-Wide Traffic Anomalies. In ACM SIGCOMM, Portland, August 2004. Google ScholarDigital Library
- A. Lakhina, K. Papagiannaki, M. Crovella, C. Diot, E. D. Kolaczyk, and N. Taft. Structural Analysis of Network Traffic Flows. In ACM SIGMETRICS, New York, June 2004. Google ScholarDigital Library
- A. Markopoulou, G. Iannaccone, S. Bhattacharyya, C.-N. Chuah, and C. Diot. Characterization of Failures in an IP Backbone. In IEEE INFOCOM, Hong Kong, April 2004.Google ScholarCross Ref
- J. Mirkovic and P. Reiher. A Taxonomy of DDoS Attacks and Defense Mechanisms. ACM CCR, April 2004. Google ScholarDigital Library
- Pathdiag: Network Path Diagnostic Tools. At http://www.psc.edu/ web100/pathdiag/.Google Scholar
- S. Sarvotham, R. Riedi, and R. Baraniuk. Network Traffic Analysis and Modeling at the Connection Level. In Internet Measurement Workshop, San Francisco, November 2001.Google ScholarDigital Library
- SLAC Internet End-to-end Performance Monitoring (IEPM-BW project). At http://www-iepm.slac.stanford.edu/bw/.Google Scholar
- R. Teixeira, A. Shaikh, T. Griffin, and J. Rexford. Dynamics of Hot-Potato Routing in IP Networks. In ACM SIGMETRICS, New York, June 2004. Google ScholarDigital Library
- N. Weaver, V. Paxson, S. Staniford, and R. Cunningham. A Taxonomy of Computer Worms. In ACM CCS Workshop on Rapid Malcode (WORM), October 2003. Google ScholarDigital Library
- V. Yegneswaran, P. Barford, and J. Ullrich. Internet Intrusions: Global Characteristics and Prevalence. In ACM SIGMETRICS, San Diego, June 2003. Google ScholarDigital Library
Index Terms
- Characterization of network-wide anomalies in traffic flows
Recommendations
Diagnosing network-wide traffic anomalies
Anomalies are unusual and significant changes in a network's traffic levels, which can often span multiple links. Diagnosing anomalies is critical for both network operators and end users. It is a difficult problem because one must extract and interpret ...
Diagnosing network-wide traffic anomalies
SIGCOMM '04: Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communicationsAnomalies are unusual and significant changes in a network's traffic levels, which can often span multiple links. Diagnosing anomalies is critical for both network operators and end users. It is a difficult problem because one must extract and interpret ...
Network traffic analysis over clustering-based collective anomaly detection
AbstractDue to the ever-growing presence of network traffic, there has been a considerable amount of research on anomaly detection in network traffic by clustering. Most of them have not considered the problem that collective anomaly detection ...
Comments