Skip to main content
SpringerLink
Log in

Integrating heterogeneous network monitoring data

  • Published:
Telecommunication Systems Aims and scope Submit manuscript

Abstract

In this paper, we investigate the integration of heterogeneous network monitoring data. Specifically, we will synchronize and integrate flow-level records, exemplified by Cisco NetFlow, and packet-level traces, exemplified by NLANR PMA. The integration can facilitate cross-validation and complementary utility. However, finding the correspondences of timestamps/flows/packets between the PMA and Netflow is non-trivial, because they have different levels of granularity, different sampling strategy, different time sources, and different IP address masking. To integrate heterogeneous monitoring data, we first synchronize their timestamps, and then match their masked IP addresses. Our key observation is that although the IP addresses are masked, some other header fields can be exploited to match different types of monitoring data. In order to reduce the search space and the processing overhead, we have adopted a top-down approach to limit the search scope, and iterative algorithms to reduce the matching errors step by step.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Duffield, N., & Lund, C. (2003). Predicting resource and estimation accuracy in an IP flow measurement collection intrastate. In ACM internet measurement conference, October 2003.

  2. Duffield, N. G., Lund, C., & Thorup, M. (2002). Properties and prediction of flow statistics from sampled packet streams. In ACM internet measurement workshop, November 2002.

  3. Duffield, N., Lund, C., & Thorup, M. (2003). Estimating flow distributions from sampled flow statistics. In ACM SIGCOMM, August 2003.

  4. Estan, C., Keys, K., Moore, D., & Varghese, G. (2002). Building a better NetFlow. In ACM SIGCOMM, August 2002.

  5. Estan, C., Keys, K., Moore, D., & Varghese, G. (2002). New directions in traffic measurement and accounting. In ACM SIGCOMM, August 2002.

  6. Estan, C., Savage, S., & Varghese, G. (2003). Automatically inferring patterns of resource consumption in network traffic. In ACM SIGCOMM, August 2003.

  7. Kumar, A., Sung, M., Xu, J., & Wang, J. (2004). Data streaming algorithms for efficient and accurate estimation of flow distribution. In ACM SIGMETRICS, June 2004.

  8. Micheel, J., Donnelly, S., & Graham, I. (2001). Precision timestamping of network packets. In ACM internet measurement workshop, November 2001.

  9. McGregor, A., Hall, M., Lorier, P., & Brunskill, J. (2004). Flow clustering using machine learning techniques. In Passive and active measurement workshop, April 2004.

  10. Moon, S. B., Skelly, P., & Towsley, D. (1999). Estimation and removal of clock skew from network delay measurement. In IEEE INFOCOM, March 1999.

  11. Mori, T., Uchida, M., & Kasahara, R., et al. (2004). Identifying elephant flows through periodically sampled packets. In ACM internet measurement conference, October 2004.

  12. Cisco NetFlow. http://www.cisco.com/warp/public/732/Tech/nmp/NetFlow/.

  13. Paxson, V. (1998). On calibrating measurements of packet transit times. In ACM SIGMETRICS, June 1998.

  14. Rupp, A., Dreger, H., Fedlmann, A., & Sommer, R. (2004). Packet trace manipulation framework for test labs. In ACM internet measurement conference, October 2004.

  15. Sommer, R., & Feldmann, A. (2002). NetFlow: information loss or win. In Internet measurement workshop, November 2002.

  16. Veitch, D., Babu, S., & Pasztor, A. (2004). Robust synchronization of software clocks across the internet. In ACM internet measurement conference, October 2004.

  17. Zhang, Y., Singh, S., Sen, S., Duffield, N., & Lund, C. (2004). Online identification of hierarchical heavy hitters: algorithms, evaluation, and applications. In Internet measurement conference, October 2004.

Download references

Author information

Authors and Affiliations

  1. Juniper Networks, 1220 North Mathilda Ave, Sunnyvale, CA, 94089, USA

    Chi Zhang

  2. Microsoft, Redmond, WA, 98052, USA

    Bin Liu

  3. Fulcrum Microsystems, Calabasas, CA, 91302, USA

    Xun Su

  4. Center for Internet Augmented Research and Assessment, Florida International University, Miami, FL, 33199, USA

    Heidi Alvarez & Julio Ibarra

Authors
  1. Chi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xun Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Heidi Alvarez
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Julio Ibarra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chi Zhang.

Additional information

This work is sponsored by the University Research Program of Cisco Systems Inc from 09/01/04 to 08/31/06.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, C., Liu, B., Su, X. et al. Integrating heterogeneous network monitoring data. Telecommun Syst 37, 71–84 (2008). https://doi.org/10.1007/s11235-008-9073-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11235-008-9073-5

Keywords

Search

Navigation