Abstract
Large cloud service providers have invested in increasingly larger datacenters to house the computing infrastructure required to support their services. Accordingly, researchers and industry practitioners alike have focused a great deal of effort designing network fabrics to efficiently interconnect and manage the traffic within these datacenters in performant yet efficient fashions. Unfortunately, datacenter operators are generally reticent to share the actual requirements of their applications, making it challenging to evaluate the practicality of any particular design.
Moreover, the limited large-scale workload information available in the literature has, for better or worse, heretofore largely been provided by a single datacenter operator whose use cases may not be widespread. In this work, we report upon the network traffic observed in some of Facebook's datacenters. While Facebook operates a number of traditional datacenter services like Hadoop, its core Web service and supporting cache infrastructure exhibit a number of behaviors that contrast with those reported in the literature. We report on the contrasting locality, stability, and predictability of network traffic in Facebook's datacenters, and comment on their implications for network architecture, traffic engineering, and switch design.
Supplemental Material
- An open network operating system. http://onosproject.org.Google Scholar
- Scribe (archived). https://github.com/facebookarchive/scribe.Google Scholar
- L. Abraham, J. Allen, O. Barykin, V. Borkar, B. Chopra, C. Gerea, D. Merl, J. Metzler, D. Reiss, S. Subramanian, J. L. Wiener, and O. Zed. Scuba: Diving into data at Facebook. Proc. VLDB Endow., 6(11):1057--1067, Aug. 2013. Google ScholarDigital Library
- M. Al-Fares, A. Loukissas, and A. Vahdat. A scalable, commodity, data center network architecture. In Proc. ACM SIGCOMM, Aug. 2008. Google ScholarDigital Library
- M. Al-Fares, S. Radhakrishnan, B. Raghavan, N. Huang, and A. Vahdat. Hedera: Dynamic flow scheduling for data center networks. In Proc. USENIX NSDI, Apr. 2010. Google ScholarDigital Library
- A. Alameldeen, M. Martin, C. Mauer, K. Moore, X. Min, M. Hill, D. Wood, and D. Sorin. Simulating a $2M commercial server on a $2K PC. IEEE Computer, 36(2):50--57, Feb. 2003. Google ScholarDigital Library
- M. Alizadeh, T. Edsall, S. Dharmapurikar, R. Vaidyanathan, K. Chu, A. Fingerhut, V. T. Lam, F. Matus, R. Pan, N. Yadav, and G. Varghese. Conga: Distributed congestion-aware load balancing for datacenters. In Proc. ACM SIGCOMM, Aug. 2014. Google ScholarDigital Library
- M. Alizadeh, A. Greenberg, D. A. Maltz, J. Padhye, P. Patel, B. Prabhakar, S. Sengupta, and M. Sridharan. Data center TCP (DCTCP). In Proc. ACM SIGCOMM, Aug. 2010. Google ScholarDigital Library
- A. Andreyev. Introducing data center fabric, the next-generation Facebook data center network. https://code.facebook.com/posts/360346274145943, 2014.Google Scholar
- B. Atikoglu, Y. Xu, E. Frachtenberg, S. Jiang, and M. Paleczny. Workload analysis of a large-scale key-value store. In Proc. ACM SIGMETRICS/Performance, June 2012. Google ScholarDigital Library
- L. A. Barroso, J. Clidaras, and U. Hölzle. The Datacenter as a Computer:An Introduction to the Design of Warehouse-Scale Machines. Morgan & Claypool, 2nd edition, 2013. Google ScholarDigital Library
- T. Benson, A. Akella, and D. A. Maltz. Network traffic characteristics of data centers in the wild. In Proc. ACM IMC, 2010. Google ScholarDigital Library
- T. Benson, A. Anand, A. Akella, and M. Zhang. Understanding data center traffic charachteristics. In Proc. ACM SIGCOMM WREN, Aug. 2009. Google ScholarDigital Library
- T. Benson, A. Anand, A. Akella, and M. Zhang. MicroTE: Fine grained traffic engineering for data centers. In Proc. ACM CoNEXT, Dec. 2011. Google ScholarDigital Library
- N. Bronson, Z. Amsden, G. Cabrera, P. Chakka, P. Dimov, H. Ding, J. Ferris, A. Giardullo, S. Kulkarni, H. Li, M. Marchukov, D. Petrov, L. Puzar, Y. J. Song, and V. Venkataramani. TAO: Facebook's distributed data store for the social graph. In Proc. USENIX ATC, June 2013. Google ScholarDigital Library
- M. Chowdhury, M. Zaharia, J. Ma, M. I. Jordan, and I. Stoica. Managing data transfers in computer clusters with orchestra. In Proceedings of the ACM SIGCOMM 2011 Conference, SIGCOMM '11, pages 98--109, New York, NY, USA, 2011. ACM. Google ScholarDigital Library
- C. Delimitrou, S. Sankar, A. Kansal, and C. Kozyrakis. ECHO: Recreating network traffic maps for datacenters with tens of thousands of servers. In Proc. IEEE International Symposium on Workload Characterization, Nov. 2012. Google ScholarDigital Library
- D. Ersoz, M. S. Yousif, and C. R. Das. Characterizing network traffic in a cluster-based, multi-tier data center. In Proc. IEEE International Conference on Distributed Computing Systems, June 2007. Google ScholarDigital Library
- N. Farrington and A. Andreyev. Facebook's data center network architecture. In Proc. IEEE Optical Interconnects, May 2013.Google ScholarCross Ref
- N. Farrington, G. Porter, S. Radhakrishnan, H. Bazzaz, V. Subramanya, Y. Fainman, G. Papen, and A. Vahdat. Helios: A hybrid electrical/optical switch architecture for modular data centers. In Proc. ACM SIGCOMM, Aug. 2010. Google ScholarDigital Library
- A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. A. Maltz, P. Patel, and S. Sengupta. VL2: A scalable and flexible data center network. In Proc. ACM SIGCOMM, Aug. 2009. Google ScholarDigital Library
- N. Gude, T. Koponen, J. Pettit, B. Pfaff, M. Casado, N. McKeown, and S. Shenker. NOX: Towards an operating system for networks. SIGCOMM CCR, 38(3), July 2008. Google ScholarDigital Library
- C. Guo, G. Lu, D. Li, H. Wu, X. Zhang, Y. Shi, C. Tian, Y. Zhang, and S. Lu. BCube: A high performance, server-centric network architecture for modular data centers. In Proc. ACM SIGCOMM, Aug. 2009. Google ScholarDigital Library
- V. Jalaparti, P. Bodik, S. Kandula, I. Menache, M. Rybalkin, and C. Yan. Speeding up distributed request-response workflows. In Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, SIGCOMM '13, pages 219--230, New York, NY, USA, 2013. ACM. Google ScholarDigital Library
- S. Kandula, J. Padhye, and P. Bahl. Flyways to de-congest data center networks. In Proc. ACM HotNets, Oct. 2009.Google Scholar
- S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken. The nature of data center traffic: Measurements & analysiss. In Proc. ACM IMC, Nov. 2009. Google ScholarDigital Library
- R. Kapoor, A. C. Snoeren, G. M. Voelker, and G. Porter. Bullet trains: A study of NIC burst behavior at microsecond timescales. In Proc. ACM CoNEXT, Dec. 2013. Google ScholarDigital Library
- T. Koponen, M. Casado, N. Gude, J. Stribling, L. Poutievski, M. Zhu, R. Ramanathan, Y. Iwata, H. Inoue, T. Hama, and S. Shenker. Onix: A distributed control platform for large-scale production networks. In Proc. USENIX OSDI, 2010. Google ScholarDigital Library
- A. Likhtarov, R. Nishtala, R. McElroy, H. Fugal, A. Grynenko, and V. Venkataramani. Introducing mcrouter: A memcached protocol router for scaling memcached deployments. https://code.facebook.com/posts/296442737213493, Sept. 2014.Google Scholar
- H. Liu, F. Lu, A. Forencich, R. Kapoor, M. Tewari, G. M. Voelker, G. Papen, A. C. Snoeren, and G. Porter. Circuit switching under the radar with REACToR. In Proc. USENIX NSDI, Apr. 2014. Google ScholarDigital Library
- R. Mack. Building timeline: Scaling up to hold your life story. https://www.facebook.com/note.php?note_id=10150468255628920, Jan. 2012.Google Scholar
- B. Pfaff, J. Pettit, T. Koponen, K. Amidon, M. Casado, and S. Shenker. Extending networking into the virtualization layer. In Proc. ACM HotNets, 2009.Google Scholar
- L. Popa, S. Ratnasamy, G. Iannaccone, A. Krishnamurthy, and I. Stoica. A cost comparison of datacenter network architectures. In Proc. ACM CoNEXT, Dec. 2010. Google ScholarDigital Library
- R. Sherwood, G. Gibb, K.-K. Yap, G. Appenzeller, M. Casado, N. McKeown, and G. Parulkar. Can the production network be the testbed? In Proc. USENIX OSDI, 2010. Google ScholarDigital Library
- A. Simpkins. Facebook open switching system (fboss) and wedge in the open. https://code.facebook.com/posts/843620439027582/facebook-open-switching-system-fboss-and-wedge-in-the-open/, 2015.Google Scholar
- A. Singla, C.-Y. Hong, L. Popa, and P. B. Godfrey. Jellyfish: Networking data centers randomly. In Proc. USENIX NSDI, Apr. 2012. Google ScholarDigital Library
- D. Sommermann and A. Frindell. Introducing Proxygen, Facebook's C++ HTTP framework. https://code.facebook.com/posts/1503205539947302, 2014.Google Scholar
- A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Antony, H. Liu, and R. Murthy. Hive -- a petabyte scale data warehouse using Hadoop. In Proc. IEEE ICDE, Mar. 2010.Google ScholarCross Ref
- G. Wang, D. G. Andersen, M. Kaminsky, K. Papagiannaki, T. S. E. Ng, M. Kozuch, and M. Ryan. c-Through: Part-time optics in data centers. In Proc. ACM SIGCOMM, Aug. 2010. Google ScholarDigital Library
- X. Zhou, Z. Zhang, Y. Zhu, Y. Li, S. Kumar, A. Vahdat, B. Y. Zhao, and H. Zheng. Mirror mirror on the ceiling: Flexible wireless links for data centers. In Proc. ACM SIGCOMM, Aug. 2012. Google ScholarDigital Library
Index Terms
- Inside the Social Network's (Datacenter) Network
Recommendations
Inside the Social Network's (Datacenter) Network
SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data CommunicationLarge cloud service providers have invested in increasingly larger datacenters to house the computing infrastructure required to support their services. Accordingly, researchers and industry practitioners alike have focused a great deal of effort ...
Virtio network paravirtualization driver
One of the techniques used to improve I/O performance of virtual machines is paravirtualization. Paravirtualized devices are intended to reduce the performance overhead on full virtualization where all hardware devices are emulated. The interface of a ...
Towards Seamless Cross-Vendor Inter-Datacenter Network Policy Migration
ICDCSW '13: Proceedings of the 2013 IEEE 33rd International Conference on Distributed Computing Systems WorkshopsIt is common that many enterprises tend to operate several datacenters applying virtualization technologies from different vendors. However, for virtual machine migration across different vendors, currently there is no feasible method to transfer the ...
Comments