Elsevier

Future Generation Computer Systems

Volume 89, December 2018, Pages 736-745
Future Generation Computer Systems

Calibers: A bandwidth calendaring paradigm for science workflows

https://doi.org/10.1016/j.future.2018.07.030Get rights and content

Highlights

  • Calibers impments a deadline-aware (SDN) flow orchestrator paradigm.

  • We leverage low-cost pacing and shaping at the edge of the network.

  • Mission-critical data can “sprint past” less critical data.

  • Calibers was tested on ESnet’s trans-continental SDN testbed.

  • Calibers is transport-layer protocol-agnostic.

Abstract

Many scientific workflows require large data transfers between distributed instrument facilities, storage and computing resources. To ensure that these resources are maximally utilized, R&E networks connecting these resources must ensure that an inherently unpredictable network behaves predictably. In practice, this amounts to the per-application over-provisioning of network resources in an attempt to guarantee that adequate throughput is provided to users. This often results in resource under-utilization over time. One promising solution is the use of deadlines and bandwidth calendaring. In this approach, “fair” resource allocation is replaced with deadline-based resource allocation. However, these approaches often suffer from issues in efficiently regulating resource allocation and failure modes. Therefore, our solution, Calibers, approaches bandwidth calendaring and deadline-awareness in a different way. Calibers uses shaping, metering, and pacing at the edge of the network and end-system to provide participating clients the ability to schedule bandwidth reservations without having to worry about network noise from non-participating clients. Calibers can also fail back to the fair resource allocation of underlying transport protocols if necessary. For example, if a non-participating flow somehow enters the core of the network, or a sudden network change causes the available bandwidth to be exceeded, the underlying transport protocol congestion avoidance implementation will be able to handle the congestion as it normally would. Furthermore, Calibers provides a novel simulation method and resource allocation algorithm.

In this paper, we present the prototype architecture for Calibers using a central controller with distributed agents to dynamically pace flows at the ingress of the network to meet deadlines. Using Globus/Grid-FTP, we experimentally demonstrate that pacing can be used to meet data transfer deadlines which cannot be achieved using TCP. Finally, we present dynamic flow pacing algorithms that maximize acceptance ratio of flows for which deadlines can be met while maximizing network utilization. Our results show that simple heuristics, optimizing locally on the most bottlenecked link, can perform almost as well as heuristics that attempt to optimize globally.

Introduction

Scientific analysis in experiments such as high-energy physics or climate modeling, usually involve extremely complex workflows to ensure successful and reliable results. These workflows include a number of tasks, involve multiple actors, software and infrastructures, that work together as a workflow from data generation to delivery. For example, in the Advanced Light source (ALS) data is generated from multiple detectors which is then collected on an NERSC supercomputing data center via high-speed network connections. It is imperative that the data is delivered in a timely manner, with minimum loss, such that further computations can be performed using supercomputing resources that have to be a priori reserved. In order that the supercomputing resources are maximally utilized, this requires the network service to allow deadlines for large data transfers.

There are two approaches to ensure that the data transfers can be made with predictable performance and within requested deadlines. One approach is to use advanced reservations of links, such as OSCARS or open NSA [1], that allow setting up circuits of specified capacities between routers. Advanced reservation schemes require additional time to setup circuits, are only associated with WAN border routers and are difficult to automate due to required user knowledge, network topology and request details. Furthermore, applications do not generate traffic all the time which leads to wasted reserved capacity.

The second approach is to run the network at low utilization and use standard TCP. New TCP protocols, such as TCP Hamilton [2] and BBR-TCP [3], can efficiently adapt to the bottleneck capacity and where multiple competing flows are involved, they equally split the bottleneck capacity. However, even with the new TCP algorithms, sustained bottlenecks lead to unpredictable throughput performance and difficulties in arbitrarily splitting bottlenecked bandwidths among competing flows. Finally, as the growth in data transfer volume out-paces the increase in the data link rates, running the network at low utilization is not cost effective [4].

To help accelerate the effort to run the network at high utilization and enable deadline aware data transfers, network automation through Software-Defined Networks (SDN) is being advanced to control network traffic depending on data demand. In principle, SDN allow individual switches to be managed and controlled following centralized traffic engineering principles [5]. Furthermore, SDN switches provide the ability to shape traffic at ingress of the network rapidly and in an on-demand fashion. These features in addition to TCP protocol, or the pacing algorithm at the source nodes [6], together provide the necessary tools to dynamically allocate bandwidth to flows for meeting deadlines while ensuring the network operates at high utilization. There has been network-utilization-focused work along these lines presented in [[7], [8]]. Simulation-based work on deadline-aware networking has also been carried out in [[9], [10]]. However, Calibers is not only capable of simulating the performance of these tools, it actually deploys a fully-functional, trans-continental deadline-aware SDN in a quasi-production network testbed, with some non-participating end-systems, to maximize deadline performance while also attempting to maximize link utilization.

This paper aims to implement a centralized traffic engineering approach and control distributed agents at the edge (ingress point of the network) to dynamically pace flow for meeting transfer deadlines, while achieving high network utilization. The dynamic pacing algorithm is able to analyze traffic patterns and follow a rolling horizon model to pace flows at appropriate rates to optimize network performance and meet deadlines. As a result, Calibers not only calenders flow, but also lays the foundation for future work where these capabilities can be coupled with advanced tools to control networks dynamically. Calibers aims to solve the problem of maximizing deadline performance and network utilization while minimizing flow rejection, given some set of network bottlenecks on a link-state routed network. Calibers is protocol-agnostic, and with the use of edge-pacing, is able to function on a network core, regardless of the presence of non-participating end-systems, whose traffic can be shaped at the network edge to relinquish bandwidth for deadline-critical flows.

Following are the main contributions of this work:

  • 1.

    We describe an architecture that implements bandwidth calendaring for scientific workflows. The architectures leverage SDN switches that can pace flows at the ingress point. The architecture implements a central controller with distributed agents at the edge of the network that monitor flow performance and implement dynamic flow pacing set by the controller.

  • 2.

    We present experimental results using Globus and GridFTP that show the importance of pacing in achieving deadline aware data transfer service. We compare our results with TCP Hamilton results.

  • 3.

    We propose different heuristic algorithms based on combining two orthogonal principles - 1) local vs global optimization and 2) Shortest Job First vs Longest Job First (LJF). We perform a preliminary performance comparison of these algorithms with respect to a performance metric efficacy that is defined as the difference between reject rate and network utilization. Our results show that simple heuristics, that optimize locally on the most bottlenecked link can perform almost as well as heuristics that attempt to optimize globally.

The remainder of this paper is organized as follows. In Section 2, we discuss the motivation of our work, specifically the importance of deadline aware data transfers in scientific workflows. We also discuss the importance of pacing and traffic shaping in deadline-aware traffic flow-scheduling. In Section 3, we present the architecture of Calibers in a software defined network. In Section 5 we present experimental results on a preliminary prototype Calibers architecture to demonstrate the effectiveness in meeting deadlines compared standard TCP. In Section 6 we present work on a dynamic flow pacing algorithm and present preliminary simulation results. In Section 4, we present the related work followed by conclusions and future work in Section 8.

Section snippets

Motivation

It is often the case that a large data transfer is inherently deadline-aware. For example, an HPC user may want to ensure that data is present at an HPC site before conducting their experiments. However, with the unpredictability of network resource utilization, this can pose a problem. Latency variation coupled with TCP’s typical “sawtooth” behavior can lead to a lack of predictability in meeting deadlines. Furthermore, even when TCP achieves pareto-optimality, this behavior may not always be

Architecture

Workflow orchestrators provision resources across the network, with assumptions that it does not introduce performance penalties. Calibers is an experimental network service, targeted to higher level resource orchestrators. It focuses on optimizing network resources such that each data flows (i.e. file transfers) performs at least at the minimum average rate over the transfer duration. This allows Calibers to provide deadline delivery guarantees.

The experimental software platform designed in

Related work

The over-arching goal of this work is to deliver deadline aware data transfers as a network service, while ensuring high network utilization. We leveraged SDN with the ability to perform dynamic traffic pacing at the network edge. There are a number of recent studies with similar goals. In the following paragraphs, we review the related work and point out the key differences from our work.

There has been a number of prior studies on flow pacing [[15], [16], [17]]. Broadly speaking, flow pacing

Experimental results

The experimental setup in Fig. 2 is based on ESnet’s SDN Testbed, a high-speed Wide-Area Network (WAN) SDN-ready testbed spanning two continents. The backbone data rates are guaranteed and setup via dedicated OSCARS circuits.

The testbed closely resembles ESnet’s high-speed production network in both hardware and topology, as it is an overlay of the ESnet production WAN [19]. Using ProxMox, a Linux container management system, we define three senders Amsterdam (AMST), New York (AOFY) and Denver

Scheduling algorithm for dynamic pacing

The notations used are shown in Table 1. The objective of the scheduler is to decrease the number of rejected data transfer requests while increasing the network utilization. This can be achieved by minimizing the sum of the completion time of all flows. This will push the completion time of all flows to the left (considering a time line), which increases the link utilization and frees up resources to accommodate future requests. The objective function is given by minfiUtficsubject tofiliFRf

Simulation analysis

Flow-level simulation was conducted to evaluate the performance of the four schedulers: (i) local-SJF, (ii) local-LJF, (iii) global-SJF, and (iv) global-LJF. The simulator was written using Python and for each simulation setup, 10 runs were executed where in each run 30k requests were generated.

Conclusions and future work

Calibers has demonstrated, in an ideal situation of a controlled environment, that TCP congestion avoidance algorithms, while performing well at maximizing network utilization, it cannot provide the desired behavior for workflow orchestration. In particular, TCP relies on network characteristic, such as RTT, packet retransmission, pace flows and ignore flow needs. As result some flows may go faster than they need, while others may go slower than they should, such to meet deadlines and maximize

Acknowledgments

This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research. This research was also supported by NSF grant CNS 1528087.

Fatma Alali is a Ph.D. candidate from the University of Virginia in Computer Engineering. Among her research interests are Software Defined Networks, enabling Virtual Circuits (VC) services, InfiniBand networks, and cloud computing. She holds a M.S. in Computer Engineering from University of Virginia and B.S. from Kuwait University.

References (22)

  • SalsanoS. et al.

    Information centric networking over sdn and openflow: Architectural aspects and experiments on the ofelia testbed

    Int. Jour. Comp. and Tele. Net. Archive

    (2013)
  • ESnet, OSCARS: On-demand secure circuits and advance reservation system...
  • D. Leith, R. Shorten, H-tcp: Tcp congestion control for high bandwidth-delay product paths, draft-leith-tcp-htcp-06...
  • CardwellN. et al.

    BBR: Congestion-based congestion control

    Queue

    (2016)
  • DOE, DOE network 2025: network research problems and challenges for doe scientists workshop,...
  • JainS. et al.

    B4: Experience with a globally-deployed software defined wan

    SIGCOMM Comput. Commun. Rev.

    (2013)
  • GettysJ. et al.

    Bufferbloat: Dark buffers in the internet

    Queue

    (2011)
  • KandulaS. et al.

    Calendaring for wide area networks

  • ZhangH. et al.

    Guaranteeing deadlines for inter-data center transfers

    IEEE/ACM Trans. Netw.

    (2017)
  • KINOSHITAK. et al.

    Joint bandwidth scheduling and routing method for large file transfer with time constraint and its implementation

    IEICE Trans. Commun.

    (2018)
  • AiharaM. et al.

    Joint bandwidth scheduling and routing method for large file transfer with time constraint

  • Cited by (2)

    Fatma Alali is a Ph.D. candidate from the University of Virginia in Computer Engineering. Among her research interests are Software Defined Networks, enabling Virtual Circuits (VC) services, InfiniBand networks, and cloud computing. She holds a M.S. in Computer Engineering from University of Virginia and B.S. from Kuwait University.

    Nathan Hanford is a Ph.D. candidate and graduate student researcher for the Graduate Group in Computer Science at the University of California, Davis. His research interests involve leveraging commodity end-system hardware over very high-speed interconnects and networks. To this end, he is interested in system-level software and lightweight middleware which enable meaningful end-system and node-level awareness. He is also interested in the HPC and cloud-computing applications of commodity hardware, and its implications for SDN. He has previously worked as a summer student in ESnet’s Advanced Network Technologies Group, where he conducted research on Data Transfer Nodes for moving large amounts of scientific data. He holds a B.S.E. in Computer Science and Engineering from the University of Connecticut.

    Eric Pouyoul is a Senior System Engineer in ESnet’s Advanced Network Technologies Group at Lawrence Berkeley National Laboratory (LBNL). His interests include all aspects of high performance big data movement, networking, hardware, software and distributed systems. He has been ESnet lead for designing Data Transfer Nodes (DTN) as defined in the Science DMZ architecture as well as ESnet’s work in Software Defined Networking (OpenFlow). Mr Pouyoul joined ESnet in 2009 and his 25 years prior experience includes real-time operating system, supercomputing and distributed systems.

    Raj Kettimuthu is a Computer Scientist in the Mathematics and Computer Science Division at Argonne National Laboratory, and a Senior Fellow in the Computation Institute at The University of Chicago and Argonne National Laboratory. He received the B.E. degree from Anna University, Chennai, India, and an M.S. and Ph. D. from the Ohio State University, all in Computer Science and Engineering. His research is focussed on high-speed transfer of large-scale data, software defined networking, rapid execution of data-intensive science workflows, parallel job scheduling and large-scale data analysis. He has co-authored more than 80 articles in the above-mentioned areas. He is a recipient of R&D 100 award. He is a senior member of both IEEE and ACM.

    Mariam Kiran is a staff scientist at LBNL sharing positions between Scientific Data Management Research Group and ESnet. Kiran’s research focuses on learning and decentralized optimization of system architectures and algorithms for high performance computing, underlying networks and Cloud infrastructures. She has been exploring various platforms such as HPC grids, GPUs, Cloud and SDN-related technologies. Her work involves optimization of QoS, performance using parallelization algorithms and software engineering principles to solve complex data intensive problems such as large-scale complex simulations. Over the years, she has been working with biologists, economists, social scientists, building tools and performing optimization of architectures for multiple problems in their domain.

    Ben Mack-Crane is a Principal Architect at Corsa Technology focusing on high performance applications of pure SDN using open protocols. Ben has experience with automated control across a broad spectrum of network technologies, from optical to packet and L0 through L7. He has been involved in behavioral specification and modeling of a wide variety of network elements over his career including packet switches and routers, transport systems, cable telephony, flexible multiplexers, and telephone switches. His current focus is on SDN and virtualization. Ben has been instrumental in numerous standards development efforts, including work in the ONF, MEF, IEEE, ITU, IETF, OIF, and ATIS. In addition to directing the Specification Area, Ben is currently leading the ongoing development and evolution of the OpenFlow protocol in the Open Datapath Working Group.

    Brian L. Tierney is a retired Staff Scientist at Lawrence Berkeley National Laboratory. His research interests include high-performance networking and network protocols; distributed system performance monitoring and analysis; network tuning issues; and the application of distributed computing to problems in science and engineering. He has been the PI for several DOE research projects in network and Grid monitoring systems for data intensive distributed computing. He was the Principal Designer of the Distributed Parallel Storage System (DPSS), where many of the ideas for GridFTP originated. He also designed the first version of the NetLogger Toolkit, and worked on the Bro Intrusion Detection System. He was co-chair of the 2nd International Workshop on Protocols for Long Distance Networks (PFLDnet) in 2004. He holds an M.S. in Computer Science from San Francisco State University, and a B.A. in Physics from the University of Iowa.

    Yatish Kumar leads Corsa’s technical vision as well as supporting SDN OpenFlow direction in the broader industry as ONF Area Director for all published specifications. He also sits on the ONF Chipmaker’s Advisory Board. Yatish has more than 23 years of networking industry experience. Prior to Corsa he was involved in a number of successful startups, including Catena Networks, which was acquired by Ciena in 2004. At Catena he led the design team responsible for developing the industry’s lowest power ADSL and POTS codecs which allowed Catena to become the leader in ADSL retrofits in all major RBOC accounts. Prior to Catena, Yatish started his career at Nortel where he contributed to and managed the development of a number of mixed signal semiconductor projects including designs for ADSL, POTS, CDMA, Cable Modems and handsets. He holds patents in DSP architectures, and data compression and has authored papers on high level synthesis, and embedded processor design as well as contributing to the development of ITU 992.1, ANSI T1.413 and Telcordia GR909 standards. But all this pales compared to the adventure of SDN.

    Dipak Ghosal is a Professor at the University of California, Davis. His research interests include high-speed networks, wireless networks, vehicular ad hoc networks, parallel and distributed systems, timing channels, and the performance evaluation of computer and communication systems. He holds a Ph.D. in Computer Science from the University of Louisiana, Lafayette.

    View full text