Raj Kettimuthu
Vijaya Natarajan
Valerio Pascucci
ABSTRACT
In preparation for the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report, the climate community will run the Coupled Model Intercomparison Project phase 5 (CMIP-5) experiments, which are designed to answer crucial questions about future regional climate change and the results of carbon feedback for different mitigation scenarios. The CMIP-5 experiments will generate petabytes of data that must be replicated seamlessly, reliably, and quickly to hundreds of research teams around the globe. As an end-to-end test of the technologies that will be used to perform this task, a multi-disciplinary team of researchers moved a small portion (10 TB) of the multimodel Coupled Model Intercomparison Project, Phase 3 data set used in the IPCC Fourth Assessment Report from three sources---the Argonne Leadership Computing Facility (ALCF), Lawrence Livermore National Laboratory (LLNL) and National Energy Research Scientific Computing Center (NERSC)---to the 2009 Supercomputing conference (SC09) show floor in Portland, Oregon, over circuits provided by DOE's ESnet. The team achieved a sustained data rate of 15 Gb/s on a 20 Gb/s network. More important, this effort provided critical feedback on how to deploy, tune, and monitor the middleware that will be used to replicate the upcoming petascale climate datasets. We report on obstacles overcome and the key lessons learned from this successful bandwidth challenge effort.
- }}http://www.ipcc.ch/index.htmGoogle Scholar
- }}Allcock, W., Bresnahan, J., Kettimuthu, R., Link, M., Dumitrescu, C., Raicu, I., and Foster, I. 2005. The Globus Striped GridFTP Framework and Server. In Proceedings of the 2005 ACM/IEEE Conference on Supercomputing (November 12--18, 2005). Google ScholarDigital Library
- }}http://www.es.net/OSCARS/Google Scholar
- }}https://computing.llnl.gov/resources/gdo/Google Scholar
- }}http://www.nersc.gov/nusers/systems/datatran/Google Scholar
- }}https://scinet.supercomputing.org/Google Scholar
- }}http://www.vacet.org/Google Scholar
- }}http://www.globus.org/toolkit/docs/5.0/5.0.0/data/gridftp/user/#globus-url-copyGoogle Scholar
- }}http://dev.globus.org/images/e/e1/SC09-globusDotOrg-datasheet.pdfGoogle Scholar
- }}http://www.cedps.net/Google Scholar
- }}https://sdm.lbl.gov/bdm/Google Scholar
- }}http://www.earthsystemgrid.org/Google Scholar
- }}B. Tierney and D. Gunter. "NetLogger: A Toolkit for Distributed System Tuning and Debugging". IFIP/IEEE Intl Symposium on Integrated Network Management Colorado Springs, CO, 2003Google Scholar
- }}http://staff.psc.edu/mathis/MTU/Google Scholar
- }}http://fasterdata.es.net/Google Scholar
- }}http://www.tcpdump.org/Google Scholar
- }}http://www.tcptrace.org/Google Scholar
- }}http://www.xplot.orgGoogle Scholar
Index Terms
- Lessons learned from moving earth system grid data sets over a 20 Gbps wide-area network
Recommendations
Grids and grid technologies for wide-area distributed computing
The last decade has seen a substantial increase in commodity computer and network performance, mainly as a result of faster hardware and more sophisticated software. Nevertheless, there are still problems, in the fields of science, engineering, and ...
Lessons learned through driving science applications in the PRAGMA grid
This paper describes the coordination, design and implementation of the PRAGMA Grid. Applications in genomics, quantum mechanics, climate simulation, organic chemistry and molecular simulation have driven the middleware requirements, and the PRAGMA Grid ...
Grid and P2P middleware for wide-area parallel processing
Grid computing emerged as a paradigm for high-performance computing and massive parallel processing. Currently, Grid systems have become an important paradigm for efficiently solving large-scale complex problems from many fields. On the other hand, P2P ...
Comments