Abstract
Recently, InfiniBand Architecture (IBA) has been proposed as the next generation interconnect for I/O and inter-process communication. The main idea behind this industry standard is to use a scalable switched fabric to design the next generation clusters and servers with high performance and scalability. This architecture provides various types of new mechanisms and services (such as multiple transport services, RDMA and atomic operations, multicast support, service levels, and virtual channels). These services are provided by components (such as queue pairs, completion queue, and virtual-to-physical address translations) and their attributes. Different implementation choices of IBA may lead to different design strategies for efficient implementation of higher level communication layer/libraries (such as Message Passing Interface (MPI), sockets, and distributed shared memory). It also has an impact on the performance of applications.
Currently there is no framework for evaluating different design choices and for obtaining insights about the design choices made in a particular implementation of IBA. In this paper we address these issues by proposing a new micro-benchmark suite (MIBA) to evaluate the InfiniBand architecture implementations. MIBA consists of several micro-benchmarks which are divided into two major categories: non-data transfer related micro-benchmarks and data transfer related micro-benchmarks. By using the new micro-benchmark suite, the performance of IBA implementations can be evaluated under different communication scenarios, and also with respect to the implementation of different components and attributes of IBA. We demonstrate the use of MIBA to evaluate the second generation IBA adapters from Mellanox Technologies.
This research is supported in part by Sandia National Laboratory’s contract #30505, Department of Energy’s Grant #DE-FC02-01ER25506, and National Science Foundation’s grants #EIA-9986052 and #CCR-0204429.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Mellanox Technologies, http://www.mellanox.com
Alexandrov, A., Ionescu, M.F., Schauser, K.E., Scheiman, C.: LogGP: Incorporating long messages into the LogP model for parallel computation. Journal of Parallel and Distributed Computing 44(1), 71–79 (1997)
Bailey, D.H., Barszcz, E., Barton, J.T., Browning, D.S., Carter, R.L., Dagum, D., Fatoohi, R.A., Frederickson, P.O., Lasinski, T.A., Schreiber, R.S., Simon, H.D., Venkatakrishnan, V., Weeratunga, S.K.: The NAS Parallel Benchmarks. The International Journal of Supercomputer Applications 5(3), 63–73 (1991)
Balaji, P., Shivam, P., Wyckoff, P., Panda, D.K.: High Performance User Level Sockets over Gigabit Ethernet. In: Cluster Computing (September 2002)
Banikazemi, M., Liu, J., Kutlug, S., Ramakrishna, A., Sadayappan, P., Sah, H., Panda, D.K.: Vibe: A micro-benchmark suite for evaluating virtual interface architecture implementations. In: Int’l Parallel and Distributed Processing Symposium (IPDPS) (April 2001)
Bell, C., Bonachea, D., Cote, Y., Duell, J., Hargrove, P., Husbands, P., Iancu, C., Welcome, M., Yelick, K.: An evaluation of current high-performance networks. In: International Parallel and Distributed Processing Symposium, IPDPS 2003 (2003)
Blumrich, M., Dubnicki, C., Felten, E.W., Li, K., Mesarina, M.R.: Virtual- Memory-Mapped Network Interfaces. IEEE Micro, 21–28 (February 1995)
Compaq, Intel, and Microsoft. VI Architecture Specification V1.0 (December 1997)
Dunning, D., Regnier, G., McAlpine, G., Cameron, D., Shubert, B., Berry, F., Merritt, A.M., Gronke, E., Dodd, C.: The Virtual Interface Architecture. IEEE Micro, 66–76 (March/April 1998)
InfiniBand Trade Association. InfiniBand Architecture Specification, Release 1.0, October 24 (2000)
Liu, J., Wu, J., Kinis, S.P., Buntinas, D., Yu, W., Chandrasekaran, B., Noronha, R., Wyckoff, P., Panda, D.K.: MPI over InfiniBand: Early Experiences. Technical Report, OSU-CISRC-10/02-TR25, Computer and Information Science department, the Ohio State University (January 2003)
Liu, J., Wu, J., Kinis, S.P., Wyckoff, P., Panda, D.K.: High Performance RDMA-Based MPI Implementation over InfiniBand. In: International Conference on Supercomputing (June 2003)
Noronha, R., Panda, D.K.: Implementing TreadMarks over GM on Myrinet: Challenges Design Experience and Performance Evaluation. In: Workshop on Communication Architecture for Clusters (CAC 2003), To be held in conjuction with IPDPS 2003 (April 2003)
Pakin, S., Lauria, M., Chien, A.: High Performance Messaging on Workstations: Illinois Fast Messages (FM). In: Proceedings of the Supercomputing (1995)
Rafael, H., Saavedra, R.: Stockton Gaines, and Michael J. Carlton. Micro benchmark analysis of the KSR1. In: Supercomputing, pp. 202–213 (1993)
Shah, G., Nieplocha, J., Mirza, J., Kim, C., Harrison, R., Govindaraju, R.K., Gildea, K., DiNicola, P., Bender, C.: Performance and experience with LAPI – a new high performance communication library for the ibm rs/6000 sp. In: International Parallel Processing Symposium (March 1998)
Shivam, P., Wyckoff, P., Panda, D.K.: EMP: zero-copy OS-bypass NIC-driven gigabit ethernet message passing. In: Proceedings of SC 2001, Denver, CO (November 2001)
Shivam, P., Wyckoff, P., Panda, D.K.: Can user level protocols take advantage of multi-CPU NICs? In: Proceedings of IPDPS 2002, Ft. Lauderdale, FL (April 2002)
von Eicken, T., Basu, A., Buch, V., Vogels, W.: U-Net: A User-level Network Interface for Parallel and Distributed Computing. In: ACM Symposium on Operating Systems Principles (1995)
von Eicken, T., Culler, D.E., Goldstein, S.C., Schauser, K.E.: Active Messages: A Mechanism for Integrated Communication and Computation. In: International Symposium on Computer Architecture, pp. 256–266 (1992)
Welsh, M., Basu, A., von Eicken, T.: Incorporating Memory Management into User-Level Network Interfaces. In: Proceedings of Hot Interconnects V (August 1997)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chandrasekaran, B., Wyckoff, P., Panda, D.K. (2003). MIBA: A Micro-Benchmark Suite for Evaluating InfiniBand Architecture Implementations. In: Kemper, P., Sanders, W.H. (eds) Computer Performance Evaluation. Modelling Techniques and Tools. TOOLS 2003. Lecture Notes in Computer Science, vol 2794. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-45232-4_3
Download citation
DOI: https://doi.org/10.1007/978-3-540-45232-4_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40814-7
Online ISBN: 978-3-540-45232-4
eBook Packages: Springer Book Archive