Efficient Hardware Multicast Group Management for Multiple MPI Communicators over InfiniBand

Abstract

MPI provides a set of primitives that allow processes to dynamically create communicators on the fly. This set of primitives can be exploited by the applications where only a certain group of processes need to participate at any given time. Also, these primitives play an important role in the context of dynamic process management of MPI-2. Special attention has to be paid in creating MPI communicators with InfiniBand’s hardware multicast support as it involves the high overhead of interaction between the application and an external multicast management entity. In this paper, we propose different design alternatives of efficiently creating the communicators dynamically. The basic idea behind the schemes proposed is to remove most of the overhead of the hardware multicast group construction from the critical path of the application. Our results indicate that by using Multicast Pool and Lazy approaches of group construction proposed in the paper, we can significantly reduce the overhead by a factor of as much as 4.8 and 3.9, repectively, compared to the Basic approach.

This research is supported in part by Department of Energy’s Grant #DE-FC02-01ER25506; National Science Foundation’s grants #CCR-0204429, #CCR-0311542 and #CNS-0403342; grants from Intel and Mellanox; and equipment donations from Intel, Mellanox, AMD and Apple.