Alternative, uniformly expressive and more scalable interfaces for collective communication in MPI

Abstract

In both the regular and the irregular MPI (Message-Passing Interface) collective communication and reduction interfaces there is a correspondence between the argument lists and certain MPI derived datatypes. As a means to address and alleviate well-known memory and performance scalability problems in the irregular (or vector) collective interface definitions of MPI we propose to push this correspondence to its natural limit, and replace the interfaces of the MPI collectives with a different set of interfaces that specify all data sizes and displacements solely by means of derived datatypes. This reduces the number of collective (communication and reduction) interfaces from 16 to 10, significantly generalizes the operations, unifies regular and irregular collective interfaces, makes it possible to decouple certain algorithmic decisions from the collective operation, and moves the interface scalability issue from the collective interfaces to the MPI derived datatypes. To complete the proposal we discuss the memory scalability of the derived datatypes and suggest a number of alternative datatypes for MPI, some of which should be of independent interest. A running example illustrates the benefits of this alternative set of collective interfaces. Implementation issues are discussed showing that an implementation can be undertaken within any reasonable MPI library implementation.

Introduction

It is well acknowledged that the MPI interfaces for the so called irregular (or vector) collective operations [7, Chapter 5] (like MPI_Alltoallw, to mention the most extreme example) suffer from scalability problems in that multiple argument lists of size p (the number of MPI processes in the communicator on which the operation is applied) have to be supplied in the calls, often in application contexts where a considerable amount of regularity and/or redundancy exists, e.g., a large fraction of count values of zero or semi-regular displacement values [2]. At the same time, despite their general appearance, the irregular collective interfaces pose certain limitations as to which data placements can be specified, limiting their use in situations where a derived datatype description of the data placements would be a natural choice. Such an observation previously motivated the generalization of the irregular MPI 1.0 MPI_Alltoallv operation to an additional MPI_Alltoallw operation in MPI 2.0 which in addition to counts and displacements takes lists of possibly different send and receive datatypes as arguments. Curiously, the same generalization was not undertaken for the other irregular collectives, one argument being that MPI_Alltoallw allows the expression of any possible exchange pattern. Although correct, this argument misses the point that for the more specialized operations much more efficient algorithmic realizations exist.

In the MPI design of the collective interfaces there is an intended, clear correspondence between the specification of send and receive buffers and certain, derived (or user-defined) datatypes [7, Chapter 4]. For instance, the data received in the regular MPI_Gather operation are stored as a contiguous block of data blocks, with the block from process i stored as the ith block. Each block is equivalent to a contiguous datatype of the given basetype, and the whole set of blocks to be received can again be described as a contiguous datatype of p blocks. For the irregular MPI_Gatherv operation, the received data are placed as described by an indexed type with p indices of contiguous blocks of the same basetype. Exactly this correspondence with the indexed type is limiting the expressivity of MPI_Gatherv. For instance, interleaving (or tiling) the data received from the p processes is tedious because the indices of the indexed type define displacements in units of the extent of the basetype. To achieve the interleaving effect, a new, resized type with extent equal to the tile size would have to be created. Gathering data of different, unrelated basetypes is not possible at all with this collective operation.

This paper proposes to push the correspondence between the arguments to the collective operations and the MPI derived datatypes to its natural limit. We propose to replace all specifications of collective communication buffers that in MPI are in terms of base addresses, counts, displacements and basetypes with only a base address and a datatype. All structure, including the repetition count of the data to be sent and received will be encapsulated in the datatype. This has the advantage of unifying the interfaces for regular and irregular collective operations, resulting in only one interface for each communication pattern and thereby reducing the overall number of collective interfaces in MPI from currently 16 to 10 (excluding MPI_Barrier). It gives full generality and similar expressivity to all collective operations, preserves the spirit of MPI of explicit distinction between regular and irregular communication patterns and can thus be implemented with the same, small overhead. Specifically, it delegates the interface scalability issues from the collective interfaces to the datatype interfaces (Section 3). To exploit this further we introduce a number of new, more space economic datatype constructors for expressing semi-regular data distributions that we believe often occur in applications. We argue that these new datatypes do not introduce any new, fundamental implementation problems, and can all be implemented efficiently by known methods as in, e.g., [9] and many subsequent papers. This alleviates the interface scalability issue for the irregular MPI collective interfaces (Section 4). The decoupling of data description from the collective interfaces makes it possible to perform certain algorithmic optimizations (finding regularities, compressing index and count arrays, etc.) in advance, and potentially amortize the costs of such analyses over a number of collective calls. This provides an alternative approach to persistent collective operations. It should be pointed out that the proposal does not change the communication patterns that can be expressed by the MPI collectives. It is orthogonal to proposals for sparse collective communication for MPI [5], that address other issues in collective communication. This and other issues and alternatives are discussed at length (Section 5).

By an example we demonstrate expressivity and scalability limitations in the current collective interfaces. The example furthermore illustrates where the derived datatype mechanism not only serves expressive purposes but can actually delegate real work to the MPI library (Section 2). Three similar applications were recently discussed and benchmarked in comparison to implementations not using MPI datatypes in [1], and in some cases better performance was achieved with the datatype solution. An elegant example of using a derived datatype together with MPI_Alltoall communication to accomplish a Fast Fourier Transform (FFT) was recently given in [4], illustrating again the considerable, often unexploited potential of the MPI derived datatype mechanism. Earlier work applied derived datatypes to the NAS parallel benchmarks and showed performance and convenience benefits [6].