Developing SPMD applications with load balancing

Abstract

The central contribution of this work is SAMBA ($\text{S}$ingle $\text{A}$pplication, $\text{M}$ultiple Load $\text{Ba}$lancing), a framework for the development of parallel SPMD (single program, multiple data) applications with load balancing. This framework models the structure and the characteristics common to different SPMD applications and supports their development. SAMBA also contains a library of load balancing algorithms. This environment allows the developer to focus on the specific problem at hand. Special emphasis is given to the identification of appropriate load balancing strategies for each application. Three different case studies were used to validate the functionality of the framework: matrix multiplication, numerical integration, and a genetic algorithm. These applications illustrate its ease of use and the relevance of load balancing. Their choice was oriented by the different load imbalance factors they present and by their different task creation mechanisms. The computational experiments reported for these case studies made possible the validation of SAMBA and the comparison, without additional reprogramming costs, of different load balancing strategies for each of them. The numerical results and the elapsed times measurements show the importance of using an appropriate load balancing algorithm and the associated reductions that can be achieved in the elapsed times. They also illustrate that the most suitable load balancing strategy may vary with the type of application and with the number of available processors. Besides the support to the development of SPMD applications, the facilities offered by SAMBA in terms of load balancing play also an important role in terms of the development of efficient parallel implementations.

Introduction

Parallel algorithms are designed exploring either functional or data parallelism. Associated with MIMD (multiple instruction, multiple data) architectures, data parallelism originated the single program, multiple data (SPMD) programming model [36]: the same program is executed on different processors, over distinct data sets. Under this model, each task is characterized by the data over which the common code is executed. The SPMD programming model has been widely used in parallel programming. Coding and debugging is usually simpler under this model than in arbitrary MIMD programs. Moreover, data decomposition is a natural approach for the design of parallel algorithms for many problems [26].

The performance of SPMD programs is strongly affected by dynamic load imbalance factors, which may include the lack of information about the processing requirements of each task, the dynamic creation of new tasks, or the variation of processor load external to the application. The use of suitable strategies is essential in overcoming these imbalance factors. It is hard to determine the most appropriate load balancing technique for each application, partly because of the large variety of load balancing algorithms that has been proposed in the literature [1], [13], [16], [25], [40], [41]. Effective load balancing strategies are essential for overcoming dynamic imbalance factors.

The main contribution of this work is SAMBA ($\text{S}$ingle $\text{A}$pplication, $\text{M}$ultiple Load $\text{Ba}$lancing), a framework which models the structure and the characteristics common to different SPMD applications and supports their development [31], [32], [33]. Special emphasis is given to the identification of appropriate load balancing strategies. The following tasks are made easier by this framework:

• Coding new SPMD applications with load balancing: given the routines for task creation and execution, an SPMD application with a load balancing strategy is automatically generated. This allows the programmer to focus on the specific problem at hand, reducing the programming effort.

• Identification of the most appropriate load balancing strategy for a specific application: the designer can easily generate different versions of the application for each of the load balancing strategies offered in SAMBA’s library. This allows different strategies to be tested and compared without additional programming cost.

• Design and evaluation of new load balancing algorithms: SAMBA offers a set of functions used in many load balancing algorithms. Also, the strategies offered by the load balancing library can be used as the basis for the development of new algorithms.

We also discuss a set of classification criteria for load balancing algorithms designed for SPMD applications. These criteria provide a basic terminology for describing different algorithms and establish a background for their comparison and classification, helping the programmer to understand the differences between them. The design of the load balancing library offered by SAMBA was guided by these classification criteria. By implementing algorithms with different structures, we have been able to check that the framework effectively represents the functionality of a wide range of load balancing algorithms.

The effectiveness of SAMBA as a development tool offering load balancing strategies was assessed in the context of three parallel applications. The first application computes the multiplication of two square matrices. The second is the numerical integration of a function using a method of quadrature. The third one is a parallel genetic algorithm for the optimization of oil fields. These applications were chosen due to the different imbalance factors which affect each of them.

SAMBA was implemented in C using the MPI library [27]. Preliminary experiments with SAMBA were reported in [32] on three different environments: an IBM SP2 system, a heterogeneous network of Sun workstations, and a cluster of IBM PCs. The computational experiments described in this paper were carried out on a cluster of 32 IBM PCs.

This paper is organized as follows. Related work is presented in the next section. Classification criteria for load balancing algorithms are described in Section 3. The SAMBA framework is presented in Section 4. Its load balancing library is described in Section 5. The computational experiments performed to validate the framework are reported in Section 6. Concluding remarks are made in Section 7.