Performance feature identification by comparative trace analysis

Abstract

This work introduces a method for instrumenting applications, producing execution traces, and visualizing multiple trace instances to identify performance features. The approach provides information on the execution behavior of each process within a parallel application and allows differences across processes to be readily identified. Traces events are directly related to the source code and call-chain that produced them. This allows the identification of the causes of events to be easily obtained. The approach is particularly suited to aid in the understanding of the achieved performance from an application centric viewpoint. In particular, it can be used to assist in the formation of analytical performance models which can be a time-consuming task for large complex applications. The approach is one of human-effort reduction: focus the interest of the performance specialist on performance critical code regions rather than automating the performance model formulation process completely. A supporting implementation analyses trace files from different runs of an application to determine the relative performance characteristics for critical regions of code and communication functions.

Introduction

It is generally accepted that the peak-performance of high performance systems is the result of a complex interplay between the hardware architecture, the communication system and the applied workload. Knowledge of the processor design, memory hierarchy, inter-processor and network system, and workload arrangement is necessary to understand the achievable performance.

As high-end computing facilities increase in scale and complexity, it is essential to consider the system performance throughout an architecture's life cycle: starting at the design stage where no system is available for measurement, through comparison of systems and procurement, to implementation, installation and verification, and finally to examine the effects of system updates over time.

A key approach that can be used at each stage of the life-cycle is to utilize detailed models that provide information on the expected performance of a workload, given a particular architectural configuration. Depending on the complexity of the model, an expectation of the achievable performance of the workload can be obtained with reasonable fidelity.

The accuracy of a model, and hence, its effectiveness lie in its ability to capture an application's performance characteristics. It is considered advantageous to parameterize a model in terms of system configuration and calculation behavior, as this allows for the exploration of the performance space without being specific to a particular ‘performance point’. Evaluation of anticipated scenarios such as the scaling effect of increasing the processor count, altering the workload size, or the impact of modifying the communication strategy can then be explored.

Performance models are widely used: from large-scale, tightly coupled systems through to dynamic and distributed Grid-based systems. For instance, performance modeling was used to: validate performance during the installation of ASCI Q at Los Alamos National Laboratory (LANL) [1] which ultimately lead to the system optimization resulting in a factor of two performance improvement [2]; compare the performance of large-scale systems such as the Earth Simulator [3]; and has been used in the procurement of many systems including ASCI purple (expected to be a 100Tflop machine). Performance models are also extensively applied in ‘Grid’ environments to consider service-orientated metrics in the provision of resource management services [4], and in the mapping of financial applications to available resources [5].

It is, however, generally acknowledged that the formation of a performance model is a complex task. It typically involves a thorough code analysis, inspection of important data structures, and analysis of profile and trace data. It can, therefore, be time-consuming to generate a detailed model given the large size of many scientific applications and the relative complexities of advanced data structures and highly optimized communication strategies. Historically, much of this work has been performed “by hand” using profilers and tracing libraries typically designed for fault analysis. As applications become more complex, the ability to facilitate the modeling process has become increasingly desirable.

Several semi-automated approaches have been proposed that aim to make the formation of a performance model a simpler task using ‘black-box’ techniques in which individual performance aspects are observed but not necessarily understood. Examples include modeling the scaling behavior of basic-block performance [6] and modeling the memory behavior of basic-blocks and extrapolating to other systems [7]. These approaches tend to be specific to a particular processor configuration and/or application problem size.

In this work, we consider an approach that aims to simplify the process of generating a performance model, but not to automate it entirely. The purpose is not to simplify the resultant performance model, or detract from the skill-set required by the modeler, but rather remove unnecessary steps during formulation. While the answer to this question lies, in part, with the experience of the performance-modeler; we believe that tools can be developed (or adapted) to help locate and focus on the performance critical regions. Such regions are typically those whose execution behavior changes when the system configuration or application input-data is varied. Should a region of code not change (in the performance sense) to such configuration and input variations, it may be possible to approximate the behavior by a single timing for a given architecture. An essential part of the performance modeler's skill is distinguishing between the elements that require parameterization and those where a single-time will suffice.

While there are a number of post-analysis and diagnosis tools that can assist with identifying performance constraints such as identifying bottlenecks [8] or visualizing communication patterns [9], many are aimed at resolving problems with the application rather than characterizing and understanding an application's performance behavior. This subtle difference in purpose differentiates this work from standard post-analysis tools. Rather than highlighting hotspots of communication in a particular application for example, this work might focus on the fact that communication in a particular code region is proportionally lower than that of a previous run of the application. In a sense, the focus is on identifying key differences and, in particular, idiosyncratic behaviors.

The approach described in this paper uses a combination of static and dynamic call-graph analysis to identity regions of code that are sensitive to data-set and scalability variations in order to considerably reduce the time-to-model. It provides a compact view of multiple executions of an application using color cues to draw attention to “areas of interest”. It can also generate graphical illustrations of differences between execution traces to rapidly identify code blocks that require attention. In addition, events can match communication patterns against a library of predefined templates that attempt to identify common parallelization strategies. The current implementation is a prototype: it is envisaged that other methods of visualization could be employed to summarize more detailed performance characteristics “at a glance”. Reducing the physical screen estate that represents key data could permit rapid large-scale analysis. Similar visualization techniques have been applied to code maintenance [10].

The paper is organized as follows: Section 2 describes an approach that leads to a performance model and identifies areas where tools can assist. Section 3 introduces a tracing tool that can create call-graphs for post-analysis using source-code instrumentation. In Section 4, methods that use multi-trace visualizations to highlight key application differences are described. Their use is illustrated with examples run on large-scale parallel codes. Section 5 summarizes the features of the approach. Conclusions and future work are discussed in Section 6.