2: The Adaptable Input Output System. A framework for high-performance data management

We present ADIOS 2, the latest version of the Adaptable Input Output (I/O) System. ADIOS 2 addresses scientific data management needs ranging


Motivation and significance
As supercomputers approach the exascale era [1], several hardware architectures have emerged [2,3] to counteract the cadence slowdown of Moore's law [4,5]. Due to the ability to run larger and more complex simulations, data input output (I/O) operations are anticipated to become a larger bottleneck than in current petascale systems [6,7]. To circumvent I/O bottlenecks, vendors and leadership facilities are proposing new high-bandwidth/low-latency memory, communication, and storage architectures [1,8]. Implementations include direct-memory access through Infiniband [9], Omni-Path [10], and NVLink [11], as well as intermediate storage using non-volatile random-access memory (NVRAM) [12]. As a result, there is a strong need to study each new paradigm in order to achieve optimal I/O performance without having to pay the cost of rewriting codebases for each platform [7].
Large-scale scientific data management is challenging during the data generation, transfer, storage, and processing stages. As illustrated in Fig. 1, data lifecycles require efficient data movement across different transport media: files, wide-area-networks (WAN) and memory staging areas. Several software tools exist to handle each aspect of the data lifecycle. For example, the HDF5 [13] and NetCDF [14] libraries have been widely used for storage and organization of large datasets. Their parallel implementations, Parallel HDF5 [13] and PNetCDF [15], expose their functionality to parallel computing environments based on the standard message passing interface, MPI [16]. At a lower level, the MPI-2 standard introduces the MPI I/O API for raw bytes file manipulation in parallel applications, a popular implementation is ROMIO [17]. On the other hand, interprocess/intraprocess network transport mechanisms are provided by well-established implementations: the MPI standard [16], ZeroMQ [18], OpenSH-MEM [19]. Hence, there are a plethora of interface options and communication models for each particular mechanism of data transport.
The present work introduces ADIOS 2: the Adaptable Input Output System version 2, a framework designed to serve the scientific community in their data management needs at several scales: from laptops, desktops, cloud services to large data producers in supercomputing applications. ADIOS 2 expands the performance legacy of the earlier versions of ADIOS that provide scalable parallel I/O as documented in previous works [20][21][22][23]. ADIOS 2 provides a unified application programming interface (API) with a level of abstraction focusing on how data is produced and consumed in scientific applications to reduce the cost of integrating different data transport technologies. Applications using ADIOS 2 do not need to dramatically modify their source code to evaluate performance trade-offs when moving their data through different transport media.
A flexible and portable framework is required due to the dynamic architectural changes shaping the path to exascale computing [2,3]. As such, the internal architecture of ADIOS 2 focuses on component reusability and extensibility for seamless integration of novel data management algorithms. ADIOS 2 is written in C++11 [24] to take advantage of the portable functionality added to the language, e.g. threads and algorithms, thus reducing reliance on platform-specific implementations. ADIOS 2 development adopts modern software engineering practices such as unit testing, continuous integration, and documentation to make the final product accessible to the scientific community.
ADIOS 2 is part of the United States Department of Energy (DOE) Exascale Computing Project (ECP) software technology stack for data and visualization [1,2].

Software description
When designing the ADIOS 2 software infrastructure three levels of interaction APIs were identified: 1. Public Low-Level API: large-scale HPC simulations codes, workflows, and ecosystem components 2. Public High-Level API: analysis and visualization using data science high-level languages frameworks 3. Private APIs: ADIOS 2 library internal components.
The design philosophy targets specific aspects to better serve interactions with each API level. The rationale is listed as follows, Public Low-level API: • Unified API: to reduce the learning curve for different data transport strategies  ADIOS 2 public interfaces are implemented using the Pointer to Implementation (PImpl) idiom [25]. ADIOS 2 provides a stable public API, as of version 2.5.0, at two levels: • Low-Level API: designed for HPC applications, provides greater level of control and granularity. Available in C++11, Fortran, C, and Python.
• High-Level API: designed for simple data analysis tasks, resembles nativate language I/O APIs for a flat learning curve. Available in C++11, Python, and Matlab. A schematic representation is provided in Fig. 2 for the lowlevel API components and their relationship from an application's point of view. ADIOS 2 uses a ''factory pattern'' [26] in which the ADIOS component is at the top-level in the factory hierarchy. ADIOS is the only component ''owned'' by the application, all other components refer to an element inside a container in its corresponding ADIOS factory. Applications can create as many ADIOS factory components as required. Table 1 provides a summary description for the role of each component in I/O workflow composition using ADIOS 2.
ADIOS 2 also provides high-level APIs in C++11, Python and Matlab, that resemble their native language implementations for file manipulation for easy integration with data analysis ecosystems available in those languages. Users will find these APIs suitable for simple tasks in which performance is not critical, e.g. quick data analysis and testing.
Examples for the public low-level and high-level APIs can be found in Section 3, Listings 1 and 2, respectively.

Private API architecture
The main goal of the ADIOS 2 internal architecture is to provide a set of extensible, reusable, and tested toolkit components for developers. The private internal architecture of ADIOS 2 adopts the Open Systems Interconnection (OSI) [33] standard. This layered abstraction model delineates the scope and functionality of each class component in the overall hierarchy of the object-oriented architecture. As shown in Fig. 3, each toolkit component can be mapped to the OSI software layers, 7 through 4.
At the heart of ADIOS 2 lies the concept of the abstract ''Engine''. Engines execute the I/O heavy tasks and are conceived as workflows tackling specific application areas as illustrated in Table 2. Fig. 4 illustrates the extendable object-oriented nature of  [27], SZ [28], MGARD [29]; lossless: PNG [30], bzip2 [31], Blosc [32] Callback Functions the Engine, layers 7 through 4, and the Transport, layer 4, classes. They provide unified interfaces for multiple raw bytes movement backends tackling different media in the lower hardware layers of the OSI model. Design choices favoring reusability allows Engine designers to have a set of available components to study their performance trade-offs when prototyping or extending an Engine. As an example, the BP serializer and deserializer components, OSI layer 6 in Fig. 3 (BP3 and BP4), are used in two staging engines: InSituMPI and SST, and by default in the BPFile engine, for their  metadata indexing, buffer aggregation across MPI processes, and data compression capabilities.

Software functionalities
ADIOS 2 has two primary focus areas in terms of data I/O transports:

Parallel file I/O 2. Parallel intra/interprocess data staging
The functionalities of ADIOS 2 can be mapped directly to a category of use-cases that each engine addresses. Table 2 summarizes these functionalities for currently available engines.
In addition, each engine provides a set of parameters to finetune performance for each application I/O pattern. For example, the current default engine (BPFile) allows for setting performance parameters: number of steps to buffer, number of generated subfiles, and frequency of collective metadata indexing.

Illustrative examples
The following examples illustrate usage of the low-level and high-level APIs. Listing 1 illustrates how ADIOS 2 C++11 low-level API is integrated with a generic HPC simulation. The example presents a sequence of interactions of the ADIOS, IO, Variable, and Engine components. A fundamental aspect is that an ADIOS 2 interface requires minimal changes when setting the engine type and parameters, so users can evaluate different transport modes either at compile or run time. Listing 2 illustrates how to read ADIOS 2 data using the Python high-level API. ADIOS 2 provides a ''pythonic'' interface of an iterable container of steps using a generic ''read'' function. As such, variable data is always returned as a numpy [34] array for a straight-forward integration with popular data science frameworks using numpy as a core component, e.g. Pandas [35] and SciPy [36].

Impact
The following is a summary of the current efforts in integrating ADIOS 2 in several scientific computing frameworks and the impact on their data management needs.

Plasma physics
-XGC: The X-point included Gyrokinetic Code (XGC) is a fusion simulation code to model gyrokinetic plasma physics [39,40]. ADIOS has been integrated to handle their largescale I/O operations on DOE's supercomputers, including Summit [8], as well as to support multi-physics coupling workflows [41,42]. Performance gains for using ADIOS's aggregation, staging, and NVRAM/burst buffers support have been reported in various research works [41][42][43][44]. This is illustrated in Fig. 5 for the write performance throughput of XGC checkpoint-restart on Summit's GPFS and NVMe I/O systems. -GTC: The Gyrokinetic Toroidal Code (GTC) [45] simulates billions of plasma particles inside a fusion reactor. The output of GTC consists of small, frequent diagnostics data (low volume, high velocity data), large-sized field data (high frequency, moderate volume data), and checkpoint-restart files (low frequency, high volume data). GTC uses ADIOS 2 to: (1) optimize data file outputs, (2) analyze diagnostics data using its Python high-level interface, and (3) as an input to visualize field and particle data [46].
-PIConGPU: the relativistic, fully-accelerated, electromagnetic particle-in-cell code with primary applications in plasma physics and laser-particle acceleration. PICon-GPU [47,48] is written in performance-portable C++ based on the Alpaka library [49,50] and benefits from ADIOS's aggregation and data compression methods. ADIOS 2 SST staging method has recently been added via the openPMD project [51], providing a high-level abstraction for meshed fields and particle data. Targeted use-cases are rapid prototyping for loosely coupled in-situ reductions with petabytes of data per simulation via on and off-node buffering (e.g. over NVRAM/burst buffers) on the Summit supercomputer [8].

Earth science
-E3SM: ADIOS 2 is integrated into the Parallel IO framework (PIO) as a backend for I/O in Energy Exascale 3D System Model [52]. Currently, ADIOS reduces I/O bottlenecks by a factor of 50x by delaying data reorganization outside extreme scale supercomputers like Titan. Efforts are currently ongoing to port this functionality to the Summit supercomputer [8]. -SPECFEM3D: ADIOS 2 is being integrated in the spectralelement global seismic wave propagation, SPECFEM3D_ GLOBE, framework's forward simulation and adjoint process [53]. Running 1,480 earth quakes simulations produces 1.5 petabytes of wave field data on 15,360 nodes of the Titan supercomputer in about 7 hours. The adjoint simulation later reads in all of the generated data. ADIOS 2 allows for efficient data transfers and self-describing mechanisms at these scales.

Radio astronomy
-Casacore SKA: The Square Kilometer Array (SKA) is being designed to be the largest data producer radio telescope in the world. ADIOS 2 is currently being deeply integrated into the fundamental radio astronomy data the Casacore Table Data System [54]. Early exploratory work suggested improved performance for their I/O workflows such as data staging, and near-real-time data compression [23,55,56].

Aerospace engineering
-UPACS-LES: ADIOS 2 has been integrated into UPACS-LES [38], an in-house program for Computational Fluid Dynamics (CFD) in the Japan Aerospace Exploration Agency (JAXA) for the simulation of the aeroacoustics in aircrafts and launch vehicles. ADIOS 2 flexibility is essential to their post-processing tasks on the JAXA's Supercomputer System Generation 2 (JSS2) system [37]. Fig. 6 shows a schematic representation of the heterogeneous nature of JSS2 and the post-processing pipeline workflow, using the SST staging engine in ADIOS 2, that bypasses the file system.

Community frameworks engagements
Efforts are currently ongoing to integrate ADIOS 2 in the following community frameworks to impact their data management and scalable I/O needs: • LAMMPS: Molecular Dynamics Simulator [57] • MFEM: Modular Finite Element Methods Library [58] • OpenFoam: The open source CFD toolbox [59] • VTK and VTK-m: The Visualization Toolkit [60,61] • The Trilinos Project [62] • PETSc: Portable, Extensible Toolkit for Scientific Computation [63].   6. Overview of the JSS2 system at JAXA [37] and layout for in transit visualization workflows using staging capabilities in ADIOS 2 in the UPACS-LES CFD framework [38].

Conclusions
ADIOS 2 serves the scientific community in their data transport, transform and storage needs. By providing a unified API, ADIOS 2 allows the scientific community to focus on publishing and subscribing their data where and when required without having to handle a plethora or low-level I/O interfaces. ADIOS 2 aims to provide scalable parallel I/O functionality geared towards the next generation of exascale supercomputers, in addition to highlevel mechanisms for data analysis. ADIOS 2 has been adopted by several high-performance scientific computing frameworks. This paper introduces the ADIOS 2 open-source framework to a broader audience seeking to explore new paradigms in their data management needs.

Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.