MOOSE: Enabling Massively Parallel Multiphysics Simulation

Harnessing modern parallel computing resources to achieve complex multi-physics simulations is a daunting task. The Multiphysics Object Oriented Simulation Environment (MOOSE) aims to enable such development by providing simplified interfaces for specification of partial differential equations, boundary conditions, material properties, and all aspects of a simulation without the need to consider the parallel, adaptive, nonlinear, finite-element solve that is handled internally. Through the use of interfaces and inheritance, each portion of a simulation becomes reusable and composable in a manner that allows disparate research groups to share code and create an ecosystem of growing capability that lowers the barrier for the creation of multiphysics simulation codes. Included within the framework is a unique capability for building multiscale, multiphysics simulations through simultaneous execution of multiple sub-applications with data transfers between the scales. Other capabilities include automatic differentiation, scaling to a large number of processors, hybrid parallelism, and mesh adaptivity. To date, MOOSE-based applications have been created in areas of science and engineering such as nuclear physics, geothermal science, magneto-hydrodynamics, seismic events, compressible and incompressible fluid flow, microstructure evolution, and advanced manufacturing processes.


Motivation and significance
As computing capability increases, scientists are exploring computational solutions to complex problems involving multiple scientific domains. These multiphysics problems tie together many disciplines and involve collaborators with specific knowledge. Traditionally, multiphysics simulation was achieved by connecting individual simulation tools built by various researchers, which is time-consuming, error-prone, and problematic for taking advantage of modern parallel computing. The Multiphysics Object Oriented Simulation Environment (MOOSE) [1] offers a better approach for development of such software. The framework provides a plug-in infrastructure that simplifies definitions of physics, material properties, and postprocessing. The design allows developers to focus on their scientific endeavor without needing to understand the intricacies of modern parallel computing. In addition, uniformity in problem specification leads to maximizing reuse. MOOSE-based projects can share capabilities without requiring additional development. This ideal is achieved by a design pattern where community-developed physics "modules" and all MOOSE-based applications are libraries and can be linked, enabling conglomerate applications to be created from existing capability. This concept opens new possibilities for collaborating across development teams to build powerful simulation tools that can be seamlessly combined. This strategy has been a paradigm shift in multiple scientific disciplines [2].

Software architecture
MOOSE is designed to facilitate the creation of production finite element method (FEM) tools for running high-fidelity, multiphysics simulations. The software is composed of "systems," each one providing an extension point for defining simulation characteristics. These systems communicate using interfaces, decoupling them to allow for greater code reuse. Each system in MOOSE has a specific C++ base class from which application developers inherit from and extend to perform a desired calculation overriding virtual methods to perform unique calculations needed for that application. This approach has two advantages; the foremost is that the application developers have complete control over their calculation using standard C++, should a need to deviate from the prescribed template arise. Secondly, the inheritance chain can be extended beyond the baseline provided by the framework to provide rich capabilities for downstream applications. MOOSE has approximately 40 systems that may be extended. Individually discussing each of these is beyond the scope of this paper, but the systems that provide the core functionality of the framework can be separated into three groups: partial differential equation (PDE) related terms, material properties, and in-situ postprocessing.
Two systems for defining PDE terms are the Kernel and Boundary-Condition systems. Kernel objects define volumetric integral terms while BoundaryCondition objects define surface integral terms that arise from the derivation of the FEM weak form [3]. For example, the C++ code snippet in Listing 1 is an example of an implementation of an advection term for a PDE. This example illustrates the Kernel system, which supports automatic differentiation (AD) (see Section 2.2). A single method must be extended, precomputeQpResidual.  This is a powerful approach to defining equation terms. By defining one method, this advection operator can be used in 1D, 2D, or 3D. It can utilize a constant velocity, couple to a field computed through solving another PDE, utilize a field computed from another application, or even use a velocity field from an experiment. This Kernel will work on one processor or 100,000 without modification. Any application needing this term can reuse this object without needing to recode it. Finally, it can also be tested in a controlled calculation to ensure correctness.
The Material system allows for the definition of "material properties," often representing coefficients in a PDE. These properties may be nonlinear, and may themselves depend on variables in the PDE. This dependence can be propagated automatically through to the system Jacobian matrix using the AD capability of the framework. Once defined, material properties may be consumed by other systems in the framework through a straight-forward producer/consumer model. MOOSE then ensures those properties are computed when needed by those objects. Material properties can couple to and depend on other material properties; inter-dependencies are tracked, and calculations are executed in the correct order. This design decouples physics calculations from the coefficients they require, allowing for properties to be shared and reused within and across applications without the need to edit code objects consuming the property.
The term "in-situ postprocessing," a perceived oxymoron, describes a set of systems used for data computation that is typically performed postsimulation which are computed along with other calculations after, during or in-between solves. Systems in this category include the Postprocessor, VectorPostprocessor, and AuxKernel systems. The Postprocessor and VectorPostprocessor systems compute scalar and vector values, respectively (e.g., total heat flux through a side or the total chemical concentration). These calculations can themselves depend on other variables, material properties, or other postprocessed values. The AuxKernel system allows the calculation of "field" or spatially varying data using finite element basis functions. In-situ postprocessing enables calculations to be performed in parallel while running at scale. These values can be fed back into other systems, including the calculation of PDE terms or material properties.
The system and interface architecture of the framework allows application developers to build simulation tools capable of solving anything from basic single physics problems to extensive, tightly coupled, cross-disciplinary multiphysics problems only achievable with multiple development teams. The resulting applications, regardless of complexity, are easily extensible and useful for research teams and analysts alike.

Software functionalities 2.2.1. Parallelism
MOOSE is designed to support highly parallel calculations involving many physical phenomena. The parallel capability is implemented using a hybrid approach consisting of Message Passing Interface (MPI) calls with optional shared-memory threading available on-node. Traditionally, parallel computing is complicated, involving many low-level serialization, deserialization, and explicit communication routines. MOOSE provides a comprehensive abstraction to parallelism, requiring minimal knowledge (if any) of parallel constructs from application developers. Additionally, the framework supports an advanced mesh "pre-splitting" capability that automatically takes into account necessary "ghosting" needed by contact, mortar, patch recovery, or user-defined, non-local requirements. Pre-split meshes can then be read efficiently in parallel, reducing startup time and memory use for very large simulations. The pre-split capability has been used to keep per-core memory consumption reasonable for billion element meshes [4]. Scalability of the framework has been demonstrated to over 30,000 processor cores on modern supercomputers for multiple applications including microstructure evolution using the phase-field method and neutron transport. For example, a scaling study is shown in Fig. 1 for a 3D grain growth model based on phase-field equations. The phase-field equations are discretized with 25 grains and 9 order parameters using a first-order Lagrange finite element method on a 3D mesh with 78,643,200 hexahedral elements and 79,300,033 nodes. The resulting system has 713,700,297 unknowns and is solved by the Jacobian-free Newton-Krylov (JFNK) [5] method, employing GMRES [6] and a Schwarz preconditioner [7,8]. The test is conducted using processor counts between 8,192 and 32,768 on the Theta supercomputer at Argonne National Laboratory [9]. Further details of this test can be found in [10,11].

Automatic Differentiation (AD)
Efficient solution of nonlinear systems of equations using Newton's method requires accurate computation of the Jacobian matrix or its action on a vector [5]. When this matrix is approximated via matrix-free methods, accurate preconditioning may still be required for efficient linear convergence. Generating an accurate Jacobian matrix or preconditioner can be a difficult and error-prone task. To reduce the burden on application developers, MOOSE has implemented forward-mode AD. Using this capability, application developers only implement residual statements, reducing the amount of code and time needed to implement a new physics object. Sharing physics code among applications can consequently be done with greater confidence. In this way, AD enhances the modularity of MOOSE-based applications.

Sub-applications
The architecture of MOOSE allows separately developed physics simulation tools to be integrated seamlessly via the MultiApp system. As an example, two different research groups might each be focused on different aspects of a problem: one on an engineering scale simulation and one on microstructure evolution. Using MOOSE, both applications can be developed in a uniform, yet flexible way. The two research groups may work independently, focusing on their respective scales. When a higher fidelity, coupled simulation is needed, these two applications can be compiled together, and a multiscale simulation can be performed using the unique MultiApp and Transfer systems without the need to develop additional code to link the applications together (see [2,12]).

Restart
The framework includes a restart system that allows for all aspects of a simulation to be stored in memory or output to checkpoint files. This allows for single applications to be recovered after an unexpected failure (e.g., a power outage), to be restored while conducting iterative solves across multiple applications (see Sub-applications above), as well as to create complex multistage initialization for large problems. Notably, this system is extendable so that user-defined data structures can be stored and reloaded when needed in each of these scenarios and when running at scale.

Testing, Visualization, and Documentation
A modular framework alone is not sufficient to create robust and useful scientific simulation tools. MOOSE also includes infrastructure for testing, visualization, and documentation, which, collectively with the framework, is referred to as the "MOOSE Platform." As with the framework, each of these utilities is designed with similar architectures and to be modular and extendable, allowing application developers to customize these tools. The testing system includes tools for creating unit, regression, and system tests, including tests for error conditions and parallel consistency. The graphical user interface (GUI) is designed to work with derivative applications containing extended objects and custom syntax. The visualization core of the GUI may be scripted to create complex visualizations. Finally, the documentation system operates with single-source markdown files to create websites, presentations, or L A T E X documents and is capable of checking cross-reference consistency as documentation or source code changes.

Illustrative Example
MOOSE is capable of solving complex multiphysics problems, including systems with highly variable space and time scales. For example, the results in Figure 2 couple an engineering scale problem to a micro-structure scale calculation. The engineering scale physics is porous flow, modeled using Darcy's equation within a cylinder assuming a porous media of closely packed steel spheres, mimicking the experiment detailed in [13]. Each end of the cylinder is attached to a vessel with a prescribed temperature and pressure. The simulation solves the mass, energy, and momentum balance equations for pressure, temperature, and thermally induced strain, respectively using a fully-coupled Newton-based solve. At each time step, a set of micro-structure calculations is performed, resolving the closely-packed spheres. These spheres are degrading, causing an increase in permeability over time. The degradation rate increases with temperature, resulting in a change to the effective thermal conductivity at the engineering scale. Detailed steps for the solve are as follows: 1. At time t = −1 the solution for temperature is initialized to 300C, pressure, and displacements are set to zero; the effective thermal conductivity is set to the known value for close-packed spheres of steel in water. 2. The engineering scale problem is solved with a time step of 0.25 sec.
(until t = 0). During these steps, the temperature and pressure boundary conditions are increased linearly to 350C and 4000Pa, respectively. 3. The engineering problem computes a solution for a time step, samples the temperature at six evenly spaced points along the domain, and sets the temperature of the corresponding micro-structure calculations. 4. The micro-structure application computes degradation of the steel spheres using the temperature from the engineering scale application and then calculates the effective thermal conductivity based on the changed geometry. The resulting effective thermal conductivity is then applied using linear interpolation to the engineering-scale heat conduction calculation. 5. Steps 3-4 are repeated until a steady-state solution is reached. Figure 2: Results after 58 seconds of simulation for a thermo-mechanical problem with feedback from a micro-structure calculation that shows the temperature at the engineering scale, the geometry of each degraded micro-structure, and the effective thermal conductivity over time for each of the micro-structure calculations.

Impact
Software development is increasingly becoming an integral part of the scientific process. Currently, scientists and engineers develop their own software, spending approximately 30% of their time writing code [14]. These domain scientists often do not follow code reuse best practices [15], which can result in continuous reinvention of software as new researchers inherit opaque works left by a previous generation. By providing a framework that follows best practices for scientific computing, it is possible to aid researchers in combating these shortcomings to improve the quality of scientific research.
Developing trust in a framework goes beyond utilizing best practices; it is equally important that the framework itself is built on a foundation of trusted tools. For this reason MOOSE relies on the well-established code bases of libMesh [16] for finite elements and on PETSc [17,18,19] for leading-edge numerical methods/solvers. Furthermore, MOOSE itself has been subjected to multiple peer-reviews and Software Quality Assurance (SQA) audits.
MOOSE is a powerful open-source, massively parallel, finite element based, multiphysics framework. Other open-source frameworks certainly exist (e.g., [20,21,22]), but MOOSE is a common platform for creation of comprehensive multiphysics applications that are inherently shareable and can be coupled without modification. MOOSE is being used by governments, private industry, and universities both nationally and internationally. For example, MOOSE is used for coupling multiple nuclear energy codes, both external and MOOSEbased [12,23,24,25]; it is used for modeling porous flow in 3D fractured porous media [26]; and is the basis of multiple efforts to model sub-surface exploration [27,28,29,30].

Conclusions
MOOSE is a software framework that allows scientists to develop state-ofthe-art, scalable applications without worrying about parallel, finite element, or solver implementation details. Moreover, the modular design of MOOSE allows these applications to be combined easily, encouraging reuse and simplifying construction of multiscale, multiphysics simulations. The framework has attracted a sizable research community, pushing the frontiers of scientific computing, solving PDE's with tens of billions of unknowns on tens of thousands of processors. MOOSE development continues to focus on robust, high-performance solution algorithms that work across broad scales, from serial to hundreds of thousands of processes.
Contract No. DE-AC07-05ID14517 with the US Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes.