The rise of exascale supercomputers has fueled competition among GPU vendors, driving lattice QCD developers to write code that supports multiple APIs. Moreover, new developments in algorithms and physics research require frequent updates to existing software. These challenges have to be balanced against constantly changing personnel. At the same time, there is a wide range of applications for HISQ fermions in QCD studies. This situation encourages the development of software featuring a HISQ action that is flexible, high-performing, open source, easy to use, and easy to adapt. In this technical paper, we explain the design strategy, provide implementation details, list available algorithms and modules, and show key performance indicators for SIMULATeQCD, a simple multi-GPU lattice code for large-scale QCD calculations, mainly developed and used by the HotQCD collaboration. The code is publicly available on GitHub.
Nature of problem: Quantum chromodynamics (QCD) is the fundamental theory behind the strong force, one of the four fundamental forces in nature. It describes interactions, mediated by gluons, between quarks, the elementary particles that build up protons, neutrons and other hadrons. One of its unique features is asymptotic freedom: with increasing energy, the interaction strength or coupling between quarks and gluons weakens, which enables studying QCD with peturbation theory. At lower energy scales, however, the strong coupling between quarks and gluons causes perturbation theory to break down, and other tools, such as lattice QCD, need to be employed to obtain QCD predictions.
Solution method: Lattice QCD is a technique to solve QCD non-perturbatively by discretizing space and time onto a four-dimensional grid. The QCD path integral is thereby rendered finite dimensional and can be evaluated numerically via Monte Carlo methods. In this paper, we present SIMULATeQCD, a C++ code to perform lattice QCD calculations on multiple GPUs, supporting multiple APIs. SIMULATeQCD implements various Monte Carlo sampling methods ranging from sampling of purely gluonic configurations to fully dynamical quark simulations with the Highly Improved Staggered Quark (HISQ) discretization. Modules for measuring various physical observables on these samples are also available.
We provide details of the first implementation of a non-linear conjugate gradient method for Landau and Coulomb gauge fixing with Fourier acceleration. We find clear improvement over the Fourier accelerated steepest descent method, with the average time taken for the algorithm to converge to a fixed, high accuracy, being reduced by a factor of 2 to 4.
We discuss and propose a proper extension of the Abelian projection based on the Maximal Abelian Gauge to gauge theories. Based on that, we investigate the properties of thermal Abelian monopoles in the deconfined phase of the pure gauge theory. Such properties are very similar to those already found for , confirming the relevance of the magnetic component close to and the possible condensation of thermal monopoles as the deconfinement temperature is crossed from above. Moreover, we study the correlation functions among monopoles related to different subgroups, which show interesting features and reveal the presence of non-trivial interactions.
A lattice gauge theory framework for simulations on graphic processing units (GPUs) using NVIDIA’s CUDA is presented. The code comprises template classes that take care of an optimal data pattern to ensure coalesced reading from device memory to achieve maximum performance. In this work we concentrate on applications for lattice gauge fixing in 3+1 dimensional SU(3) lattice gauge field theories. We employ the overrelaxation, stochastic relaxation and simulated annealing algorithms which are perfectly suited to be accelerated by highly parallel architectures like GPUs. The applications support the Coulomb, Landau and maximally Abelian gauges. Moreover, we explore the evolution of the numerical accuracy of the SU(3) valued degrees of freedom over the runtime of the algorithms in single- (SP) and double-precision (DP). Therefrom we draw conclusions on the reliability of SP and DP simulations and suggest a mixed-precision scheme that performs the critical parts of the algorithm in full DP while retaining 80%–90% of the SP performance. Finally, multi-GPUs are adopted to overcome the memory constraint of single GPUs. A communicator class which hides the MPI data exchange at the boundaries of the lattice domains, via the low bandwidth PCI-Bus, effectively behind calculations in the inner part of the domain is presented. Linear scaling using 16 NVIDIA Tesla C2070 devices and a maximum performance of 3.5 Teraflops on lattices of size down to is demonstrated.
