We explore the ferromagnetic quantum critical point in a three-dimensional semimetallic system with upward- and downward-dispersing bands touching at the Fermi level. Evaluating the static spin susceptibility to leading order in the coupling between the fermions and the fluctuating ferromagnetic order parameter, we find that the ferromagnetic quantum critical point is masked by an incommensurate, longitudinal spin density wave phase. We first analyze an idealized model that, despite having strong spin-orbit coupling, still possesses $O\left(3\right)$ rotational symmetry generated by the total angular momentum operator. In this case, the direction of the incommensurate spin density wave propagation can point anywhere, while the magnetic moment is aligned along the direction of propagation. Including symmetry-allowed anisotropies in the fermion dispersion and the coupling to the order parameter field, however, we find that the ordering wavevector instead breaks a discrete symmetry and aligns along either the [111] or [100] direction, depending on the signs and magnitudes of these two types of anisotropy.

• Received 1 June 2015
• Revised 7 July 2015

DOI:https://doi.org/10.1103/PhysRevB.92.035137