A number of strongly correlated heavy-fermion compounds, such as samarium (Sm), ytterbium (Yb), plutonium (Pu) hexaboride, are predicted to become topological insulators at low temperatures. These systems support massless Dirac fermions near certain (three) points of the surface Brillouin zone, hereafter referred to as the valleys. In strong perpendicular magnetic fields, the conical Dirac dispersions of these surface states quench onto three sets of Landau levels and we predict various possible hierarchies of incompressible quantum Hall states on the surface of hexaborides. In addition, we address the effects of strong electron-electron interaction within the surface zeroth Landau levels. Specifically, we show that depending on the relative strength of the long-range (Coulomb-type) and the finite-range (Hubbard-type) interactions, the ground state can display either a valley-polarized or a valley-coherent distribution of electronic density. We also show that the transition between two valley-polarized states is always discontinuous, while that between a valley-polarized and a valley-coherent phase is continuous. The Zeeman splitting and/or an applied uniaxial strain on the surface can drive the system through various quantum phase transitions and place it in different broken-symmetry phases. Application of uniaxial strain is also shown to considerably modify the precise sequence of quantum Hall states. We also highlight the role of topology in determining the broken-symmetry phases, disorder on the surface of topological hexaborides in strong magnetic fields.

• Received 10 August 2015

DOI:https://doi.org/10.1103/PhysRevB.92.235144