We investigate the Hubbard model on the honeycomb lattice with intrinsic spin-orbit interactions as a paradigm for two-dimensional topological band insulators in the presence of interactions. Applying a combination of Hartree-Fock theory, slave-rotor techniques, and topological arguments, we show that the topological band insulating phase persists up to quite strong interactions. Then we apply the slave-rotor mean-field theory and find a Mott transition at which the charge degrees of freedom become localized on the lattice sites. The spin degrees of freedom, however, are still described by the original Kane-Mele band structure. Gauge-field effects in this region play an important role. When the honeycomb layer is isolated then the spin sector becomes already unstable toward an easy-plane Neel order. In contrast, if the honeycomb lattice is surrounded by extra “screening” layers with gapless spinons, then the system will support a fractionalized topological insulator phase with gapless spinons at the edges. For large interactions, we derive an effective spin Hamiltonian.

• Received 31 March 2010

DOI:https://doi.org/10.1103/PhysRevB.82.075106