We investigate the emergence of antiferromagnetic ordering and its effect on the helical edge states in a quantum spin Hall insulator, in the presence of strong Coulomb interaction. Using dynamical mean-field theory, we show that the breakdown of lattice translational symmetry favors the formation of magnetic ordering with nontrivial spatial modulation. The onset of a nonuniform magnetization enables the coexistence of spin-ordered and topologically nontrivial states. An unambiguous signature of the persistence of the topological bulk property is the survival of bona fide edge states. We show that the penetration of the magnetic order is accompanied by the progressive reconstruction of gapless states in subperipheral layers, redefining the actual topological boundary within the system.

• Received 10 May 2018

DOI:https://doi.org/10.1103/PhysRevB.98.045133