We present an ab initio relativistic $\mathbf{k}·\mathbf{p}$ theory of the effect of magnetic exchange field on the band structure in the gap region of bulk crystals and thin films of three-dimensional layered topological insulators. For the field perpendicular to the layers (along $z$), we reveal unconventional scenarios of the response of the band-gap edges to the magnetization. The modification of the valence and conduction states is considered in terms of their $\mathrm{\Gamma }$-point spin $s^z$ and total angular momentum $J^z$ on the atomic sites where the states are localized. The actual scenario depends on whether $s^z$ and $J^z$ have the same or opposite sign. In particular, the opposite sign for the valence state and the same sign for the conduction state give rise to an unconventional response in ${\mathrm{Bi}}_2{\mathrm{Te}}_3$—both in the bulk crystal and in ultrathin films, which fundamentally distinguishes this topological insulator from ${\mathrm{Bi}}_2{\mathrm{Se}}_3$, where both states have the same sign. To gain a deeper insight into different scenarios in insulators with both inverted and noninverted zero-field band structure, a minimal four-band third-order $\mathbf{k}·\mathbf{p}$ model is constructed from first principles. Within this model, we analyze the field-induced band structure of the insulators and identify Weyl nodes that appear in a magnetic phase and behave differently depending on the scenario. We characterize the topology of the modified band structure by the Chern number $\mathcal{C}$ and find the unconventional response to be accompanied by a large Chern number $\mathcal{C}=\pm 3$.

• Received 22 December 2020
• Accepted 24 March 2021

DOI:https://doi.org/10.1103/PhysRevB.103.155114