We study the Kitaev-Heisenberg-$\mathrm{\Gamma }$ model with antiferromagnetic Kitaev exchanges in the strong anisotropic (toric code) limit to understand the phases and the intervening phase transitions between the gapped $Z_2$ quantum spin liquid and the spin-ordered (in the Heisenberg limit) as well as paramagnetic phases (in the pseudodipolar, $\mathrm{\Gamma }$, limit). We find that the paramagnetic phase obtained in the large $\mathrm{\Gamma }$ limit has no topological entanglement entropy and is proximate to a gapless critical point of a system described by equal superposition of differently oriented stacked one-dimensional $Z_2\times Z_2$ symmetry protected topological phases. Using a combination of exact diagonalization calculations and field theoretic analysis we map out the phases and phase transitions to reveal the complete phase diagram as a function of the Heisenberg, the Kitaev, and the pseudodipolar interactions. Our work shows a rich plethora of unconventional phases and phase transitions and provides a comprehensive understanding of the physics of anisotropic Kitaev-Heisenberg-$\mathrm{\Gamma }$ systems along with our recent paper [Phys. Rev. B 102, 235124 (2020)] where the ferromagnetic Kitaev exchange was studied.

• Received 1 August 2021
• Accepted 27 September 2021

DOI:https://doi.org/10.1103/PhysRevB.104.195115