In geometrically frustrated materials with low-dimensional and small spin moment, the quantum fluctuation can interfere with the complicated interplay of the spin, electron, lattice, and orbital interactions, and host exotic ground states such as the nematic spin state and chiral liquid phase. While the quantum phases of the one-dimensional chain and $S=\frac{1}{2}$ two-dimensional triangular-lattice antiferromagnet (TLAF) have been more thoroughly investigated by both theorists and experimentalists, the work on the $S=1$ TLAF has been limited. We induced the lattice distortion into the TLAFs $A_3\mathrm{NiN}{\mathrm{b}}_2{\mathrm{O}}_9$ ($A=\mathrm{Ba}$, Sr, and Ca) with $S\left(\mathrm{N}{\mathrm{i}}^{2+}\right)=1$, and applied thermodynamic, magnetic, and neutron scattering measurements. Although $A_3\mathrm{NiN}{\mathrm{b}}_2{\mathrm{O}}_9$ kept the noncollinear $120^{\circ }$ antiferromagnetic phase as the ground state, the $\mathrm{N}{\mathrm{i}}^{2+}$ lattice changed from an equilateral triangle ($A=\mathrm{Ba}$) into an isosceles triangle ($A=\mathrm{Sr}$ and Ca). The inelastic neutron scattering data were simulated by the linear spin-wave theory, and the competition between the single-ion anisotropy and the exchange anisotropy from the distorted lattice are discussed.

• Received 11 May 2018

DOI:https://doi.org/10.1103/PhysRevB.98.094412