We present observations of highly frustrated quasi-two-dimensional (2D) magnetic correlations in the honeycomb lattice layers of the $S_{\text{eff}} =1/2$ compound $\gamma -{\mathrm{BaCo}}_2{\left({\mathrm{PO}}_4\right)}_2$ ($\gamma$-BCPO). Specific heat shows a broad peak comprised of two weak kink features at $T_{N1}\sim 6$ K and $T_{N2}\sim 3.5$ K, the relative weights of which can be modified by sample annealing. Neutron powder diffraction measurements reveal short range quasi-2D order that is established below $T_{N1}$ and $T_{N2}$, at which two separate, incompatible, short range magnetic orders onset: commensurate antiferromagnetic correlations with correlation length ${\xi }_c=60\pm 2$ Å ($T_{N1}$) and in quasi-2D helical domains with ${\xi }_h=350\pm 11$ Å ($T_{N2}$). The ac magnetic susceptibility response lacks frequency dependence, ruling out spin freezing. Inelastic neutron scattering data on $\gamma$-BCPO is compared with linear spin wave theory, and two separate parameter regions of the XXZ $J_1-J_2-J_3$ model with ferromagnetic nearest-neighbor exchange $J_1$ are favored, both near regions of high classical degeneracy. High energy coherent excitations ($\sim 10$ meV) persist up to at least 40 K, suggesting strong in-plane correlations persist above $T_N$. These data show that $\gamma$-BCPO is a rare highly frustrated, quasi-2D $S_{\text{eff}} =1/2$ honeycomb lattice material which resists long range magnetic order and spin freezing.

• Received 30 November 2017
• Revised 10 February 2018

DOI:https://doi.org/10.1103/PhysRevB.97.134409