Spin molecular-orbit coupling and magnetic properties of the decorated honeycomb layers of Mo3S7(dmit)3 crystals

We explore the magnetic properties of isolated a-b planes of trinuclear organometallic crystals, Mo3S7(dmit)3, in which an interplay of strong electronic correlations and spin molecular-orbital coupling (SMOC) occurs. The magnetic properties can be described by a XXZ+120, S=1 Heisenberg model on a honeycomb lattice with single-spin anisotropy, D, which depends strongly on SMOC. Based on ab initio estimates of SMOC in Mo3S7(dmit)3 crystals, we predict that the honeycomb layers of Mo3S7(dmit)3 are Neel ordered. However, in materials with a greater degree of magnetic frustration, Neel order can give way to the large-D phase.


I. INTRODUCTION
Strong spin orbit coupling (SOC) in weakly interacting systems can lead to unconventional insulating states such as topological insulators. Strongly correlated phases such as the topological Mott insulator can emerge from the interplay of strong Coulomb repulsion and SOC which may be realized in Ir-based transition metal oxides. 1 For instance, in Na 2 IrO 3 and Li 2 IrO 3 materials, SOC removes the orbital degeneracy of 5d electrons leading to S = 1/2 pseudospins which interact through anisotropic and quantum compass exchange interactions on a honeycomb lattice. These materials are potential realizations of the Heisenberg-Kitaev model, closely related to the Kitaev model which sustains a spin liquid state 2 .
Multinuclear coordinated organometallic complexes are strongly correlated systems in which spin-orbit coupling (SOC) can be relevant. The multinuclear complex, Mo 3 S 7 (dmit) 3 , consists of honeycomb networks of Mo 3 S 7 (dmit) 3 molecules stacked on top of each other along the c-direction of the crystal. Since the Mo 3 S 7 (dmit) 3 molecules can be described by three Wannier orbitals 3 , their packing on the honeycomb lattices within the layers lead to decorated honeycomb lattices, as shown in Fig. 1. Based on perturbative expansions including Coulomb repulsion and SMOC, the effective spin exchange model which describes the magnetic properties of these layers is a S = 1 XXZ+120 0 quantum compass model on the honeycomb lattice. 4,5 In the present paper, we discuss the role played by the single-spin anisotropy induced by SMOC in the magnetic properties of the honeycomb layers of Mo 3 S 7 (dmit) 3 crystals shown in Fig. 1. We predict that isolated honeycomb layers of Mo 3 S 7 (dmit) 3 crystals are Néel ordered but increasing the magnetic frustration of the lattice can drive the system into a large-D phase. planes of Mo3S7(dmit)3 crystals. Mo3S7(dmit)3 molecules are described by three Wannier orbitals (black dots), which form the triangles of the decorated lattice encircled by a dashed green line. The intracluster hopping amplitude between two molecular orbitals inside the triangle is denoted by tc while the intercluster hopping amplitude between two orbital in neighboring molecules is denoted by t. The red arrows indicate the Bravais lattice vectors of the underlying honeycomb lattice. The three bonds connecting equal sites between nearestneighbor clusters are numbered from 1 to 3.

A. Model for isolated trimers
The isolated triangular clusters can be described by a Hubbard-Heisenberg model in the presence of the spin molecular-orbit coupling (SMOC): where: with t c the hopping between hybrid metal-ligand orbitals at sites in the triangular clusters and a † iσ creates an electron at the ith Wannier orbital with spin σ.
An important question is how SOC affects the low energy states described by these Wannier orbitals. Generically the one-electron spin-orbit coupling takes the form H SOC = α K α ·s α , where s α is the spin of the α th electron and K α is a pseudovectorial operator that acts only on the spatial part of the wavefunction. For example, in the Pauli approximation, where p is the single electron momentum operator and φ(r) is the potential through which the electron moves. Spherical symmetry implies that for an isolated atom K ∝ L, the angular momentum operator. Thus, in ab initio calculations, it is common to write the SOC as a linear superposition of atomic contributions, for example, in the mean-field Breit-Pauli approximation 7,8 where Z ef f A is the effective charge of nucleus A which accounts for the screening effects of the rest of the electrons on the nuclear potential, l αA = (r α − R A ) × p α , is the angular momentum of the α th electron relative to the nucleus A at R A . This atomic approach does not provide any insight into how SOC acts on the low-energy states described by the Wannier orbitals and requires detailed first principles calculations for each new material studied. A more chemically intuitive description should consider the coupling between the electronic spin and the molecular orbital degrees of freedom.
Symmetry dictates the general form of the SMOC 6 . For cyclic molecules with C N symmetry the coupling is 6 For a C 3 symmetric molecule, such as Mo 3 S 7 (dmit) 3 , this simplifies to where L x,y,z α are the Cartesian components of the oneelectron molecular orbital "angular momentum" operator for the α th electron, λ xy , is the transverse SMOC and λ z the longitudinal SMOC.
The SMOC hamiltonian (6) gives a far clearer understanding of the relevant low-energy physics than the atomistic description (4). This is because molecular orbitals, not atomic orbitals, are the natural basis for the discussion of molecular physics. As molecular orbitals for cyclic molecules are Bloch states of the Wanniers, H SM OC describes the coupling of orbital currents running around the plane of the molecule to the electron's spin. The low-energy model constructed from Wannier spinors from four component relativistic band structure calculations 9 of Mo 3 S 7 (dmit) 3 yields single molecule SMOC exactly as predicted by Eq. (6).
Finally, the Hubbard-Heisenberg contribution describing the Coulomb and exchange energy in each triangular cluster is: where U is the onsite Hubbard interaction, J F is an intracluster exchange interaction, and n iσ = a † iσ a iσ the number operator. The direct ferromagnetic exchange, J F < 0, plays a crucial role in generating magnetic anisotropies. 4 The nearest-neighbor triangular clusters are connected through the hopping amplitude, t: where lm denotes two nearest-neighbor triangular clusters and i a cluster site in a given molecule.

B. Effective spin exchange model
By straightforward diagonalization of the full hamiltonian (1), we have found that in the strong coupling limit, U t, isolated triangular clusters with four electrons effectively behave as pseudospin-1 localized moments 4 . The many-body states of the cluster can be classified according to the z-component of the total angular momentum, J z = L z + S z . The ground state of the isolated cluster with no SMOC is a triplet. As shown in Fig.2, SMOC splits the lowest energy triplet into a nondegenerate singlet (j = 0) which is the ground state and a doublet (j = ±1), where j denotes the eigenstates of J z . Since we have an even number of electrons in the cluster, Kramers theorem does not apply and nondegenerate states are possible. SMOC can be effectively accounted for as a single-spin anisotropy contribution at each cluster. Hence, the effective spin model for the m-th Mo 3 S 7 (dmit) 3 molecule in the crystal is just:

o h 5 x m k I o 6 V D x m D O K V u q 7 b g / h C b X M D V d D A T j r u 1 W / 5 i / g r Z O g I F V S o N F 3 v 3 q D h G U S F D J B j e k G f o p h T j V y J m B W 6 W U G U s r G d A h d S x W V Y M J 8 c f n M u 7 D K w I s T b U u h t 1 B / T + R U G j O V k e 2 U F E d m 1 Z u L / 3 n d D O P b M O c q z R A U W y 6 K M + F h 4 s 1 j 8 A Z c A 0 M x t Y Q y z e 2 t H h t R T R n a s C o 2 h G D 1 5 X X S u q o F f i 1 4 u K 7 W r 4 s 4 y u S M n J N L E p A b U i f 3 p E G a h J E J e S a v 5 M 3 J n R f n 3 f l Y t p a c Y u a U / I H z + Q O D G Z Q r < / l a t e x i t > H e↵ < l a t e x i t s h a 1 _ b a s e 6 4 = " D M n N q f i V m D s 6 H N R i g S w X g s w c t Y I = " > A A A B + X i c b V B N S 8 N A E N 3 4 W e t X 1 K O X Y B E 8 l U Q E 9 V b w 0 m M F + w F t L J v t p F 2 6 2 Y T d S b G E / B M v H h T x 6 j / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A T X 6 L r f 1 t r 6 x u b W d m m n v L u 3 f 3 B o H x 2 3 d J w q B k 0 W i 1 h 1 A q p B c A l N 5 C i g k y i g U S C g H Y z v Z n 5 7 A k r z W D 7 g N A E / o k P J Q 8 4 o G q l v 2 / X H r I f w h C r K I A z z v G 9 X 3 K o 7 h 7 N K v I J U S I F G 3 / 7 q D W K W R i C R C a p 1 1 3 M T 9 D O q k D M B e b m X a k g o G 9 M h d A 2 V N A L t Z / P L c + f c K A M n j J U p i c 5 c / T 2 R 0 U j r a R S Y z o j i S C 9 7 M / E / r 5 t i e O N n X C Y p g m S L R W E q H I y d W Q z O g C t g K K a G U K a 4 u d V h I 6 o o Q x N W 2 Y T g L b + 8 S l q X V c + t e v d X l d p t E U e J n J I z c k E 8 c k 1 q p E 4 a p E k Y m Z B n 8 k r e r M x 6 s d 6 t j 0 X r m l X M n J A / s D 5 / A E b Y l A k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " D M n N q f i V m D s 6 H N R i g S w X g s w c t Y I = " > A A A B + X i c b V B N S 8 N A E N 3 4 W e t X 1 K O X Y B E 8 l U Q E 9 V b w 0 m M F + w F t L J v t p F 2 6 2 Y T d S b G E / B M v H h T x 6 j / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A T X 6 L r f 1 t r 6 x u b W d m m n v L u 3 f 3 B o H x 2 3 d J w q B k 0 W i 1 h 1 A q p B c A l N 5 C i g k y i g U S C g H Y z v Z n 5 7 A k r z W D 7 g N A E / o k P J Q 8 4 o G q l v 2 / X H r I f w h C r K I A z z v G 9 X 3 K o 7 h 7 N K v I J U S I F G 3 / 7 q D W K W R i C R C a p 1 1 3 M T 9 D O q k D M B e b m X a k g o G 9 M h d A 2 V N A L t Z / P L c + f c K A M n j J U p i c 5 c / T 2 R 0 U j r a R S Y z o j i S C 9 7 M / E / r 5 t i e O N n X C Y p g m S L R W E q H I y d W Q z O g C t g K K a G U K a 4 u d V h I 6 o o Q x N W 2 Y T g L b + 8 S l q X V c + t e v d X l d p t E U e J n J I z c k E 8 c k 1 q p E 4 a p E k Y m Z B n 8 k r e r M x 6 s d 6 t j 0 X r m l X M n J A / s D 5 / A E b Y l A k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " D M n N q f i V m D s 6 H N R i g S w X g s w c t Y I = " > A A A B + X i c b V B N S 8 N A E N 3 4 W e t X 1 K O X Y B E 8 l U Q E 9 V b w 0 m M F + w F t L J v t p F 2 6 2 Y T d S b G E / B M v H h T x 6 j / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A T X 6 L r f 1 t r 6 x u b W d m m n v L u 3 f 3 B o H x 2 3 d J w q B k 0 W i 1 h 1 A q p B c A l N 5 C i g k y i g U S C g H Y z v Z n 5 7 A k r z W D 7 g N A E / o k P J Q 8 4 o G q l v 2 / X H r I f w h C r K I A z z v G 9 X 3 K o 7 h 7 N K v I J U S I F G 3 / 7 q D W K W R i C R C a p 1 1 3 M T 9 D O q k D M B e b m X a k g o G 9 M h d A 2 V N A L t Z / P L c + f c K A M n j J U p i c 5 c / T 2 R 0 U j r a R S Y z o j i S C 9 7 M / E / r 5 t i e O N n X C Y p g m S L R W E q H I y d W Q z O g C t g K K a G U K a 4 u d V h I 6 o o Q x N W 2 Y T g L b + 8 S l q X V c + t e v d X l d p t E U e J n J I z c k E 8 c k 1 q p E 4 a p E k Y m Z B n 8 k r e r M x 6 s d 6 t j 0 X r m l X M n J A / s D 5 / A E b Y l A k = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " D M n N q f i V m D s 6 H N R i g S w X g s w c t Y I = " > A A A B + X i c b V B N S 8 N A E N 3 4 W e t X 1 K O X Y B E 8 l U Q E 9 V b w 0 m M F + w F t L J v t p F 2 6 2 Y T d S b G E / B M v H h T x 6 j / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A T X 6 L r f 1 t r 6 x u b W d m m n v L u 3 f 3 B o H x 2 3 d J w q B k 0 W i 1 h 1 A q p B c A l N 5 C i g k y i g U S C g H Y z v Z n 5 7 A k r z W D 7 g N A E / o k P J Q 8 4 o G q l v 2 / X H r I f w h C r K I A z z v G 9 X 3 K o 7 h 7 N K v I J U S I F G 3 / 7 q D W K W R i C R C a p 1 1 3 M T 9 D O q k D M B e b m X a k g o G 9 M h d A 2 V N A L t Z / P L c + f c K A M n j J U p i c 5 c / T 2 R 0 U j r a R S Y z o j i S C 9 7 M / E / r 5 t i e O N n X C Y p g m S L R W E q H I y d W Q z O g C t g K K a G U K a 4 u d V h I 6 o o Q x N W 2 Y T g L b + 8 S l q X V c + t e v d X l d p t E U e J n J I z c k E 8 c k 1 q p E 4 a p E k Y m Z B n 8 k r e r M x 6 s d 6 t j 0 X r m l X M n J A / s D 5 / A E b Y l A k = < / l a t e x i t > j = 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " i v k V C z g F h z C u g t / r g O o 0 8 V i 5 t g Q = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 C o k I e h E K X j x W t B / Q h r L Z b t q 1 m 0 3 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Y S q F Q c / 7 d l Z W 1 9 Y 3 N k t b 5 e 2 d 3 b 3 9 y s F h 0 y S Z Z r z B E p n o d k g N l 0 L x B g q U v J 1 q T u N Q 8 l Y 4 u p n 6 r S e u j U j U A 4 5 T H s R 0 o E Q k G E U r 3 T 9 e e 7 1 K 1 X O 9 G c g y 8 Q t S h Q L 1 X u W r 2 0 9 Y F n O F T F J j O r 6 X Y p B T j Y J J P i l 3 M 8 N T y k Z 0 w D u W K h p z E + S z U y f k 1 C p 9 E i X a l k I y U 3 9 P 5 D Q 2 Z h y H t j O m O D S L 3 l T 8 z + t k G F 0 F u V B p h l y x + a I o k w Q T M v 2 b 9 I X m D O X Y E s q 0 s L c S N q S a M r T p l G 0 I / u L L y 6 R 5 7 v q e 6 9 9 d V G t u E U c J j u E E z s C H S 6 j B L d S h A Q w G 8 A y v 8 O Z I 5 8 V 5 d z 7 m r S t O M X M E f + B 8 / g C 7 G 4 1 b < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " i v k V C z g F h z C u g t / r g O o 0 8 V i 5 t g Q = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 C o k I e h E K X j x W t B / Q h r L Z b t q 1 m 0 3 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Y S q F Q c / 7 d l Z W 1 9 Y 3 N k t b 5 e 2 d 3 b 3 9 y s F h 0 y S Z Z r z B E p n o d k g N l 0 L x B g q U v J 1 q T u N Q 8 l Y 4 u p n 6 r S e u j U j U A 4 5 T H s R 0 o E Q k G E U r 3 T 9 e e 7 1 K 1 X O 9 G c g y 8 Q t S h Q L 1 X u W r 2 0 9 Y F n O F T F J j O r 6 X Y p B T j Y J J P i l 3 M 8 N T y k Z 0 w D u W K h p z E + S z U y f k 1 C p 9 E i X a l k I y U 3 9 P 5 D Q 2 Z h y H t j O m O D S L 3 l T 8 z + t k G F 0 F u V B p h l y x + a I o k w Q T M v 2 b 9 I X m D O X Y E s q 0 s L c S N q S a M r T p l G 0 I / u L L y 6 R 5 7 v q e 6 9 9 d V G t u E U c J j u E E z s C H S 6 j B L d S h A Q w G 8 A y v 8 O Z I 5 8 V 5 d z 7 m r S t O M X M E f + B 8 / g C 7 G 4 1 b < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " i v k V C z g F h z C u g t / r g O o 0 8 V i 5 t g Q = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 C o k I e h E K X j x W t B
/ Q h r L Z b t q 1 m 0 3 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Y S q F Q c / 7 d l Z W 1 9 Y 3 N k t b 5 e 2 d 3 b 3 9 y s F h 0 y S Z Z r z B E p n o d k g N l 0 L x B g q U v J 1 q T u N Q 8 l Y 4 u p n 6 r S e u j U j U A 4 5 T H s R 0 o E Q k G E U r 3 T 9 e e 7 1 K 1 X O 9 G c g y 8 Q t S h Q L 1 X u W r 2 0 9 Y F n O F T F J j O r 6 X Y p B T j Y J J P i l 3 M 8 N T y k Z 0 w D u W K h p z E + S z U y f k 1 C p 9 E i X a l k I y U 3 9 P 5 D Q 2 Z h y H t j O m O D S L 3 l T 8 z + t k G F 0 F u V B p h l y x + a I o k w Q T M v 2 b 9 I X m D O X Y E s q 0 s L c S N q S a M r T p l G 0 I / u L L y 6 R 5 7 v q e 6 9 9

d V G t u E U c J j u E E z s C H S 6 j B L d S h A Q w G 8 A y v 8 O Z I 5 8 V 5 d z 7 m r S t O M X M E f + B 8 / g C 7 G 4 1 b < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " i v k V C z g F h z C u g t / r g O o 0 8 V i 5 t g Q = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 C o k I e h E K X j x W t B
/ Q h r L Z b t q 1 m 0 3 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Y S q F Q c / 7 d l Z W 1 9 Y 3 N k t b 5 e 2 d 3 b 3 9 y s F h 0 y S Z Z r z B E p n o d k g N l 0 L x B g q U v J 1 q T u N Q 8 l Y 4 u p n 6 r S e u j U j U A 4 5 T H s R 0 o E Q k G E U r 3 T 9 e e 7 1 K 1 X O 9 G c g y 8 Q t S h Q L 1 X u W r 2 0 9 Y F n O F T F J j O r 6 X Y p B T j Y J J P i l 3 M 8 N T y k Z 0 w D u W K h p z E + S z U y f k 1 C p 9 E i X a l k I y U 3 9 P 5 D Q 2 Z h y H t j O m O D S L 3 l T 8 z + t k G F 0 F u V B p h l y x + a I o k w Q T M v 2 b 9 I X m D O X Y E s q 0 s L c S N q S a M r T p l G 0 I / u L L y 6 R 5 7 v q e 6 9 9 d V G t u E U c J j u E E z s C H S 6 j B L d S h A Q w G 8  where S rm describes the effective pseudospin-1 localized at a triangular cluster. The single-spin anisotropy, D, depends strongly on SMOC as shown in Fig. 3. Using perturbation theory 4,5 we have obtained an effective spin exchange model describing the magnetic exchange coupling between the pseudospin-1 in the a − b planes of Mo 3 S 7 (dmit) 3 . This model is an XXZ+120 0 honeycomb quantum compass model 4 leading to:

D R j / r l Y h o k m + A = " > A A A B 7 n i c b V B N S w M x E J 2 t X 7 V + V T 6 C R b B 0 7 I r g l 6 E g h e P F e w H t E v J p t k 2 N s m G J C u U p T / C i w d F v P p 7 v P l v T N s 9 a O u D g c d 7 M 8 z M i x V n x g b B t d a W 9 / Y 3 C p v V 3 Z 2 9 / Y P q o d H L Z N m m t A m S X m q O z E 2 l D N J m 5 Z Z T j t K U y x i T t v x + H b m t 5 + o N i y V D 3 a i a C T w U L K E E W y d H 6 8 6 S m B w n 6 F v j B H G i V h A W p Q Y F G v / r V G 6 Q k E R a w r E x 3 T B Q N s q x t o x w
where J = (J xx + J yy )/2, ∆ = J zz /J and Q = (J xx − J yy )/2 and φ i = 2π(i − 1)/3 where i labels the three bonds around each shown in Fig. 1. Thus, we see that the second term (proportional to J) is simply the XXZ model and the third term (proportional to Q) is the honeycomb 120 • compass model. 10 The exchange couplings and single-spin anisotropy entering model (10) have been obtained numerically from perturbation theory 4 as well as analytically using a canonical transformation. 5 In Fig. 3 we show the dependence of the model (10) parameters: J, Q, ∆ and D with λ xy for the longitudinal SMOC, λ z = λ xy /2, appropriate for Mo 3 S 7 (dmit) 3 crystals. With no SMOC, the exchange couplings are isotropic: J ≈ 0.04t c , Q = 0, ∆ = 1, J xz = 0, as it should. As SMOC increases, spin exchange anisotropy, i. e., Q = 0, ∆ = 1 (or equivalently J xx = J yy = J zz ) increases. On the other hand, Fig. 3 shows how the single-spin anisotropy is rapidly enhanced by SMOC.

III. LARGE-D PHASE INDUCED BY SMOC
One can see from the results of Fig. 3 that, for sufficiently large SMOC, the single-spin anisotropy overcomes the exchange couplings, D > J, ∆J = J zz . We can expect that in the limit, D J, J zz , the model is dominated by single-spin anisotropy and the ground state of the model is just the tensor product of S z r = 0 at each lattice site. Hence, it is important to first obtain an estimate of the critical D c at which a transition to a large-D phase occurs and the magnitude of SMOC needed to reach such critical D. In order to do so, we explore below the magnetic properties of a slightly simplified version of the full model (10) in the limit of weak SMOC. This limit is relevant to the a − b planes of Mo 3 S 7 (dmit) 3 , for which ab initio calculations 9 give λ Mo3S7(dmit) 3 xy = 0.042t c with λ z = λ xy /2. We can see from Fig. 3 that for λ xy 0.5 exchange coupling anisotropies are negligible which justifies using an isotropic Heisenberg model for describing the magnetic properties of the a − b planes of the crystal.
We have recently explored the Néel to large-D transition of the isotropic version of model (10) using a SU (3) description of the spins which introduces three Schwinger bosons to properly account for the three projections: S z i = 0, ±1 of the spin-1 at each lattice site. 11 We find a transition from the Néel ordered phase to a large-D phase for D c ≈ 3J. From this condition and the dependence of D with λ xy we can obtain a realistic estimate of SMOC needed for the large-D phase to occur in isolated a − b layers of Mo 3 S 7 (dmit) 3 . In Fig. 3 we show the critical SMOC (λ xy /t c ) critic ≈ 0.38 at which the D c = 3J condition is satisfied. This critical value is about nine times larger than SMOC in Mo 3 S 7 (dmit) 3 crystals. Hence, we conclude that isolated a−b layers of Mo 3 S 7 (dmit) 3 should be in a magnetically ordered Néel state.

IV. MAGNETIC FRUSTRATION EFFECTS
Magnetic frustration plays an important role in inducing disordered spin liquid states. We analyze the effect of a next-nearest neighbor exchange coupling, J , on the magnetic properties of model (10) in the limit of weak SMOC i. e. with isotropic couplings. Hence, the model we study is a S = 1 J-J Heisenberg on the honeycomb We first discuss the phase diagram of this model with no single-spin anisotropy, D = 0, based on recent SU(2) Schwinger boson mean-field theory (SBMFT) calculations. 11 For J = 0, the ground state of the model is a Néel state, as expected since the lattice is bipartite. A direct transition from Néel order to spiral order occurs for J ≈ 0.24. We finally discuss the effect of J on the Néel to large-D transition of model (11) based on SU(3) SBMFT calculations of model (11). In Fig. 4 we show the critical D at which a spin gap opens up signaling the formation of the paramagnetic large-D phase. We find that the critical D c = 3J obtained for J = 0 of the unfrustrated model, is strongly suppressed by J . Hence, magnetic frustration favors the large-D phase. The kink around J = 0.12J in Fig. 4 is related to the expected poor performance of the SU(3) SBMFT approach as D c is suppressed below J, D c < J. 11

V. CONCLUSIONS
The magnetic properties of the a − b planes of Mo 3 S 7 (dmit) 3 crystals can be modeled through a S = 1, XXZ+120 0 quantum compass model with singlespin anisotropy. 4 For the weak SMOC present in Mo 3 S 7 (dmit) 3 crystals 9 , quantum compass couplings and single-spin anisotropy play a minor role so that the effective model in this regime is just the isotropic S = 1 Heisenberg model on the honeycomb lattice. Hence, we predict that isolated honeycomb layers of Mo 3 S 7 (dmit) 3 are Néel ordered. However, we find that magnetic frustration induced by a next-nearest neighbor exchange coupling, J , favors the large-D phase.