Generating coherence and entanglement with a finite-size atomic ensemble in a ring cavity

We propose a model to study the coherence and entanglement resulting from the interaction of a finite-size atomic ensemble with degenerate counter-propagating field modes of a high-Q ring cavity. Our approach applies to an arbitrary number of atoms N and includes the spatial variation of the field throughout the ensemble. We report several new interesting aspects of coherence and entangled behavior that emerge when the size of the atomic ensemble is not taken to the thermodynamic limit of N>>1. Under such conditions, it is found that the counter-propagating cavity modes, although in the thermodynamic limit are mutually incoherent and exhibit no one-photon interference, the modes can, however, be made mutually coherent and exhibit interference after interacting with a finite-size atomic ensemble. It is also found that the spatial redistribution of the atoms over a finite size results in nonorthogonality of the collective bosonic modes. This nonorthogonality leads to the super-bunching effect that the correlations of photons of the individual cavity modes and of different modes are stronger than those of a thermal field. However, we find that the correlations are not strong enough to violate the Cauchy-Schwarz inequality and to produce squeezing and entanglement between the modes. Therefore, we investigate the spectral distributions of the logarithmic negativity and the variances of the output fields. These functions determine squeezing and entanglement properties of the output cavity fields and can be measured by a homodyne technique. We find that the entanglement is redistributed over several components of the spectrum and the finite-size effect is to concentrate the entanglement at the zero-frequency component of the spectrum.


Introduction
Generation of continuous variable entangled states with atomic ensembles coupled to a radiation field has been intensively discussed both theoretically and experimentally in recent years [1,2,3,4,5,6,7,8]. Atomic ensembles are macroscopic systems composed of a large number of atoms, and therefore it is a common practice in the theoretical treatments to work in the thermodynamic limit which takes the number of atoms N inside an ensemble to infinity, N → ∞. Under this approximation, the collective atomic operators are often represented, by using the Holstein-Primakoff representation of angular momentum operators [9], in terms of mutually independent bosonic modes, called collective bosonic modes. A large number of studies of a such system have been carried out in searching for superradiance and quantum phase transitions [10,11,12]. The atomic ensembles have also been used to demonstrate the deterministic creation of nonclassical light fields in the interaction of atoms with a cavity field. Cavities, in particular microwave and ring cavities, provide efficient and controllable setting for a strong interaction between macroscopic atomic ensembles and the electromagnetic field [13,14,15,16]. For example, Parkins et al. [17] have demonstrated that atomic ensembles interacting collectively with laser fields inside a high-Q ring cavity can be unconditionally prepared in a two-mode squeezed state. The scheme, which is a generalization of the Guzman et al. [18] scheme to four-level atoms, is based on a suitable driving of the atomic ensembles with two external laser fields and coupling to a damped cavity mode that prepares the atoms in a pure squeezed (entangled) state. Similar schemes have been proposed to realize an effective Dicke model operating in the phase transition regime, to create a stationary subradiant state in an ultracold atomic gas [19]. This approach has also been considered as a practical scheme to prepare trapped and cooled ions in pure entangled vibrational states [20] and to prepare four ensembles of hot atoms in pure entangled cluster states [21,22]. Recently, Krauter et al. [23] have proposed to employ dissipation for generating a steady state entanglement between two distant atomic ensembles.
Studies of macroscopic systems composed of atomic ensembles interacting with a cavity field do not have to be confined to the thermodynamic limit. It has recently been demonstrated experimentally that small atomic ensembles could serve as a resource for quantum metrology and quantum information science [24,25,26,27]. This is the purpose of the present paper to consider a spatially extended finite-size atomic ensemble interacting with counter-propagating modes of a high-Q ring cavity. Special emphasis is given to identifying intrinsically finite-size effects. The approach adopted here is based on the solution of the master equation of an effective two-level system involving ground states of the four-level atoms forming the atomic ensemble. The approach has similarities with some previous treatments, except that we introduce a spatial dependence of the interaction of the cavity modes and the laser fields with the atoms.
The spatial dependence arises naturally in the interaction of the fields with a finitesize atomic ensemble [28,29], and the objective is to explore explicitly the issue of size effects in creation of coherence and entanglement in continuous variable systems. Examples of coherence processes are given to illustrate the effect of a finite size of the atomic ensemble on creation of an entanglement between bosonic modes of the system. We find that the dynamics of the finite-size atomic ensemble differs qualitatively from those given in the thermodynamic limit. The inclusion of finite-size effects leads to a wide variety of unusual features. In particular, we find that collective bosonic modes of a finite-size atomic ensemble are not in general orthogonal to each other. In the course of the derivation of an effective Hamiltonian of the system, we observe that one of the finitesize effects is to create a direct coupling between the counter-propagating cavity modes. The mode nonorthogonality that couples the counter-propagating modes can drastically modify the property of the system. The important modification is that the coupling lifts the degeneracy of the cavity modes and leads to significantly different statistical properties of the modes. We present solutions for the second-order statistical moments of different modes of the system and find that the mode nonorthogonality gives rise to phase locking between the cavity counter-propagating modes, which leads to interesting first-order coherence effects. We also study the second-order correlation functions of the counter-propagating modes and show that the nonorthogonality leads to the superbunching effect. In addition, we show that the nonorthogonality creates correlations that are necessary for entanglement between the intracavity modes. However, we find that the correlations created are not strong enough to produce and entangle between the cavity counter-propagating modes. We are therefore led to consider spectral distributions of the field variances and logarithmic negativity [30,31] and find that the two-mode squeezing and the entanglement can actually be created between spectral components of the output cavity fields.
The paper is organized as follows. In section 2, we describe in more detail the cavity and atomic ensemble under consideration. We derive an effective Hamiltonian of the system and show that the major finite-size effect is in the nonorthogonality of the collective bosonic modes. We then apply the Hamiltonian to derive the Heisenberg equations of motion for the field operators, and solve them in terms of the Fourier transform variables. Section 3 is devoted for the study of the mode nonorthogonality on coherence and entanglement properties of the counter-propagating cavity modes. In particular, in sections 3.1 and 3.2, we analyze the first and second order coherence, respectively, between the counter-propagating cavity modes. We pay particular attention to the role of the mode nonorthogonality in the creation of coherence and correlations between the modes. Spectral distributions of the logarithmic negativity and the variances of the output fields are considered in section 3.3, where we illustrate the possibility of the creation of entanglement between spectral components of the output fields of the cavity modes. A summary of results is presented in section 4. Finally, in the Appendix, we present analytical expressions for the steady-state mode occupation numbers, average amplitudes and correlations between the modes.

Atomic system and Hamiltonian
The model we are considering is illustrated in Fig. 1. It consists of an atomic ensemble located inside a high-Q ring cavity. The cavity field is composed of two degenerate in frequency and overlapped counter-propagating modes, called clockwise (R) and anticlockwise (L) modes, characterized by equal frequencies ω R = ω L ≡ ω c , and anti-parallel wave vectors k R = − k L ≡ k c , respectively. The modes are represented by operatorŝ a R (â L ) andâ † R (â † L ) which are, respectively, the annihilation and creation operators for the cavity clockwise (anti-clockwise) mode. Figure 1. A schematic diagram of a ring cavity containing an atomic ensemble trapped along the cavity axis. The clockwise and anti-clockwise cavity modes are damped with the same rate κ. The driving laser fields are injected through one of the cavity mirrors and co-propagate with the clockwise cavity mode. The output cavity modes are mixed at a 50/50 beamsplitter and detected by two photodetectors. The output photocurrents are then registered by the coincidence counter CC.
The atomic ensemble is composed of N identical four-level atoms interacting with external driving fields and a cavity field. An atom of the ensemble, say jth one, is represented by two non-degenerate ground states |0 j , |1 j , two non-degenerate excited states |u j , |s j , and its position r j , as illustrated in Fig. 2. In practice such a four-level system could correspond to an F = 1 ↔ F ′ = 1 transition as occurs in 87 Rb atoms.
The cavity modes couple equally, i.e., with the same coupling strengths g R = g L ≡ g, to atomic transitions |0 j → |u j and |1 j → |s j . This is acceptable since the degenerate overlapped cavity modes have the same polarization and geometry [13,14,15]. In addition, the atomic ensemble is driven by pulse laser fields injected through one of the cavity mirrors and co-propagating with one of the cavity modes. The lasers are characterized by frequencies ω ls and ω lu , wave vectors k ls = k lu ≡ k l , and drive atomic transitions |0 j → |s j and |1 j → |u j , with Rabi frequencies Ω s and Ω u , respectively.
Energy level scheme of the atoms and coupling configurations of the laser fields and the cavity modes. The laser fields of the Rabi frequencies Ω u and Ω s drive the atomic transitions |1 j → |u j and |0 j → |s j , respectively. The atomic transitions |1 j → |s j and |0 j → |u j are coupled to the cavity modes with the coupling strengths g u = g s = g.
The total Hamiltonian for the atoms and the cavity modes can be written aŝ is the free Hamiltonian of the cavity modes and the atoms, is the interaction Hamiltonian between the atoms and the driving fields, and â R e i kc· r j +â L e −i kc· r j |u j 0 j | + â R e i kc· r j +â L e −i kc· r j |s j 1 j | + H.c.
is the interaction Hamiltonian between the atoms and the two cavity modes. Here, φ u and φ s are phases of the laser fields, and ω 1 , ω s and ω u are atomic frequencies, corresponding to transitions |1 j ↔ |0 j , |s j ↔ |0 j , and |u j ↔ |1 j , respectively. We have put zero energy at the ground state |0 j . As we shall be interested in the generation of entanglement that requires minimal losses in the system, we consider an effective Hamiltonian in a dispersive regime that determines dynamics only between the ground states of the atoms. In this case, the cavity modes and the laser fields induce transitions between the ground states of the atoms via virtual transitions to far-off-resonant upper states. The effective Hamiltonian readsĤ Detailed derivation of the effective Hamiltonian together with the definitions of the parameters involved is presented in Appendix A. Note that the advantage of working in the dispersive limit of large detunings ∆ u and ∆ s is to avoid spontaneous emission from the upper atomic states. In the derivation of (5), we have assumed further that the detunings ∆ u and ∆ s are much larger than the splitting of the ground states, i.e. ∆ u , ∆ s ≫ ω 1 . This allows us to ignore decoherence of the ground states due to elastic Rayleigh scattering. Recently, Uys et al. [32] have demonstrated, both theoretically and experimentally, that in the case of the detunings ∆ u and ∆ s comparable to the splitting of the ground states may result in a considerable Rayleigh decoherence in the system. Among many parameters involved in the Hamiltonian (5), the most important for the purpose of the present paper is the parameter which stands for the strength of the direct coupling between the cavity modes. The coupling is caused by the spatial variation of the cavity modes that arises from the interaction of the modes with the finite-size atomic ensemble. The spatial variation is completely determined by the parameter α k , which is of the from This position dependent factor is recognized as the usual phase matching condition and represents an effective spread in phase difference between the cavity modes at r j . It follows that the factor will be different from zero when N is not too large and r j are small. It is easy to establish that the factor vanishes in the thermodynamic limit of N → ∞. The Hamiltonian (5) describes the interaction of a collection of N two-level systems with the cavity counter-propagating modes. It involves linear interaction terms, proportional to β u , as well as nonlinear interaction terms, proportional to β s . Generally speaking, there are three different types of virtual transitions in the atoms; one is due to absorption of a photon of frequency ω ls from a pulse laser accompanied by the emission of a photon to either R or L cavity mode. This process takes the atom from the state |0 j to the state |1 j . The second process is due to absorption of a photon of frequency ω lu from a pulse laser accompanied by the emission of a photon to either cavity mode R or L. This process takes the atom from the state |1 j to the state |0 j . Finally, the third process is due to absorption of a photon from either R or L cavity mode accompanied by the emission of a photon of the same frequency to the counter-propagating mode. This process does not change the state of the atom.
The later process is the most interesting, because it is related to finite-size effects and is not encountered at all under the thermodynamic limit of N → ∞. It shows that, after the interaction with the finite-size atomic ensemble, there is generally mutual coherence between the cavity modes. The parameter α k δ characterizes the strength of the coupling between the cavity modes and expresses the coherent exchange of photons between the modes. This simply reflects the presence of a phase relation between the counter-propagating cavity modes. The efficiency of the coupling depends on the parameter α k which, according to (7), is given by the phase mismatch of the propagation vectors of the cavity modes evaluated at the position of the individual atoms. The dependence of α k on the phase mismatch factor k R − k L = ±2 k c indicates that for a given cavity mode, the other mode can be viewed as a 'phase-conjugate' field of the mode. The coupling happens because the counter-propagating cavity modes force an atom to move in the opposite directions. Since for a finite-size ensemble the force depends on the position of the atom, it creates a potential energy between atoms located at different positions. The energy averages to zero in the limit of N → ∞ due to a random redistribution of the atoms inside the atomic ensemble.
Another interesting feature of a finite-size of the atomic ensemble is in the spatial dependence of the interaction between the atoms and the cavity fields that the multi-atom operatorsĴ ±k ,Ĵ † ±k andĴ z do not satisfy the standard angular momentum commutation relations. The reason is in the presence of the phase factors exp[i( k l ± k c ) · r j ], which arise from the phase mismatch between the propagation direction of the cavity modes and directions of the laser fields. These factors represent an effective spread in phase difference between the laser and cavity fields at r j . As a consequence, the interaction is affected in a different way than the cavity modes. Moreover, the presence of two different phase mismatch factors indicates that the atomic ensemble may be coupled to the cavity modes in two distinctly different ways.
In order to explore this feature more explicitly, we adopt the Holstein-Primakoff representation of angular momentum operators [9], in which the two collective atomic operators,Ĵ ±k , are expressed in terms of annihilation operatorsĈ ±k of the corresponding bosonic modes as follows: are collective bosonic operators with the annihilationb j and creationb † j operators obeying the standard bosonic commutation relation It is easily verified that the collective bosonic operators do not in general commute, i.e.
Again, the reason is in the presence of the position dependent phase factors. Hence, the collective modes of a finite-size atomic ensemble are not orthogonal to each other. The degree of nonorthogonality of the modes is determined by the phase matching parameter α k , and the modes become orthogonal in the thermodynamic limit of N → ∞.
The commutation relation can also be viewed as a non-distinguishability criterion for the collective modes. In the thermodynamic limit, α k = 0, and then the modes are completely distinguishable. For α k = 0, the modes are partly distinguishable and become completely indistinguishable when α k = 1. As we shall see below, the non-orthogonality and thus indistinguishability of the modes will result in correlations between different modes of the system.
Before proceeding further, we note here that the bosonic representation of the collective atomic operators of a finite-size atomic ensemble places no restriction on the number of atoms composing the ensemble [33]. The representation is valid for an arbitrarily small number of atoms with the condition of a very low excitation probability of each atom, i.e. σ j 11 ≪ 1, where σ j 11 = |1 j 1 j |. The effective Hamiltonian (5) expressed in terms of the collective bosonic operators describes the interaction of a "fictitious" bosonic system with the two-mode cavity field, and has the form Note that the Hamiltonian is symmetric under reversal of the direction of propagation of either the laser field or the cavity mode. Instead of working with the collective operatorsĈ ±k , we shall find convenient to work with two operatorŝ which are linear symmetric and antisymmetric superpositions of the bosonic collective operators, respectively. It is easily checked that the superposition operators are orthogonal to each other and obey the standard bosonic commutation relations, In terms of the superposition operators (12), the effective Hamiltonian (11) simplifies tô where are in-phase and out-of-phase quadrature components of the cavity modes and the bosonic field operators, witĥ It is seen from (13) that one of the finite-size effects on the system is to lift the degeneracy of the cavity modes by creating linear symmetric and antisymmetric superpositions of the modes with frequencies ω + α k δ and ω − α k δ, respectively. It is also seen that the superposition collective modes are degenerate in frequency, but they do not behave similarly. The modes couple to the cavity superposition modes with different coupling strengths. The symmetric moded 1 couples to the cavity modeâ 1 with an enhanced coupling strength λ 1 = β √ 1 + α k , whereas the antisymmetric modê d 2 couples to the modeâ 2 with a reduced strength λ 2 = β √ 1 − α k . It is interesting to note that the pairs of modes (â 1 ,d 1 ) and (â 2 ,d 2 ) are decoupled from each other. This means that each pair can be independently prepared in a desired state. While the cavity superposition modes result from the linear coupling between the cavity counterpropagating modes, the collective bosonic modes couple to the cavity modes in linear as well as in a nonlinear way. This is the nonlinear coupling that may create entanglement between the cavity and the collective bosonic modes.
In the physical terms, the Hamiltonian (13) contains terms describing four-wave mixing of the up-shifted (signal) and down-shifted (idler) cavity modes with the degenerate collective modes. Other terms proportional to the products of creation and annihilation operators for the same mode result in a dispersive effect. There are also terms that couple creation and annihilation operators of the cavity modes with the creation and annihilation operators of the collective modes. This interaction is responsible for the back-action evading nature of quantum non-demolition detection.
It is worthwhile noting that, in spite of the fact that the finite-size feature of the system is manifested by the presence of three phase mismatch factors, the difference between the Hamiltonians of finite-and infinite-size atomic ensembles is embodied in a single parameter α k . In other words, the dynamics of the system are independent of the direction of propagation of the laser fields. They depend solely on the phase mismatch of the cavity counter-propagating modes. It is only the presence of α k that pulls of the degenerate cavity modes above and below their resonance by equal amounts, δ c , and introduces an asymmetry to the coupling constants of the collective bosonic modes to the cavity modes.
Before moving on to the consideration of coherence and correlation features in the system, we first briefly comment about the threshold behaviour of the Hamiltonian (13). One can notice that the Hamiltonian (13) is of the form of two independent one-mode Dicke modelsĤ with Ω 1 = ω + α k δ and Ω 2 = ω − α k δ. Hence, many features predicted previously by other authors for the one-mode Dicke model in the thermodynamic limit can also be seen in our model [10,17,18]. However, instead of focusing on these one-mode features, we prefer to specialize our considerations to novel features of the two-mode Dicke model that might be brought by finite-size effects. For example, there might be coherence and correlations existing between modes that are simultaneously involved in both Dicke models. We examine these properties shortly, but first we examine a manifestation of the finite-size effects in the threshold behaviour of the system. It is easy to see that in the case of α k = 0, the coupling strength λ 1 = λ 2 . As a consequence, there are two rather than one critical values of β: Thus, an interesting notable feature of the finite-size effects is the existence of two distinctive critical values of the coupling strength β. It is easily verified that the critical values β c1 and β c2 shift in opposite directions as α k increases. Note that in the limit of α k → 1, β c1 approaches a finite value, whereas β c2 goes to infinity. The existence of the two threshold values for β indicates that the properties of the system could be different for different values of β. It what follows, we confine our considerations to the case of below the thresholds, i.e. β < β c1 .

Coherence and entanglement induced by the finite-size effects
We now proceed to discuss the coherence and correlation features of the cavity modes brought by the finite-size effects of the atomic ensemble. As we have already mentioned, coherence and correlations can be created between different modes of the system. Here, we confine ourselves to the study of the coherence and correlations of the cavity counterpropagating modes only. The reason is that properties of the cavity modes can be directly measured by detecting of the out-put cavity fields. The coherence properties of the other modes of the system can be found from the properties of the out-put cavity fields.
In order to keep the considerations close to practical situations, we include a possible loss of cavity photons due to the damping of the cavity mode. With the cavity damping included, the state of the system is a statistical mixture determined by the density operator ρ whose time evolution is governed by the master equatioṅ whereĤ e is given in (13) and κ is the cavity damping rate. This is the only damping we consider here as we have already eliminated spontaneous emission from the atoms by choosing large detunings of the driving lasers and the cavity modes.
Our treatment is based on the solution of the Heisenberg equations of motion for the mode operators that are readily obtained from the master equation (19), and are given byȧ along with the corresponding equations for the adjoint operators. In these equations, the operatorâ in j (t) describes the quantum noise injected at the cavity input. It is easy to show that the set of differential equations for the mode operators splits into two independent sets, each composed of four coupled differential equations. The sets of the differential equations are conveniently solved by taking the Fourier transform of the operators whereû denotes any one of the operatorsâ 1 ,â 2 ,d 1 ,d 2 andâ in 1 ,â in 2 . Expressed as equations for the transforms of the operators, the solution for the mode operators may be written asd where The frequency dependent solution (23) will be used for the calculations of the collective modes and the field correlation functions necessary for evaluation of the two-mode squeezing and entanglement spectra.
Since we are also interested in the steady-state coherence between the modes, we transform the solutions (23) back to the time domain and take the steady-state limit. Then all the functions necessary for the explicit calculation of the coherence are obtained by the average over the initial vacuum state with zero occupation numbers for all the modes of the system. The steady-state solution for the field averages and correlation functions are listed in the Appendix B.

First-order coherence of the cavity modes
First, we consider the first-order mutual coherence between the counter-propagating cavity modes. The mutual coherence between the cavity modesâ R andâ L is measured by the cross correlation â † Râ L , the so-called coherence function, where the average is taken over the initial vacuum state of the modes [34]. The degree of coherence between the modes is given by The visibility V of the interference pattern, on the other hand, is given by The degree of coherence and the visibility of the stationary cavity fields can be readily calculated using the steady-state solutions (B.1). Since we see that the visibility equals to the degree of coherence independent of the parameters of the system. Moreover, where with We see that the coherence function depends directly on the finite-size parameter α k . This implies that the cavity modes are correlated only when α k = 0. The mode nonorthogonality can transfer photons from one mode to the other. Thus, one of the aspects of finite-size effects is the creation of the first-order correlation between the cavity counter-propagating modes. This feature is not encountered at all under the thermodynamic limit of N → ∞. In physical terms, we may attribute the appearance of the coherence to the fact that the counter-propagating modes are unresolved at the atomic ensemble, that it is impossible to tell to which mode the photon was emitted.  Figure 3 illustrates variation of the steady-state degree of coherence |γ (R,L) | with β for several different values of α k . It is seen that after the interaction with the finite size atomic ensemble, there is a non-zero mutual coherence between the counterpropagating cavity modes. The coherence increases with α k and the modes become perfectly correlated, |γ (R,L) | → 1 as α k tends towards unity. Moreover, the coherence becomes less sensitive to β as α k increases. In addition, for α k ≈ 1 the coherence attains its maximal value of |γ (R,L) | = 1 independent of β. Notice, that the threshold value of the coupling strength at which |γ (R,L) | approaches unity, shifts towards smaller β as α k increases. The threshold value of β corresponds to a critical value of β.
We close this section by evaluating the degree of coherence in the case of α k = 1, that is, when the size of the atomic ensemble is much smaller than the resonant wavelength of the cavity modes, i.e., when 2 k c · r j ≪ 1. To consider that limit, we can return to the effective Hamiltonian and note that with α k increasing to the value unity, the collective mode d 2 becomes decoupled from the cavity field mode a 2 . In this case, the effective Hamiltonian reduces to that of a single-mode Dicke system involving only the field mode a 1 and the collective mode d 1 . The modes a 1 and d 1 undergo the time evolution, whereas the modes a 2 and d 2 remain constant in time that they retain their initial values for all times. Although the behaviours of the system effectively as a single-mode Dicke system, it, in fact, involves two modes since the mode a 1 is a superposition of the cavity counter-propagating modes.
The steady-state solutions (B.1) are not valid for α k = 1. However, almost all mode correlation functions can be obtained from (B.1) by putting λ 2 = 0 except of â † 2â 2 and d † 2d 2 . These two correlation functions are constants of motion when α k = 1. As a result, the modes do not evolve in time, they retain their initial values for all times t. Thus, if initially the modes were unpopulated, they will stay unpopulated for all times. This somewhat unusual result is a consequence of the fact that the modes are totally decoupled from the applied field. The immediate consequence of the decoupling of the modesâ 2 andd 2 from the field is the appearance of the perfect correlation between the cavity counter-propagating modes. It is easy seen from (27) and (29) that in this case, the degree of the first-order coherence |γ (R,L) | = 1 irrespective of the parameters involved. This result has a simple interpretation, the modeâ 1 that can be prepared in an arbitrary state, is a linear superposition of the cavity counter-propagating modes that enter with equal weights. Therefore, both modes are always equally prepared, so that cannot be resolved, which is reflected in the coherence equal to unity.

Second-order coherence of the cavity modes
We now consider the second-order correlation functions of the fields of individual modes and of two different modes. We are particularly interested in the correlations in the cavity counter-propagating modes, represented by the operatorsâ R andâ L , and between these modes. The correlations are determined by â † Lâ RâL , respectively. More specifically, correlation functions describe the photon statistics of the field of the individual modes, and the cross correlations between photons from two different modes. As we shall see below, the second-order correlation functions, especially the cross correlation function â †

Râ †
Lâ RâL can be used to determine nonclassical effects and entanglement between the modes [35,36].
We shall consider normalized correlation functions and assume that the cavity modes are Gaussian-state modes that they obey the moment-factorization rules of a Gaussian random variable [37]. This is to be expected, since the collective bosonic modes are the sum of a large but finite number of atoms N, whose fluctuations have been supposed to be statistically independent. Therefore, the Gaussian form of the collective bosonic modes is expected to be reflected in a Gaussian form of the other modes. With this condition, the normalized second-order correlation functions can be written as where γ (R,L) is the degree of the first-order coherence, defined in equation (25), and are degrees of the so-called "anomalous" coherence [38,39,40,41,42]. We see that the most important contribution to the second-order correlation functions comes from the anomalous coherence functions. These relations also show that g RR ≥ 2 and g LL ≥ 2, which means that photons emitted in the same direction, R or L, are always strongly correlated. This is a reflection of the Gaussian statistics of the cavity modes that the four-order moments â †
Let us examine the dependence of the correlation functions on the ensemble size parameter α k . For purposes of an explicit analytical analysis, it is somewhat more convenient to rewrite (25) in terms of the superposition cavity modes 1 and 2. With the help of (15), we arrive at the expressions We observe that the inter-mode coherence function |γ (R,L) | depends on the difference between the number of photons in the superposition modes, whereas the inter-mode coherence function |η (R,L) | depends on the difference between the anomalous coherence functions of the modes. Thus, some kind of asymmetry between the superposition modes is needed to create the coherence between the cavity counter-propagating modes. The coherence (32) and (33) can be readily evaluated using the steady-state solutions (B.1).
Consider first the correlation functions in the thermodynamic limit, in which case α k = 0. From the steady-state solutions, (B.1), it follows that in the limit of α k = 0, â † 1â 1 = â † 2â 2 , â 2 1 = − â 2 2 , and | â 2 and from (31), we immediately obtain that LL = 2, and g (2) These results show that in the thermodynamic limit the cavity modes and the correlation between the modes exhibit correlations characteristic of a thermal field. It is, of course, a reflection of the fact that the system operates below the threshold where the modes are in thermal states. This is the kind of behavior that is expected for the cavity modes. One could argue that the same circumstances apply for the presence of the correlations between the modes. However, the circumstances for the second-order correlations g (2) RL = 2 are different. The source of the correlation between the modes is not in the thermal fluctuations, as it takes place in the well-known Hanbury-Brown-Twiss effect, but is in the impossibility to distinguish from which mode each of the two photons came. This is represented by the anomalous coherence |η (R,L) |, and for this reason, we could call this effect as an anomalous Hanbury-Brown-Twiss effect.
We have already seen that in the thermodynamic limit only the mutual anomalous coherence η (R,L) is different from zero and, in fact, attains its maximal value of unity. It is clear by inspection of (32) that an asymmetry between the superposition modes is required to get all of the coherence different from zero. It is easily verified from (B.1) that the required asymmetry is provided by the ensemble size parameter α k . In this case, the second-order correlations can be larger than that for the thermal field. This is known in the literature as a super-bunching effect [48,49,50,51]. Hence, α k = 0 is the general condition for the super-bunching effect. The variation of the correlation functions with α k for several different values of β is illustrated in figure 4. It is evident that the finite-size effects enhance the correlations between photons emitted in the same as well as in the opposite directions.
The largest value of the correlations is achieved when α k = 1, in this case g RL = 3. This value is the border value between classical and nonclassical Gaussian states [49,52]. We may conclude that the output cavity modes behaviour as an unusual classically correlated reservoir which exhibits strong classical correlations simultaneously inside the individual modes and also between the modes. Typical sources of correlated beams, such as optical parametric oscillators exhibit correlations stronger than that of a thermal field only between the modes.
One can also notice from the figure 4 that the correlations functions g (2) RR and g (2) LL behaviour similarly to the mutual correlation function g (2) RL . However, there is a relation between the correlation functions, given by the Cauchy-Schwartz inequality [34] χ RL = g RR g  Figure 4. The stationary second-order correlation functions g (2) RR (left frame) and g which says that the cross correlations between photons from the two different cavity modes are smaller than the correlation between photons of the individual modes. An interesting question arises whether the Cauchy-Schwartz inequality can be violated in the system.  Figure 5 illustrates the variation of the Cauchy-Schwartz parameter χ RL with α k . We see that even when the correlations functions g (2) RR and g (2) LL behaviour similar to the mutual correlation function g (2) RL , the Cauchy-Schwartz parameter varies with α k . It is apparent that χ RL is always more than or equal to unity for all α k , with equality at α k = 0 and α k = 1, indicating that the Cauchy-Schwartz inequality is not violated. Thus, the finite-size effects create strong correlations between the cavity modes but do not allow the relation (36) to be violated.
It is not difficult to show from (31) and (32) that the inequality (36) is equivalent to the inequality η (R,L) ≤ 1, that for the Cauchy-Schwartz inequality to be satisfied, the mutual anomalous coherence must be smaller than unity. Thus, for a violation of the Cauchy-Schwartz inequality it is necessary that the degree of the mutual anomalous coherence to be larger than unity. It is worth noting that such values can be achieved only by a quantum field [34,53,36]. Although the Cauchy-Schwartz inequality is not violated between the cavity modes, it may be violated between other modes. Figure 6 shows Cauchy-Schwarz parameters χ 11 and χ 22 defined as which provide measures of the second-order correlations between photons from two superposition modes (â 1 andd 1 ), and from other two superposition modes (â 2 andd 2 ), respectively. The correlations are said to violate the Cauchy-Schwartz inequality if χ ii (i = 1, 2) is smaller than unity. It is seen that the Cauchy-Schwartz inequality is violated for both pairs of the modes indicating a strong nonclassical correlation between the superposition modes. These violations exist even for α k = 0 and decrease with increasing pumping strength β. The physical reason for the violation of the Cauchy-Schwartz inequality can be traced to nonlinear processes that are known to produce quantum effects in the interaction between bosonic modes [54]. It is evident from the effective Hamiltonian (13) that such processes exist in the system. There is a nonlinear coupling between modesâ 1 andd 1 , and between modesâ 2 andd 2 . The effect of these nonlinear couplings is to produce nonzero anomalous correlation functions that are responsible for enhanced inter-mode correlations.
In summary of this section, we have found that the role played by the finite-size effects in the second-order correlations is principally to create correlations which are larger than that achievable with thermal fields. However, there is a limitation on the values of the second-order correlations that could be created by the finite-size effects. The second-order correlation functions vary with the finite-size parameter α k from g (2) ij = 2 for α k = 0 to the maximum of g (2) ij = 3 for α k = 1, which is achieved when the dimensions of the atomic ensemble are much smaller that the resonant wavelength. These results show that the total field emerging from the cavity is a classical but strongly correlated thermal field. It was found that quantum effects such as the violation of the Cauchy-Schwartz inequality can be created between the superposition modes. Unfortunately, the quite large violations of the Cauchy-Schwartz inequality for the superposition modes do not lead to violation of the Cauchy-Schwartz inequality for the correlations between the cavity counter-propagating modes.

Entanglement and two-mode squeezing spectra of the output cavity modes
Since the finite-size effects create first-order coherence and the second-order correlations between the modes, there actually could be two-mode squeezing and entanglement between the modes associated with a nonlinear coupling between the modes as well. An inspection of the effective Hamiltonian (13) reveals that a nonlinear coupling actually exists only between modesâ 1 andd 1 , and between modesâ 2 andd 2 . Thus, the modes (â 1 ,d 1 ) and (â 2 ,d 2 ) could be entangled between themselves. This suggests that the other pairs of the modes cannot be entangled. We now examine the possibility to create two-mode squeezing and entanglement between the cavity counter-propagating modes a L and a R and how these effects could depend on the finite-size parameter α k .
In order to find if entanglement and two-mode squeezing can be created between the cavity modes a R and a L modes, we introduce the position and momentum operators for the annihilation operators of the superposition modes,â 1 andâ 2 , which can be defined as where θ is the quadrature phase. To see if an entanglement exists between the cavity counter-propagating modes, a R and a L , we use a condition based on the two-mode squeezing, proposed by van Loock and Furusawa [55]. By use of the mode transformations for the modes a L , a R a 1 , and a 2 , (15), the sufficient condition for the entanglement between the two cavity modes a R and a L is of the form [55] : ∆X θ where the normally ordered variances are given by with ϕ j = arctan(κ/ω j ). Here, the double colon :: stands for the normal ordering of the operators. When we evaluate the normally ordered variances (40) using the steady state solutions (B.1), we then easily find that : ∆X θ Thus, there is no squeezing between the cavity modes. Equivalently, the cavity modes are separable. We may conclude that any measurable criterion predicts no squeezing and thus no entanglement of the total field of the counter-propagating cavity modes a L and a R . On the other hand, from (28) and (32), we see that â † Râ L = 0, and â RâL = 0 when α k = 0. This means that the modes are correlated and the strength of correlation depends on the values of α k . Hence, the modes are correlated but not strong enough to be squeezed (entangled). This conclusion agrees with our previous findings that the modes are correlated to the degree of g (2) RL = 3, which is the border value between classical and nonclassical Gaussian states. This means that the correlations created in the pairs (â 1 ,d 1 ) and (â 2 ,d 2 ) can be transferred to the cavity modes, but are not strong enough to entangle the modes.
We stress that the calculated correlations corresponded to that of the total cavity field. It is well known that in some situations there is no squeezing in the total field, but there could be squeezing between spectral components of the field [56,57]. In other words, the question of whether the total output field is squeezed (entangled) may be irrelevant to the problem of obtaining large amount of squeezing (entanglement) at some particular spectral frequency. Nevertheless, we shall show that a strong squeezing (entanglement) exists between spectral components of the output cavity fields.
We now use the frequency dependent solutions for the cavity modes and the relations between the input and output fields [58] a out whereã j (ν) are the intracavity field operators andã in j (ν) are the input noise operators. We shall use the frequency dependent operators to calculate the measurable spectra of the output fields transmitted by one of the cavity mirror with decay constant κ. Let ν is the frequency of the output clockwise (R) mode and ν ′ is the frequency of the output anti-clockwise (L) mode. We may introduce finite frequency intervals δν and δν ′ defined as such that at δν = δν ′ the modes are symmetrically located about 2ω l , i.e. ν + ν ′ = 2ω l . We consider spectral distributions of the variances whereX θ 1 (ν) andP θ 2 (ν ′ ) are Fourier transforms of the quadrature phase operators of the superpositionsâ 1 andâ 2 of the output modes, defined as We also consider the spectral distribution of the logarithmic negativity criterion that is known as the necessary and sufficient condition for entanglement of two-mode Gaussian states [59,60] where V s (ν) is the smallest sympletic eigenvalue of the partially transposed covariance matrix of the output field. We shall compare the criterion with the two-mode squeezing criterion to quantify squeezing as an alternative necessary and sufficient condition for entanglement [61]. The advantage of the two-mode squeezing criterion over the negativity is that the former can be directly measured in experiments whereas the later can be inferred from the reconstruction of the density matrix of the system. To evaluate E n (ν), that describe entanglement of a two-mode output Gaussian state, we use Wigner characteristic function where ξ = (ξ * a R , ξâ R , ξ * a L , ξâ L ) is a vector of complex variables, ξ T is the transposed for of ξ, and V (ν) is the covariance matrix of the form with This shows that f 2 (ν), f 3 (ν) and f 4 (ν) determine the correlation between the two output cavity modes. Having the covariance matrix, we may discuss in detail the establishment of entanglement between two output cavity modes. We shall be particularly interested in the role of the finite-size effects on the output entanglement of the two counterpropagating cavity modes. Figure 7 shows the spectral distribution of the logarithmic negativity and the variances of the output fields for different α k . We also vary the phase θ due to varying of the optimal squeezing with an increasing α k . First, we note that independent of α k , it is possible for S(ν, θ) to be negative for some frequencies, so that S(ν, θ) dips below the quantum limit at those frequencies, even though there is no two-mode squeezing in the total field. Moreover, we see that entanglement occurs for all frequencies and the maxima of entanglement correspond to the minima of the variances S(ν, θ) and S(ν, θ + π/2). The maxima of entanglement occur at frequencies corresponding to the imaginary parts of the roots of the D j (ν) polynomials. Note that the entanglement that lies in the range of low frequencies (|ν| ≤ 1) is attributed to squeezing in the θ quadrature component of the output field, S(ν, θ), whereas the entanglement that lies in the range of high frequencies (|ν| > 1) is attributed to squeezing in the θ + π/2 component S(ν, θ + π/2). It is interesting to observe that the cavity modes can be entangled regardless of the size of the atomic ensemble. However, the frequency range at which the modes are entangled varies with the finite-size parameter α k , i.e., the modes can be entangled at several different frequencies. When α k = 0, the logarithmic negativity and the variances can both have four peaks. When α k = 0, the peaks merge towards δν = 0. In this case, two-mode squeezing occurs only in the θ quadrature. It follows that with a finitesize ensemble, the largest entanglement is observable in principle at the frequencyω l . Thus, the size effect is to concentrate the entanglement at the central component of the spectrum. In other words, an optimal degree of squeezing (entanglement) at δν = 0 indicates that the output modes are correlated about the average laser frequencyω l . Figure 8 illustrates how the pumping strength β alters spectral redistribution of two-mode squeezing and entanglement. Again, each curve is plotted with θ optimized to give best squeezing. It can be seen that only the low frequency entanglement and two-mode squeezing (|ν| ≤ 1) shifts towards the central frequency as β increases. Notice significant variations of the low frequency entanglement and squeezing with relatively small variations of β. In contrast, the high frequency entanglement and squeezing are almost insensitive to β. The effect of raising β causes only a slight shift of the peaks. Thus, the entanglement and two-mode squeezing associated with the θ + π/2 quadrature component are less sensitive to β than those associated with the θ quadrature component. Nevertheless, this does not prevent us from achieving the largest entanglement at the cavity frequency.

Conclusions
We have shown that finite sizes of atomic ensembles coupled to counter-propagating modes of a high-Q cavity do have a non-negligible effect on coherence, correlations and entanglement between the cavity modes. In particular, we have shown that finite sizes of the atomic ensemble may result in nonorthogonality of the collective bosonic modes. We have found that the mode nonorthogonality can create the first-order coherence between the modes and appears as the transfer mechanism of the fluctuations between the superpositions of the cavity counter-propagating modes. We have shown that the mode nonorthogonality may result in the second-order correlations that are stronger than that of a thermal field. The correlations are manifested in the photon super-bunching effect. In addition, the nonorthogonality creates correlations between the modes that are necessary for two-mode squeezing and entanglement between the modes. However, we have found that the correlations created are not strong enough to violate the Cauchy-Schwartz inequality and to produce entanglement between the modes. Therefore, we have also considered the spectral distributions of the logarithmic negativity and the variances of the output cavity fields and have found that entanglement, although not present in the total field, can be created between spectral components of the output cavity fields. The effect of the mode nonorthogonality is to concentrate the entanglement at the central component of the spectrum.
In this appendix we shall discuss in detail the derivation of the effective Hamiltonian (5). The method is based on the adiabatic elimination of the excited atomic states valid in the limit of large detunings of the laser fields and cavity mode frequencies from the atomic transition frequencies (dispersive regime). The method is well known in the literature and more details can be found in Refs [62,63,64].
We shall derive an effective Hamiltonian in a dispersive regime of the interaction of the atoms with the cavity modes and external laser fields. Employing the unitary transformationÛ (t) = exp[−i(Ĥ ′ 0 / )t] to the total Hamiltonian (1), whereĤ ′ 0 is given bŷ â R e i kc· r j +â L e −i kc· r j |u j 0 j | + â R e i kc· r j +â L e −i kc· r j |s j 1 j | + H.c. , (A. 5) in which Ω u ( r j ) = Ω u e i( k l · r j −φu) , Ω s ( r j ) = Ω s e i( k l · r j −φs) , (A.6) are the position dependent Rabi frequencies of the laser fields, are detunings of the atomic transition frequencies and of the cavity frequency from the laser field frequencies, with ω l = 1 2 (ω ls + ω lu ), ω d = 1 2 (ω ls − ω lu ), (A.8) standing for the average frequency and detuning between the laser frequencies, respectively. We may extract a part of the HamiltonianH 0 that involves the upper states of the atomsH 2 Ω u ( r j )|u j 1 j | + g â R e i kc· r j +â L e −i kc· r j |u j 0 j | e i∆ut + 1 2 Ω s ( r j )|s j 0 j | + g â R e i kc· r j +â L e −i kc· r j |s j 1 j | e i∆st + H.c. The first-order contribution is of the form 2 Ω u ( r j )|u j 1 j | + g â R e i kc· r j +â L e −i kc· r j |u j 0 j | e i∆ut ∆ u + 1 2 Ω s ( r j )|s j 0 j | + g â R e i kc· r j +â L e −i kc· r j |s j 1 j | e i∆st ∆ s + . . . (A.12) in which the first term is linear in time and the other terms oscillate with detunings ∆ u and ∆ s , respectively. The second-order contribution can be written as gΩ * u ( r j ) ∆ u â R e i kc· r j +â L e −i kc· r j |1 j 0 j | + H.c.
If we assume that the detunings of the laser fields are much greater than the Rabi frequencies, the cavity coupling constants and the atomic spontaneous emission rates |∆ u |, |∆ s | ≫ Ω u , Ω s , g, γ, (A.14) where γ is the spontaneous emission rate of the excited states of the atoms, the oscillatory terms make a negligible contribution to the time evolution operator and, after discarding them, we obtainÛ I (t) ≈ 1 − itĤ e / , wherê (|1 j 1 j | − |0 j 0 j |) ,Ĵ ±k = N j=1 |0 j 1 j |e i( k l ± kc)· r j (A. 16) are position dependent collective atomic operators, are detunings of the cavity field frequency and of the atomic frequency ω 1 from the laser frequencies modified by the intensity-dependent Stark shifts, are the effective Rabi frequencies which quantify the strength of the coupling of the effective two-level system to the cavity modes due to virtual transitions to the highly detuned |u j and |s j states, and α k δ = |α k | Ng 2 ∆ , (A. 19) with α k = |α k |e ±iφ N = 1 N N j=1 e ±2i kc· r j . (A. 20) In the derivation of Eq. (A.15), we have chosen ∆ u = ∆ s ≡ ∆, which involves no loss of generality, and have redefined the cavity field operators that now read a R ≡â R exp(−iφ N /2) andâ L ≡â L exp(iφ N /2). We have assumed further that the laser phases φ u = −φ s = φ N /2, where the phase φ N is defined in (A.20). It should also be noted here that the assumption of equal detunings of the laser fields and the cavity modes from the atomic upper states, as illustrated in Fig. 2, gives ∆ c = ω 1 − ω d . However, due to the presence of the Stark shifts, this condition is modified to ω = ω 0 .