We investigate the near-critical dynamics of atomic density fluctuations in the nonequilibrium self-organization transition of an optically driven quantum gas coupled to a single mode of a cavity. In this system cavity-mediated long-range interactions between atoms, tunable by the drive strength, lead to softening of an excitation mode recently observed in experiments. This phenomenon has previously been studied within a two-mode approximation for the collective motional degrees of freedom of the atomic condensate, which results in an effective open-system Dicke model. Here, including the full spectrum of atomic modes we find a finite lifetime for a rotonlike mode in the Bogoliubov excitation spectrum that is strongly pump dependent. The corresponding decay rate and critical exponents for the phase transition are calculated explaining the nonmonotonic pump-dependent atomic damping rate observed in recent experiments. We compute the near-critical behavior of the intracavity field fluctuations that has been previously shown to be enhanced with respect to the equilibrium Dicke model in a two-mode approximation. We highlight the role of the finite size of the system in the suppression of it below the expectations of the open Dicke model.

• Received 21 June 2013

DOI:https://doi.org/10.1103/PhysRevLett.111.220408