An extended Dicke model, which includes atom-atom interactions and a driving classical laser field, is established for a Bose-Einstein condensate inside an ultrahigh-finesse optical cavity. A feasible experimental setup with a strong atom-field coupling is proposed, where most parameters are easily controllable and thus the predicted a second-order superradiant-normal phase transition may be detected by measuring the ground-state atomic population. More intriguingly, second-order phase transition from the superradiant phase to the “Mott” phase is also revealed. In addition, a rich and exotic phase diagram is presented.

• Received 18 March 2008

DOI:https://doi.org/10.1103/PhysRevA.78.023634