A self-consistent theory of solitary hole structures in coasting beams is presented. These phase space vortices are known from particle simulations and appear, e.g., due to a resistive wall instability. The analysis reveals new intrinsic nonlinear modes which owe their existence to a deficiency of particles trapped in the potential well, showing up as notches in the thermal range of the distribution function, where linear wave theory would predict strong Landau damping. This sheds light on the spectrum of small amplitude perturbations proving the incompleteness of linear and associated nonlinear wave theories in the kinetic regime and offers a new interpretation of recent synchrotron experiments.

• Received 4 March 1997

DOI:https://doi.org/10.1103/PhysRevLett.79.2811