Abstract
An investigation is made of the stability of steady-state reflection from a boundary of a nonlinear medium with respect to the growth of small three-dimensional perturbations. It is shown that in the range of parameters of interest, where the plane monochromatic wave reflection theory predicts hysteretic effects and bistable states, the nonlinear refraction regime is instable. The dependence of the growth increment on the perturbation wavelength is determined for an arbitrary angle of incidence and an arbitrary incident wave intensity. The nonlinear stage of the development of an instability is investigated numerically in a one-dimensional approximation. In the bistable region a transition is discovered to a self-oscillatory regime in which periodic switching takes place between total internal reflection and transmission.