We study the topological structure of basin boundaries of open chaotic Hamiltonian systems in general. We show that basin boundaries can be classified as either type I or type II, according to their topology. Let B be the intersection of the boundary with a one-dimensional curve. In type I boundaries, B is a Cantor set, whereas in type II boundaries B is a Cantor set plus a countably infinite set of isolated points. We show that the occurrence of one or the other type of boundary is determined by the topology of the accessible configuration space, and also by the chosen definition of escapes. We show that the basin boundary may undergo a transition from type I to type II, as the system’s energy crosses a critical value. We illustrate our results with a two-dimensional scattering system.

• Received 5 February 2002

DOI:https://doi.org/10.1103/PhysRevE.66.046214