Over-mode cylindrical waveguide in a high-power microwave, millimeter-wave generation, amplification, transmission has a very wide range of applications. Based on the dispersion equation, we study a cylindrical waveguide with a lossy layer’s propagation and attenuation constant. When the electric conductivity of the waveguide wall is finite and lossy material coated on its surface the characteristic of the propagation of an electromagnetic wave will vary from different mode. An analytical expression of mode propagation constant is very important for studying microwave loss mechanism in the waveguide. Considering the impact of loss of material with thickness, we derive the general dispersion equation in different modes and in the limit of approximation formula. The propagation characteristics are studied and numerically calculated for analyzing and comparing the difference of the calculated results with the general dispersion equation and approximation formula. These equations can be applied to the case which near the cutoff frequency. By using this approximation formula and general dispersion equation we can obtain attenuation and propagation constant versus lossy layer thickness, operating frequency, and lossy layer conductivity. And make a comparison with the simulation software HFSS (High Frequency Structure Simulator). The results show that in the high conductivity and in the vicinity of the cutoff frequency, the propagation constant calculated by general dispersion equation to be in good agreement with HFSS.