There are many other modes besides TM01 mode in Ku-band relativistic backward wave oscillator working at low magnetic field due to inhomogeneous emission of explosive cathode. The circular couplers currently used are for pure TM01 mode only. The measurement will be disturbed by other modes and the waveform obtained will be abnormal, which will make measurement difficult and inaccurate. A mode suppressor consisting of five cavities is designed in this paper. The attenuation coefficient of the mode suppressor is lower than 0.1 dB for TM01 mode between 14.8 GHz and 15 GHz according to the simulation and measurement results. Other modes are total reflective for comparison. The mode suppressor is placed between Relativistic Backward Wave Oscillator(RBWO) and coupler in the experiment. The waveforms obtained from coupler as well as the output power are consistent well with that from radiation field, indicating beneficial effect of the proposed mode suppressor.