We study quark mass matrices in the Randall-Sundrum (RS) model with bulk symmetry $SU(2{)}_L\times SU(2{)}_R\times U(1{)}_{B-L}$. The Yukawa couplings are assumed to be within an order of magnitude of each other, and perturbative. We find that quark mass matrices of the symmetrical form proposed by Koide et al. [Y. Koide, H. Nishiura, K. Matsuda, T. Kikuchi, and T. Fukuyama, Phys. Rev. D 66, 093006 (2002)] can be accommodated in the RS framework with the assumption of hierarchyless Yukawa couplings, but not the Hermitian Fritzsch-type mass matrices. General asymmetrical mass matrices are also found which fit well simultaneously with the quark masses and the Cabibbo-Kobayashi-Maskawa matrix. Both left-handed (LH) and right-handed (RH) quark rotation matrices are obtained that allow analysis of flavor changing decay of both LH and RH top quarks. At a warped down scale of 1.65 TeV, the total branching ratio of $t\to Z+\mathrm{\text{jets}}$ can be as high as $\sim 5\times {10}^{-6}$ for symmetrical mass matrices and $\sim 2\times {10}^{-5}$ for asymmetrical ones. This level of signal is within reach of the LHC.

• Received 12 June 2008

DOI:https://doi.org/10.1103/PhysRevD.78.096003