Observation of the top quark flavor changing neutral process $t\to c+\gamma$ at the LHC would be the signal of physics beyond the standard model. If anomalous $t\to c\gamma$ coupling exists, it will affect the precisely measured $\mathcal{B}(\overline{B}\to X_s\gamma )$. In this paper, we study the effects of a dimension 5 anomalous $tc\gamma$ operator in $\overline{B}\to X_s\gamma$ decay to derive constraints on its possible strength. It is found that, for real anomalous $t\to c\gamma$ coupling ${\kappa }_{tcR}^{\gamma }$, the constraints correspond to the upper bounds $\mathcal{B}(t\to c+\gamma )<6.54\times {10}^{-5}$ (for ${\kappa }_{tcR}^{\gamma }>0$) and $\mathcal{B}(t\to c+\gamma )<8.52\times {10}^{-5}$ (for ${\kappa }_{tcR}^{\gamma }<0$), respectively, which are about the same order as the $5\sigma$ discovery potential of ATLAS ($9.4\times {10}^{-5}$) and slightly lower than that of CMS ($4.1\times {10}^{-4}$) with $10\mathrm{f}{\mathrm{b}}^{-1}$ integrated luminosity operating at $\sqrt{s}=14\mathrm{TeV}$.

• Received 18 October 2010

DOI:https://doi.org/10.1103/PhysRevD.83.013004