Top Decays with Flavor Changing Neutral Higgs Interactions at the LHC

We investigate the prospects for discovering a top quark decaying into one light Higgs boson along with a charm quark in top quark pair production at the CERN Large Hadron Collider (LHC). A general two Higgs doublet model is adopted to study the signature of flavor changing neutral Higgs decay $t \to c\phi^0$, %or $\bar{t} \to \bar{c}\phi^0$ where $\phi^0$ could be CP-even ($H^0$) or CP-odd ($A^0$). The dominant physics background is evaluated with realistic acceptance cuts as well as tagging and mistagging efficiencies. For a reasonably large top-charm-Higgs coupling ($\lambda_{tc}/\lambda_{t} \agt 0.09$), the abundance of signal events and the %that our acceptance cuts reduction in physics background allow us to establish a $5\sigma$ signal for $M_\phi \sim 125$ GeV at the LHC with a center of mass energy ($\sqrt{s}$) of 8 TeV and an integrated luminosity of 20 fb$^{-1}$. The discovery potential will be greatly enhanced with the full energy of $\sqrt{s} = 14$ TeV.


I. INTRODUCTION
The Standard Model has been very successful in explaining almost all experimental data to date, culminating in the discovery of the top quark [1,2] and the tau neutrino [3], and finally, a scalar particle that appears to be the long awaited Higgs boson has emerged at the Large Hadron Collider [4][5][6][7][8][9]. The most important experimental goals of the CERN Large Hadron Collider (LHC) are the investigation of the mechanism for electroweak symmetry breaking (EWSB) -the discovery of the Higgs bosons or the proof of their non-existence -and the search for new physics beyond the Standard Model (SM). It is important now to check whether the emerging Higgs boson fits the SM prescription.
In the Standard Model there is just one Higgs doublet, which generates masses for vector bosons and fermions, but the differences among Yukawa couplings of fermions with the Higgs boson are not explained. In addition, there are no flavor changing neutral currents (FCNC) mediated by gauge interactions or by Higgs interactions at the tree level. At present, the top quark is the most massive elementary particle ever discovered. It might provide clues to study the mechanisms of EWSB and FCNC. At the one loop level, the branching fraction of t → cH is 4.6 × 10 −14 for M H = 120 GeV [10,11]. If this decay mode is detected in the near future, it would likely indicate a large effective coupling of tree-level origins, or very large enhancement from beyond SM loop effects.
A general two Higgs doublet model usually contains flavor changing neutral Higgs (FCNH) vertices if there is no discrete symmetry to turn off tree-level FCNC [12,13]. In the context of invoking [14] a Fritzsch-like quark mixing Ansatz [15] to evade low energy FCNC constraints induced by FCNH couplings, it was pointed out long ago [16] that top-charm FCNH coupling could be prominent because of the large top mass. With this in focus, a special two Higgs doublet model for the top quark (T2HDM) [17] offers a good explanation of why the top quark is much heavier than other elementary particles. In the T2HDM, the top quark is the only elementary fermion acquiring its mass from a special Higgs doublet (φ 2 ) with a large vacuum expectation value (VEV). Since the up and charm quarks couple to another Higgs doublet (φ 1 ), there are FCNH interactions among the up-type quarks.
In the near future, for an integrated luminosity of L = 10 fb −1 at √ s = 8 TeV, 1 the LHC will produce approximately 2 × 10 6 top quark pairs (tt) [19][20][21] for m t ≃ 173 GeV. For the same integrated luminosity at √ s = 14 TeV, the number of (tt) pairs generated would increase to about 1 × 10 7 . Thus, the LHC is becoming a top quark factory, providing great opportunities to study electroweak symmetry breaking as well as other important properties of the top quark. If the top quark is heavier than a neutral Higgs boson that interacts with a top quark and a charm quark, then a promising FCNH signature will appear in pp collisions as pp → tt → tcφ 0 + X or pp → tt → cφ 0t + X at the LHC [22]. With the emerging scalar object at 125 GeV, the pursuit of this signature has become mandatory.
In this letter, we study the discovery potential of the LHC in the search for the rare top decay t → cφ 0 , where φ 0 is a scalar (H 0 ) or a pseudoscalar (A 0 ). The Higgs boson then decays into a pair of bottom quarks (bb). We have evaluated production rates with full tree-level matrix elements including Breit-Wigner resonances for both the signal and the physics background. In addition, we optimize the acceptance cuts to effectively reduce the background with realistic b-tagging and mistagging efficiencies. Promising results are presented for the LHC with √ s = 8 TeV as well as √ s = 14 TeV. Section II shows the production cross sections for the Higgs signal and the dominant background, as well as our strategy to determine the reconstructed masses for the top quark and the Higgs boson. Realistic acceptance cuts are discussed in Section III. Section IV presents the discovery potential at the LHC for √ s = 8 TeV and √ s = 14 TeV. Our conclusions, which are quite optimistic, are drawn in Section V.

A. The Higgs Signal
We adopt a general two Higgs doublet model to study flavor changing neutral Higgs interactions with the following effective Lagrangian where H 0 is a CP-even scalar, A 0 is a CP-odd pseudoscalar and v = 2M W /g W ≃ 246 GeV. For M φ < m t , the t → cφ 0 decay width [16] is and + or -corresponds to φ 0 being a scalar or a pseudoscalar. We assume that the total decay with of the top quark is Then the branching fraction of t → cφ 0 becomes As a case study, we take the FCNH Yukawa couplings to be the geometric mean of the Yukawa couplings of the quarks 2 [14,15] with m t = 173.3 GeV and m c = 1.4 GeV. Then the branching fraction of t → cφ 0 becomes GeV. For illustration we use this ansatz in all our figures except for Fig. 5. Later, in the section on LHC discovery potential, we will consider λ tc as a free parameter. We assume that the width and branching fraction of the Higgs scalar decay to bb are similar to those of the standard Higgs boson, while the Higgs pseudoscalar decays to the W + W − or the ZZ pairs are negligible. We do not consider CP violation in this study.
We employ the programs MadGraph [23,24] and HELAS [25] to evaluate the exact matrix element for the FCNH signal in top decays from gluon fusion and quark-antiquark annihilation, gg → tt → tcφ 0 → bℓ + νcbb and qq → tt → tcφ 0 → bℓ + νcbb as well as tt → cφ 0t → cbbbℓ −ν , where ℓ = e or µ. The signal cross section at the LHC for pp → tt → tcφ 0 → bℓνcbb + X is evaluated with the parton distribution functions of CTEQ6L1 [26]. In addition, we have checked the signal cross section by narrow width approximation. That is, the cross section σ(pp → tt → tcφ 0 → bℓνcbb + X) is calculated as the product of cross section times branching fractions: The factorization scale and the renormalization scale are chosen to be Q = M tt , the invariant mass of tt. This choice of scale leads to a K factor of 2 for top quark pair production [27,28].
In our analysis, we consider the FCNH signal from both tt → tcφ 0 → bℓ + νcbb and tt → cφ 0t → cbbbℓ −ν , which will be commonly described as tt → cφ 0t → bℓνcbb or tt → cφ 0t → bℓνbbj. In every event, we require there should be 3 b jets and one light jet (j = u, d, s, c, or g). In addition, there is an isolated lepton (ℓ = e or µ), and the neutrino will lead to missing transverse energy (E / T ). Unless explicitly specified, q generally denotes a quark (q) or an anti-quark (q) and ℓ will represent a lepton (ℓ − ) or anti-lepton (ℓ + ). That means our FCNH signal leads to the final state of bℓνbbj or bbbjℓ + E / T .

B. The Physics Background
The dominant physics background to the final state of bbbjℓν comes from top quark pair production followed by top and W decays: where a c-jet is mis-identified as a b-jet. We have also considered backgrounds from tt → bℓνbud, as well as backgrounds from the production of bbbbℓν and bbccℓν. According to the ATLAS and CMS Technical Design Reports [29,30], the b tagging efficiency is 50% − 60%, the probability that a c-jet is mistagged as a b-jet (ǫ c ) is approximately 10%, while the probability that any other jet is mistagged as a b-jet (ǫ j ) is 1%. If ǫ c is chosen to be the same as that of a light jet (ǫ j = 0.01), the background from tt → bℓνbcs will be underestimated by a factor of 10 such as in the analysis in Ref. [22]. This point would be elaborated in our study.

C. Mass Reconstruction
In this subsection, we discuss our strategy to determine the reconstructed top mass as the invariant mass of M bbj for the top quark with FCNC (t ort → cφ 0 → cbb), as well as the other reconstructed top mass with leptonic decay (t or t → bW → bℓν): In the process of doing so, we reconstruct the Higgs mass as the invariant mass of a pair of b jets and one W mass as the invariant mass of a charged -lepton/neutrino pair ℓν. We also reconstruct a potential second W mass from a b-jet/light jet pair for vetoing the background: Figure 1 shows invariant mass distributions with basic cuts: In each event, we assume that three b jets and one non-b jet are identified through b-tagging. We then assign the three b jets (b 1 , b 2 , b 3 ) according to the following procedure: Since our FCNC signal comes from tt → cφ 0b ℓν → bbcbℓν → bbbjℓ + E / T , we will choose the pair of b jets that minimize |M bbj − m t | as b 1 b 2 and label the other b jet as b 3 . For a correctly reconstructed event, b 1 and b 2 are the products of a Higgs decay as well, such that their invariant mass has a peak near M φ . For a background event, one b is likely coming from the top decay t → bW → bcj while the other is either a mistagged c or a light quark jet coming from W decay, or a real b quark coming from the decay of t. Let us identify b 2 as the member of this pair that minimizes M bj − m W . In a good reconstruction, the remaining b quark, b 3 should reproduce the top quark mass with the charged lepton and neutrino momenta. In this figure, we present the reconstructed masses for signal and background with M φ = 125 GeV: We have used the Higgs scalar case here and in Fig. 2 as an example; the shapes are virtually the same for a pseudoscalar.
In our analysis, we assume that the FCNH signal comes from top quark pair production with one top quark decaying via FCNC while the other decays leptonically (t → bW → bℓν). For a real W decay (W → ℓν), the momentum of the neutrino (k) and that of the lepton (p) have the following relation If the missing transverse energy comes solely from the neutrino in W decay, we can estimate the longitudinal momentum of the neutrino (k z ) with measured lepton energy and momentum (E ℓ , p), transverse missing momentum (k x = E / x and k y = E / y ) and E ν = | k|.
Assuming an on-shell W , we can evaluate k z of the neutrino with with There are two possible values for k z if ∆ 2 > 0. We select whichever leads to a better reconstruction of the top-quark mass, and define this reconstructed top mass as M R t 2 = M b 3 ℓν . In case ∆ 2 < 0, we set ∆ = 0 to evaluate k z , corresponding to a virtual W . We require that the reconstructed W mass (M ℓν ) should be close to the on-shell mass m W .

III. REALISTIC ACCEPTANCE CUTS
To study the discovery potential of this FCNH signal at the LHC, we have applied realistic cuts and tagging efficiencies for three combinations of the CM energy ( TeV) with high luminosity L = 300 fb −1 [29,30]. For (a) the early LHC and (b) full CM energy with low luminosity, we require that in every event there should be • exactly 4 jets that have p T > 15 GeV and |η| < 2.5, and three of them must be tagged as b-jets; • exactly one isolated lepton that has p T > 20 GeV and |η| < 2.5; • the missing transverse energy (E / T ) must be greater than 20 GeV; • at least one pair of b-jets such that the invariant mass of b 1 b 2 j should be near m t : • the pair of b-jets, b 1 b 2 , that reconstructs the top quark with FCNH decay should also • a third b jet such that the invariant mass of b 3 ℓν should be near m t : • the reconstructed leptonic W must satisfy |M ℓν − m W | ≤ 15 GeV.
Additionally, to effectively reduce backgrounds from W → jj, we require |M b 2 j − m W | > 15 GeV. We also require ∆R = √ ∆φ 2 + ∆η 2 > 0.4 between every pair of jets and between each jet and the charged lepton, to limit QCD production of multi-jets and ensure good reconstruction of isolated jets and the charged lepton.
In the early stage of LHC with √ s = 8 TeV, the b-tagging efficiency (ǫ b ) is taken to be 50%, the probability that a c-jet is mistagged as a b-jet (ǫ c ) is 10% and the probability that any other jet is mistagged as a b-jet (ǫ j ) is taken to be 1%. At the full CM energy ( √ s = 14 TeV) with an integrated luminosity (L) of 30 fb −1 , we follow the tagging and mistagging efficiencies in the ATLAS Technical Design Report [29]: ǫ b = 60%, ǫ c = 14% and ǫ j = 1%. For the full CM energy ( √ s = 14 TeV) with a high integrated luminosity of 300 fb −1 , we require p T (b, j) > 30 GeV, p T (ℓ) > 20 GeV, |η(b, j, ℓ)| < 2.5, and E / T > 40 GeV. The tagging and mistagging efficiencies are taken to be ǫ b = 50%, ǫ c = 14% and ǫ j = 1%. Furthermore, a powerful acceptance cut on the charm-jet energy was proposed in a study to search for FCNH top decays at linear colliders [31]. In the rest frame of the top quark, the energy of the charm jet from t → cφ 0 is For M φ = 125 GeV, E R c ≃ 42 GeV, while for M φ = 150 GeV, E R c ≃ 22 GeV. In the background, the non-b jet, which is most likely not a charm quark and which arises from W decay, has a distribution of energy which is more spread out [31].
In Fig. 2, we present the distribution with respect to the charm energy in the top rest frame (E R c ) for pp → tt → bℓνcbb + X with (a) M H = 125 GeV and (b) M H = 150 GeV. Also shown is the same distribution for the physics background from pp → tt → bℓνbcs + X where the charm quark is mistagged as a b and the strange quark fakes a charm jet. In our complete analysis, we choose 32 GeV < E R c < 52 GeV for M φ = 125 GeV, and 12 GeV < E R c < 32 GeV for M φ = 150 GeV. Measurement uncertainties in jet and lepton momenta as well as missing transverse momentum give rise to a spread in the reconstructed masses about the true values of m t and M φ . Based on the ATLAS [29] and the CMS [30] specifications we model these effects by Gaussian smearing of momenta: FIG. 3: The cross section of pp → tt → tcφ 0 → bℓνcbb + X at the LHC with √ s = 8 TeV, as a function of M φ . We present results for Higgs scalar H 0 (dash-dot blue) and Higgs pseudoscalar A 0 (solid cyan). Also shown are the background cross section (σ B ) (dash-dot-dot red) and the thresholds for 5σ discovery with 20 fb −1 (dash green) of integrated luminosity, or 3σ evidence with either 20 fb −1 (dot magenta) or 5 fb −1 (dot black). We take the ansatz of Eq. (5) to set the FCNH coupling strength and have applied K factors, acceptance cuts, and efficiencies of b-tagging and mistagging.
for jets and for charged leptons with individual terms added in quadrature.

IV. DISCOVERY POTENTIAL AT THE LHC
Our results for the signal and background at the LHC with √ s = 8 TeV and √ s = 14 TeV are presented in Figs. 3 and 4 respectively. To estimate the discovery potential at the LHC we include curves that correspond to the minimal cross section of signal (σ S ) required by our discovery criterion described in the following. We define the signal to be observable if the lower limit on the signal plus background is larger than the corresponding upper limit on the background [32] with statistical fluctuations or equivalently, Here L is the integrated luminosity, σ S is the cross section of the FCNH signal, and σ B is the background cross section. The parameter N specifies the level or probability of discovery. We take N = 2.5, which corresponds to a 5σ signal. For Lσ B ≫ 1, this requirement becomes similar to where N S is the signal number of events, N B is the background number of events, and N SS is the statistical significance, which is commonly used in the literature. If the background has fewer than 25 events for a given luminosity, we employ the Poisson distribution and require that the Poisson probability for the SM background to fluctuate to this level is less than 2.85 × 10 −7 , i.e. an equivalent probability to a 5-sigma fluctuation with Gaussian statistics. Figure 3 shows the signal and background cross sections for the CERN Large Hadron Collider with √ s = 8 TeV. All tagging efficiencies and K factors discussed above are included.
It is expected that the 2012 run of the LHC will continue at √ s = 8 TeV and accumulate an integrated luminosity of L ∼ 20 fb −1 for each detector at the end of the year [18]. As can be seen from the figure, with 20 fb −1 of data at 8 TeV running we can potentially discover this FCNC decay mode for a Higgs with mass less than ∼ 123 GeV or we will be able to establish a 3σ discrepancy from the Standard Model for masses below ∼ 130 GeV. At high masses the pseudoscalar signal is somewhat larger than the scalar case. This is due to the absence of W + W − and ZZ branching modes in the pseudoscalar decay. GeV for a scalar coupling or M A ∼ 138 GeV for a pseudoscalar. With a high luminosity L = 300 fb −1 , these ranges can be slightly improved to M H ∼ 138 GeV and M A ∼ 142 GeV respectively. High luminosity does not extend our range much, owing to (a) the kinematic limitations as the Higgs mass is increased towards the top mass, and (b) the higher p T cuts and lower tagging efficiency we assume.

V. CONCLUSIONS
It is a generic possibility of particle theories beyond the standard model to have contributions to tree-level FCNHs, especially for the third generation quarks. These contributions arise naturally in models with additional Higgs doublets, such as the T2HDM, wherein the top quark uniquely couples to one doublet, or a general 2HDM that has suppressed FCNH for lower generations. For a Higgs boson with a mass below the top mass -a case that seems realized -this could engender the rare decay t → cφ 0 .
We investigated the prospects for discovering such a decay at the LHC, focusing on the channel where tt are pair produced and subsequently decay, one leptonically and one through the FCNH mode. The primary background for this signal is a tt pair with one standard hadronic decay and the other leptonic. This background involves one jet mis-tagged as a b jet, which is much more likely to occur for a c quark than for lighter jets. Nonetheless, by taking advantage of the available kinematic information, one can reconstruct the resonances of the signal and reject much of the background.
Based on a simple geometric ansatz for the size of the FCNC coupling, namely λ tc /λ t ∼ 0.09, we find that such a decay mode may be discovered at the LHC for Higgs masses up to 123 GeV with the current running energy, and up to almost 140 GeV with the design energy of 14 TeV. With the emerging scalar object at 125 GeV, this looks rather promising. We therefore also presented results where the Higgs branching fraction and FCNC coupling are allowed to vary for fixed masses. With a new scalar particle very similar to the standard Higgs boson discovered [4][5][6][7] recently at the LHC by the ATLAS and CMS experiments, we look forward to being guided by more new experimental results as we explore interesting physics of EWSB and FCNH. While the properties (for example, scalar vs pseudoscalar) of the Higgs-like object goes under further scrutiny as data accumulates, perhaps a dedicated FCNH t → cφ 0 search should be undertaken.