Rare Higgs three body decay induced by top-Higgs FCNC coupling in the littlest Higgs model with T-parity

Motivated by the search for flavor-changing neutral current (FCNC) top quark decays at the LHC, we calculate the rare Higgs three body decay H → Wbc induced by top-Higgs FCNC coupling in the littlest Higgs model with T-parity (LHT). We find that the branching ratio of H → Wbc in the LHT model can reach O(10−7) in the allowed parameter space.


Introduction
The discovery of a Higgs-like resonance near 125 GeV [1] at the LHC is a great triumph for theoretical and experimental particle physics. So far, most measurements of this new particle are consistent with the Standard Model (SM) prediction, but the experimental investigation of this new particle has only just begun. It is not impossible that more in-depth studies will reveal non-SM properties.
Compared with normal decay modes, flavourchanging neutral current (FCNC) decays are highly suppressed in the SM due to the Glashow-Iliopoulos-Maiani (GIM) mechanism [2]. So, any large enhancements in these branching ratios will be smoking-gun signals for physics beyond the SM.
As the heaviest known elementary particle, the top quark is widely speculated to be sensitive to the electroweak symmetry breaking (EWSB) mechanism and new physics at TeV-scale. An interesting possibility is the presence of FCNC interactions between the Higgs boson and the top quark. This interaction not only participates in the top quark FCNC decays [3], but also participates in the Higgs FCNC decays [4].
Except for the dominant decay mode H → bb, the so called below-threshold decay modes induced by the HVV(V = W; Z) couplings are also very important, with the decay H → VV having one (or two) V's off-shell and decaying to fermions. In some new physics, the decay mode of Higgs bosons is much richer and 3-body decays may be even more important. Now, almost all Higgs boson decay modes have been measured at the LHC, but they are plagued by large SM backgrounds. So, the rare Higgs 3-body decays may bring us more surprises. In some new physics models, the GIM suppression can be relaxed and/or new particles can contribute to the loops, so that the top-Higgs FCNC couplings tqH, especially the tcH coupling, can be enhanced by orders of magnitude larger than those of the SM [5].
In this paper, we study the rare Higgs 3-body decay H → Wbc induced by the top-Higgs FCNC coupling in the littlest Higgs Model with T-parity (LHT). This decay includes the FCNC vertex tcH, which receives the contribution from the new T-odd gauge bosons and Todd fermions. The results of this process will help to test the SM and probe the LHT model.
The paper is organized as follows. In Section 2 we give a brief review of the LHT model related to our work. In Section 3 we calculate the rare Higgs 3-body decay H → Wbc induced by the top-Higgs FCNC coupling in the unitary gauge under current constraints. Finally, we draw our conclusions in Section 4.

A brief review of the LHT model
The LHT model is based on an SU (5)/SO (5) nonlinear σ model [6]. At the scale f ∼ O (TeV), the SU (5) global symmetry is broken down to SO(5) by the vacuum expectation value (VEV) of the σ field, Σ 0 , given by After the global symmetry is broken, there arise 14 Goldstone bosons (GB) which are described by the "pion" matrix Π. Then the kinetic term for the GB matrix can be expressed in the standard non-linear sigma model formalism as The σ field kinetic Lagrangian is given by with the [SU (2) ⊗ U (1)] 2 covariant derivative defined by where W µ j = 3 a=1 W µ a j Q a j and B µ j = B µ j Y j are the heavy SU (2) and U (1) gauge bosons, with Q a j and Y j being the gauge generators, and g j and g j the respective gauge couplings.
The VEV Σ 0 also breaks the gauged subgroup (3), the masses of the T-parity partners of the W boson (W ± H ), Z boson (Z H ) and photon (A H ) after EWSB are given by where g and g denote the SM SU (2) and U (1) gauge couplings, respectively. v represents the VEV of the Higgs doublet, which is related to the SM Higgs VEV v SM = 246 GeV through the following formula: In the quark sector, the T-odd mirror partners for each SM quark are added to preserve T-parity. The up and down-type mirror quarks can be denoted by u i H and d i H , where i(= 1, 2, 3) is the generation index. One can write down a Yukawa interaction to give masses to the mirror quarks After the EWSB, their masses up to O(v 2 /f 2 ) are given by where κ i are the eigenvalues of the mass matrix κ.
Under T-parity, in order to cancel the large radiative correction to the Higgs mass parameter induced by the top quark, an additional T-even heavy quark T + and its T-odd mirror partner T − are introduced. Their masses are given by where x L is the mixing parameter between the top-quark and heavy quark T + . This mixing parameter can also be expressed by a ratio R = λ 1 /λ 2 with where λ 1 and λ 2 are two dimensionless top quark Yukawa couplings. When the mass matrix √ 2κ ij f is diagonalized by two U (3) matrices, a new flavor structure can come from the mirror fermions. In the mirror quark sector, the existence of two CKM-like unitary mixing matrices V Hu and V Hd is one of the important ingredients. Note that V Hu and V Hd are related through the SM CKM matrix: 3 Branching ratio for H → Wbc in the LHT model The Feynman diagrams of the tree level H → W + bc and the rare decay H → W + bc are shown respectively in Fig. 1 and Fig. 2, which includes the W + and W − modes. The rare Higgs decay H → Wbc is mediated by the same Yukawa coupling that leads to the t → cH decay [9], so we show the Feynman diagrams of the LHT one-loop correction to vertex V tcH in unitary gauge in Fig. 3, where the Goldstone bosons do not appear. We can see that the flavor changing interactions between SM quarks and mirror quarks are mediated by the heavy gauge bosons W ± H , Z H , and A H . We find that dominant contribution to the branching ratio of the decay H → Wbc is from the interference between Fig. 1 and Fig. 2. Each loop diagram is composed of some scalar loop functions [10], which are calculated by using LOOPTOOLS [11].
In our numerical calculations, we take the SM parameters as follows [12] G F = 1.16637 × 10 −5 GeV  The LHT parameters related to our calculations are the scale f , the mixing parameter x L , the Yukawa couplings κ i of the mirror quarks and the parameters in the matrices V Hu , V Hd . Due to the weak influence of the mixing parameter x L , we take x L = 0.1 as an example in our calculations. For the mirror quark masses, we get For the Yukawa couplings, the search for mono-jet events at the LHC Run-1 [14] gives the constraint κ i 0.6. Considering the constraints in Ref. [13], we scan over the free parameters f , κ 12 and κ 3 within the following region 500 GeV f 2000 GeV, 0.6 κ 12 3, 0.6 κ 3 3.
For the parameters in the matrices V Hu , V Hd , we follow Ref. [15] to consider two scenarios as follows: 1) Scenario I: V Hd = I, V Hu = V † CKM ; 2) Scenario II:

043103-3
Furthermore, we will consider the constraint from the global fit of the current Higgs data and the electroweak precision observables (EWPOs) [16]. In Fig. 4, we present the excluded regions by the global fit of the Higgs data, EWPOs and R b in the κ ∼ f plane of the LHT model for case A and case B, where the parameter R is marginalized over. In this global fit, the three generation Yukawa couplings κ i are considered to be degenerate, which will give a stronger constraint than the nondegenerate case here.
In Fig. 5, we show the branching ratios of H → Wbc in the κ 3 ∼ f plane for two scenarios with excluded regions of case A and case B, where the W + and W − modes have been summed. From the left-hand panel of Fig. 4, the branching ratio of H → Wbc in scenario I can reach 1×10 −7 at 2σ level for case A. This branching ratio will become larger under the constraint of case B. From the right-hand panel of Fig. 5, we can see that the branching ratio of H → Wbc in scenario II can reach 4 × 10 −7 at 2σ level, which is three or even four times larger than that in scenario I. Comparing the two scenarios, we find that the enhanced effects come from the large departures from the SM caused by the mixing matrices in scenario II. From the two panels of Fig. 5, the large branching ratios mainly lie in the upper-left and lower-left corners of the contour figures, where the scale f is small and the Yukawa coupling κ 3 is either too small or too large.
According to Ref. [15], the branching ratio of t → cH is enhanced by the mass splitting between the three generation mirror quarks. The same thing will happen to the branching ratios of H → Wbc. In order to see this dependence, we show the branching ratios of H → Wbc in the | M 3 −M 12 |∼ f plane for the two scenarios in Fig. 6. We can see that the small branching ratios correspond to the region that has small mass splitting | M 3 − M 12 | values.
The largest branching ratios lie in the upper-left corners of the contour figure with small f and | M 3 − M 12 | of 1 ∼ 2 TeV, rather than the regions that have the largest | M 3 −M 12 |, because the branching ratios are suppressed by the high scale f .
For observability, the SM decay H → WW * → Wbc is an important irreducible background that will generate the same final state. Due to the off-shell top in the signal decay H → t * c → Wbc, we can use the invariant mass cut |M Wb −m t | > 20 GeV to isolate the signal. Besides, the c-jet in our signal comes from the Higgs decay, which is usually harder than that in the SM background H → WW * → Wbc. Thus, we can use the high transverse momentum p c T cut to suppress the background.   Due to the same Yukawa couplings that lead to the t → cH decays, the decays H → t * c → Wbc can be indirectly constrained by ATLAS and CMS searches [17]: Br(H → t * c → Wbc) 5.73 × 10 −4 , where the W + and W − modes have been summed over. At the LHC, the tt(→ WbWb) background is undoubtedly a challenge, which will complicate the analysis for detecting the decay H → t * c → Wbc. Given this, a linear collider with clean background may be an ideal place for investigating this process. For example, a future muon collider could test the FCNC decay t → cH via Higgs decay H → t * c → Wbc down to values of Br(t → cH) ∼ 5 × 10 −3 [18].

Conclusions
In this paper, we have calculated the rare Higgs three body decay H → Wbc induced by top-Higgs FCNC coupling in the LHT model. According to the parameters in the mixing matrices, we considered two scenarios and found that the branching ratio for H → Wbc can reach O(10 −7 ) in the allowed parameter space.