Collider signals of a composite Higgs in the Standard Model with four generations

Recent fits of electroweak precision data to the Standard Model (SM) with a 4th sequential family (SM4) point to a possible"three-prong composite solution": (1) the Higgs mass is at the TeV-scale, (2) the masses of the 4th family quarks t',b' are of O(500) GeV and (3) the mixing angle between the 4th and 3rd generation quarks is of the order of the Cabibbo angle, \theta_{34} ~ O(0.1). Such a manifestation of the SM4 is of particular interest as it may suggest that the Higgs is a composite state, predominantly of the 4th generation heavy quarks. Motivated by the above, we show that the three-prong composite solution to the SM4 can have interesting new implications for Higgs phenomenology. For example, the Higgs can decay to a single heavy 4th generation quark via the 3-body decays (through an off-shell t' or b') H ->t'(bar) t'* ->t'(bar) b W+ and H ->b'(bar) b'* ->b'(bar) t W-. These flavor diagonal decays can be dramatically enhanced at the LHC (by several orders of magnitudes) due to the large width effects of the resonating heavy Higgs in the processes gg ->H ->t'(bar) t'* ->t'(bar) b W+ and gg ->H ->b'(bar) b'* ->b'(bar) t W-, thus yielding a viable signal above the corresponding continuum QCD production rates. In addition, the Higgs can decay to a single t' and b' in the loop-generated flavor changing (FC) channels H ->b' b(bar), t' t(bar). These FC decays are essentially"GIM-free"and can, therefore, have branching ratios as large as 10^{-4} - 10^{-3}.


Introduction
The SM4 is one of the simplest new physics scenarios, in which the SM is enlarged by a complete sequential 4th family of chiral matter: a new (t ′ , b ′ ) and (ν ′ , ℓ ′ ) heavy doublets in the quark and lepton sectors, respectively (for reviews see [1,2,3]). For many years, since the LEPI measurement of the Z-width and up to the early 2000's, the SM4 was thought to be ruled out by experimental data [4]; however, this interpretation no longer appears to be correct. Indeed recent fits of the SM4 to precision EW observables [5,6,7] and to flavor data [8,9,10,11] have clarified that the SM4 is not in disagreement with the present data. According to these recent studies, the masses of the 4th generation quark doublet may lie in the range 300 − 700 GeV and should be slightly spilt, i.e., m t ′ − m b ′ ∼ 45 − 75 GeV (consistent also with the current CDF limits on the 4th generation quark masses: m t ′ > ∼ 310 GeV [12] and m b ′ > ∼ 340 GeV [13]). In turn, in order to be consistent with electroweak (EW) precision data, this favors a heavier Higgs, thereby, completely removing the tension between the LEPII bound m H > ∼ 115 GeV and the central (best fitted) Higgs mass value m H ∼ 85 GeV obtained from a fit of the SM to the data. This effect is mainly due to an interesting interplay between the contributions of the 4th family fermions to the oblique parameters S and T, as was noted in [5,7].
Indeed, an extremely interesting outcome of these new studies is that the best fitted Higgs mass in the SM4 can be driven up to its unitarity bound, m H < ∼ 1 TeV, if the mixing angle between the 4th and 3rd generation quarks, θ 34 , is of the size of the Cabibbo angle (i.e., θ 34 ∼ O(0.1)) and if m t ′ , m b ′ ∼ O(500) GeV [6]. The combination of a heavy O(500) GeV 4th generation doublet with a TeV-scale Higgs might be a hint that the Higgs is not a fundamental particle but is, rather, a composite state, primarily, of the heavy 4th generation quarks [1,14,15,16,17]. In particular, within the SM4, such a heavy Higgs and heavy 4th generation doublet can drive the Landau pole down to the TeV-scale, in which case, the simplest compositeness condition (used in top-condensate type models) which relates the masses of the heavy fermions to that of the Higgs, suggests that m H ∼ √ 2m Q , where Q is the heavy fermion that forms the composite state [1,15].
Another interesting possibility which was recently raised in [16] is that if the masses of t ′ , b ′ are sufficiently heavy, then the Higgs quartic and 4th generation Yukawa couplings can have a fixed point at a scale of several TeV (rather than a Landau pole), around which the 4th generation heavy quarks can form the composite state. These recent theoretical developments and experimental indications in favor of the SM4 with a heavy Higgs and a heavy 4th generation doublet, as well as the possibility of an alternative non-SUSY solution to the hierarchy problem (see [16]), lead us to study what we henceforward name the "three-prong composite solution" of the SM4: which addresses the composite Higgs scenario. We will show that this three-prong composite solution of the SM4 can have interesting new implications on Higgs phenomenology at the LHC. In particular, we will focus on the case of 1 This range of Higgs and 4th generation quark masses opens up the interesting possibilities of a heavy (composite) Higgs decaying to a single 4th generation heavy quark via an off shell t ′ or b ′ in the flavor diagonal channels: or via the 1-loop FC channels: As we will show below, the rates for the 3-body decays in (1), which, within the three-prong composite solution to the SM4, are the only probe of the Higgs Yukawa couplings to the 4th family quarks, can be dramatically enhanced due to the large width effects in the heavy Higgs resonance production processes and can reach a detectable level above the QCD production rate.
In addition, the FC decays in (2) can have branching ratios (BR) as large as 10 −4 − 10 −3 , owing to the rather large θ 34 ∼ O(0.1) and to the large t ′ and b ′ masses which render these decays essentially GIM-free.
Throughout our analysis to follow we set the masses of the 4th generation leptons (which enter in the calculation of the total Higgs width) to m ℓ ′ = 250 GeV and m ν ′ = 200 GeV. We furthermore assume that V tb = V t ′ b ′ = 1 and that the mixing of the 4th generation quarks with the first two generations ones is much smaller than θ 34 , in particular, that 2 Heavy Higgs production at the LHC The Higgs production rate at the LHC is known to increase by almost an order of magnitude in the SM4, depending on the Higgs and 4th family quark masses [5,18]. This is manifest in the gluongluon fusion hard process gg → H and is caused by the enhancement of the ggH 1-loop coupling due to the extra heavy 4th family quarks that run in the loop. This enhancement comes in handy in particular for the production of a TeV-scale Higgs, where the SM rate might be marginal. In Fig. 1 we plot the gluon-gluon fusion cross-section convoluted with the PDF for pp → H + X in 1 Our study complements the case m H < ∼ 500 GeV with m t ′ , m b ′ < ∼ 400 GeV which was discussed in Ref. [5].
the SM and in the SM4 for three values of the t ′ mass, m t ′ = 400, 500 and 600 GeV. Here and henceforward we always set m b ′ = m t ′ − 70 GeV, which is the value of m b ′ appropriate for Higgs masses m H > ∼ 600 GeV. 2 That is, in order for the SM4 to be consistent with EW precision data the mass splitting in the 4th family quarks is constrained by [5,19]: As can be seen in Fig. 1, for m H ∼ 800 GeV, the gluon-fusion cross-section increases from O(100) fb in the SM to O(1000) fb in the SM4, thus expecting the LHC to produce about 10 5 heavy Higgs with a luminosity of O(100) inverse fb.
Note that, when m H < 2m t ′ , 2m b ′ , the decay pattern of the SM4 heavy Higgs is similar to that of the SM Higgs, up to its possible decays to the 4th family leptons. In particular, the leading decay channels of the heavy SM4 Higgs remain H → ZZ, W W, tt. Thus, as was recently noted in [5], the expected enhancement in the gluon-fusion production channel implies that the "golden mode" H → ZZ → 4µ holds up as a useful Higgs discovery channel throughout the Higgs mass range.  TeV, in the SM (solid curve) and in the SM4 for several values of (m t ′ , m b ′ ), as a function of the Higgs mass. We take K = 1.5 for the K-factor to account for the QCD corrections to the gluon-gluon fusion cross-section [20].
GeV, which is the range of Higgs masses of interest in this work.

Flavor conserving heavy Higgs decays to a single 4th generation quark
Within the three-prong composite solution of the SM4, where m t ′ < m H < 2m t ′ and m b ′ < m H < 2m b ′ , the 2-body Higgs decays H → t ′t′ , b ′b′ are kinematically forbidden. In this case, the only probe of the Higgs Yukawa couplings to the 4th family quarks are the Higgs 3-body decays in (1) which proceed through an off-shell heavy 4th family quark.
Here we are interested in the case of vanishing mixings between the 4th generation quarks with the 1st two generation ones, in which case , so that the flavor diagonal 3-body Higgs decays in (1) will lead to: and the charged conjugate channels.
Note in particular the signature of the b ′ Yukawa coupling in (5): ttW − W + → bbW + W + W − W − , which, with the appropriate kinematical cuts, is expected to be rather distinct, e.g., leading to a signal of same-sign leptons + missing energy accompanied by 2 b-jets and 4 light-jets: H → b ′b′⋆ → ℓ + ℓ + bbjjjj + E T (and the charged conjugate signal).
Naively one may expect these 3-body decays to have very small rates, with BR's at the level of 10 −6 − 10 −5 when m H < 2m t ′ , 2m b ′ . However, as we will show below, these decays can be dramatically enhanced due to finite width effects of the heavy Higgs. Indeed, the non-zero width of heavy particles is known to cause substantial enhancement when these particles appear in the final state of a decay which occurs just around its kinematical threshold [21]. Alternatively, when the decaying heavy particle emerges as an intermediate state the effects of its width close to threshold is usually handled with the Breit-Wigner prescription. In this case, special attention is required for the computation of the overall resonant production and subsequent decay of the heavy particle.
Here we consider the leading resonance production of the heavy Higgs via the gluon-fusion process (which is further enhanced in the SM4, see previous section), followed by its decays H → F as in (5). Using the relativistic Breit-Wigner resonance formula, we estimate the corresponding hard cross-sections by:σ withσ andBW is the normalized Breit-Wigner function: where, in order to incorporate the appropriate kinematic dependencies away from the resonance, we replace m H Γ H → √ŝ Γ H (ŝ) and set Γ H (ŝ) = √ŝ Γ H /m H (Γ H being the total Higgs width) [4]. We thus get (for the hard process gg → H → F ): which, after being convoluted with the gluons distribution functions (PDF's): gives: In the Narrow Width Approximation (NWA), whereBW (ŝ) ≈ δ(ŝ − m 2 H ), we obtain: where τ H = m 2 H /s and all the other terms in (12) are evaluated atŝ = m 2 H . Using the Breit-Wigner formula in (9), we have estimated the width effects of the heavy Higgs on the cross-sections for producing a single 4th family quark through the 3-body Higgs decays: by integrating the corresponding hard cross-sections over some finite range around the resonance, depending on the Higgs mass and width. In Fig. 2 we give a sample of our results which show the dramatic enhancement near the threshold for the 3-body Higgs decays H →t ′ bW + ,b ′ tW − , due to the large width of the decaying Higgs when it is produced at resonance at the LHC. Notice that, due to the large Higgs width, the NWA is not adequate for estimating these cross-sections (as expected). This can be seen in the figure for m H masses sufficiently away from the threshold of the decay to on-shell pair of heavy 4th generation quarks H → t ′t′ , b ′b′ . For example, for m H = 1000 TeV and m t ′ = 400 GeV, is the Breit-Wigner estimate of (9) for the corresponding cross-sections. This difference between the NWA and the BW approaches indicates the extent to which the NWA is valid in this case. We see that around the threshold the enhancement due to the finite width effect of the resonating Higgs can reach several orders of magnitudes, elevating these cross-sections to O(1000) [fb], which should be within the detectable level at the LHC. This enhancement is practically independent of the mixing angle θ 34 , since as long as the mixings of the 4th generation with the 1st and the 2nd generation quarks are much smaller than θ 34 the decays t ′ → bW and b ′ → tW occur with a BR of order one. In Table 1 we evaluate these cross-sections for m H = 800 GeV, m t ′ = 500 GeV and we list the potential (leading) background to these single 4th generation quark signals from the QCD continuum production. 3 We see that with a luminosity of the O(100) inverse fb, one expects 10 4 −10 5 such single t ′ and b ′ events with a (naive) signal to background ratio of S/B ∼ 1/10−1/3 and a corresponding statistical significance of S/ √ B ∼ 50 − 200, depending on the range of integration of the hard cross-sections. Nonetheless, the detection of these Higgs decay signals at the LHC is rather challenging. In particular, after the t ′ and b ′ decay (promptly) via t ′ → bW, b ′ → tW these signals will give two distinct final state topologies: which lead to large multiplicity events with leptons, jets and missing transverse energy (the t ′t′ one resembling the case of tt + jets events in the SM). The reconstruction of such events will, therefore, require a detailed and rather demanding study of both the signal and background samples. For example, it is possible to use subsets of the above signatures which can be distinguishable even without a full reconstruction of the event. In fact, such analysis for b ′ and t ′ -pair production   topologies similar to (14) was already performed by CDF in [12,13], and may be used as a starting point for the search of our heavy Higgs resonance signals H →t ′ t ′⋆ ,b ′ b ′⋆ at the LHC. In particular, for the detection of b ′ via b ′b′ → ttW + W − → bbW + W + W − W − , [13] required the signature ℓ ± ℓ ± bj E T : two same-charge reconstructed leptons, at least two jets one of them b-tagged and missing transverse energy (with cuts on the transverse energy and rapidity of the jets/leptons, see [13]). This same-sign leptons signature was found to be particularly sensitive to the new physics signal b ′b′ → ttW + W − → bbW + W + W − W − (the main background coming from tt and W+jets production) and was used to obtain a bound on the corresponding b ′ mass m b ′ > ∼ 340 GeV [13]. Such an analysis can be similarly applied to our signal (at the LHC In the t ′t′ → bbW + W − case CDF required a signature of one lepton, four or more jets and missing transverse energy ℓ + 4j + E T [12], which allowed a useful discrimination of the t ′t′ signal from the background (here also the SM background mostly consists of tt and W+jets events as well as multi hadronic jet events from QCD). An analysis in this spirit might also be useful for searching for our H →t ′ t ′⋆ → (bW − ) t ′ bW + signal at the LHC.
However, we should stress that the event topologies and the background problems are expected to be more serious at the LHC and are likely to represent a serious challenge. Nonetheless, we are cautiously optimistic that it can be handled.

Flavor changing Higgs decays to a single 4th generation quark
In the SM the FC decay H → tc is severely suppressed by the GIM-mechanism, leading to an un-observably small BR SM (H → tc) ∼ 10 −13 [23]. On the other hand, in the SM4, we expect this decay mode and also the FC decays involving the 3rd and 4th generation quarks, H → t ′t and H → b ′b , to be significantly enhanced due to the extra heavy 4th generation quarks in the loops which essentially removes the GIM suppression. This can be seen by estimating the width for the 1-loop FC decay H → Uū in the limit that only the heaviest down-type quark (D) runs in the loop and its contribution is multiplied by the appropriate GIM-suppression factor (see e.g., [24]): For example, we can use (15) to estimate the ratio between Γ SM (H → tc) (where D = b) and Thus, taking m b ′ = 500 GeV, V t ′ b ′ = V tb = 1 and V tb ′ ≡ θ 34 ∼ 3V cb ∼ 0.15, and assuming that the total width of the heavy Higgs in the SM and in the SM4 is the same (this roughly holds for m H < 2m b ′ , 2m t ′ ), we expect BR SM 4 (H → t ′t ) ∼ 10 9 · BR SM (H → tc) ∼ 10 −4 , which, as we will show below, is indeed the case. On the other hand, for the case of H → tc in the SM4, we expect (using the above values for the CKM elements involved and for m b ′ ) BR SM 4 (H → tc) ∼ V 2 cb ′ · 10 9 · BR SM (H → tc), which at best gives BR SM 4 (H → tc) ∼ 10 6 · BR SM (H → tc) ∼ 10 −7 once V cb ′ is set to its largest allowed value V cb ′ ∼ 0.03 − 0.04, see e.g. [10].
This expected enhancement in FC transitions involving the 4th family heavy quarks of the SM4, has ignited a lot of activity in the past two decades. However, previous studies of FC effects in the SM4 have assumed that m t ′ , m b ′ > m H and, therefore, focused on the FC decays b ′ → bH, bV [9,10,25,26] and t ′ → tH, tV [9,10], where V = g, γ, Z. The BR's for these t ′ and b ′ FC decays were found to be typically within the range of 10 −4 − 10 −2 . The authors of [9,10] have also studied the (SM-like) FC decays t → cH, cV within the SM4 and found that, in spite of the many orders of magnitudes enhancement, these FC top decays remain below the LHC sensitivity, i.e., having a BR typically smaller than 10 −7 .
Here, motivated by the estimate in (16), we wish to study the FC heavy Higgs decays in (2): H → t ′t and H → b ′b . Recall that our working assumption is that GeV, in which case the 4th generation quarks are expected to decay mainly via t ′ → bW and b ′ → tW . Thus, both FC Higgs decays will lead to the final state bbW + W − , but with different kinematics: We base our results below on the explicit analytical expressions that were given for the decay H → b ′b in [25]. In Fig. 3 we plot the BR(H → t ′t + h.c.) and BR(H → b ′b + h.c.), as a function of m t ′ , for θ 34 = 0.15 and for Higgs masses between 600 to 900 GeV. We see, for example, that for m t ′ ∼ 500 GeV these BR's are at the level of few × 10 −4 if m H ∼ 800 GeV and θ 34 ∼ 0.15, which is the value of θ 34 required in order for the SM4 to fit EW precision measurements when m H ∼ 800 GeV [6]. Note that in Fig. 3 we did not consider the finite width effects of the heavy Higgs. We thus expect the BR's in Fig. 3 to be enhanced by an additional factor of a few, i.e., reaching 10 −3 , when these FC signals are integrated over some finite range around m H in the Higgs production process, e.g., integrated over in the previous section). Given that the LHC with a luminosity of O(100) fb −1 will be able to produce about 10 5 TeV-scale Higgs particles (see Fig. 1), one naively expects tens of such events a year, if indeed θ 34 ∼ 0.15.  We have explored some of the phenomenological implications for heavy Higgs physics in the SM4 within what we named the "three-prong composite solution" to EW precision data, which accommodates the possibility of a composite Higgs: , and we have assumed that m H < 2m b ′ , 2m t ′ in accordance with compositeness.
We focused on the flavor diagonal H →t ′ t ′⋆ →t ′ bW + and H →b ′ b ′⋆ →b ′ tW − 3-body decay channels and the flavor changing (one-loop) H → t ′t , b ′b decays of the composite Higgs to a single heavy 4th family quark. We are cautiously optimistic that these signals can be observed at the LHC, in spite of the decrease in the production cross-section for such a heavy Higgs. In particular, the one-loop FC decay channels are essentially "GIM-free" and can, therefore, reach a branching ratio as large as BR ∼ 10 −4 − 10 −3 if θ 34 ∼ O(0.1), while the rate of the flavor diagonal channels is dramatically enhanced due to large width effects of the decaying Higgs -potentially yielding 10 4 − 10 5 such events with a luminosity of O(100) inverse fb, which is comparable to the number of events expected from the QCD continuum production oft ′ t ′⋆ andb ′ b ′⋆ .
These mechanisms for producing single heavy 4th family fermions are particularly interesting for the LHC when applied to the leptonic sector, due to the absence of the QCD background. As noted in [5] (see also [27]), the Higgs decay to a pair of on-shell ν ′ followed by ν ′ → ℓW , can yield a rate to a four lepton final state (via H → ν ′ν′ → 4ℓ+ E T ) which is comparable to the rate expected from the "golden mode" H → ZZ → 4ℓ, if the mixing between the 4th generation leptons with the lighter leptons (via the charged current U i4 ℓ ± i ν ′ W ∓ , i = 1, 2 or 3) is not exceedingly small. However, if m H < 2m ν ′ , 2m ℓ ′ , then the decays H → ν ′ν′ , ℓ ′l′ are kinematically not open and, as in the quark sector, the Higgs Yukawa couplings to the 4th generation leptons can be probed only through its decays to a single ν ′ and ℓ ′ . For instance, applying our results above to the leptonic channels, e.g., taking m H ∼ 800 GeV and m ν ′ ∼ 500 GeV we expect the BR to the flavor diagonal 3-body decay H → ν ′ν′⋆ , followed by ν ′ → ℓW , to be dramatically enhanced due to the large width effects around the heavy Higgs resonance in gg → H → ν ′ ℓW (independent of the mixing angle U i4 , see section 3), thus yielding about 10 4 H → ν ′ ℓW → (ℓW ) ν ′ ℓW events at the LHC with an integrated luminosity of O(100) inverse fb. When combined with the BR of the W's into leptons, this will again yield a 4ℓ + E T signal comparable to that of the golden mode H → ZZ → 4ℓ.
The FC decays H → ν ′ν i and H → ℓ ′l i are expected to have a BR of O(10 −4 ) if U i4 ∼ O(0.1), i.e., similar to the corresponding FC decays in the quark sector. However, the leptonic mixing angles between the 4th generation leptons and the SM light leptons are unfortunately expected to be at best U i4 < ∼ O(0.01) [1,4]. Finally, let us comment on the possibility of searching for new CP-violation effects via our Higgs decays to single 4th generation fermions. As is well known, the extension of a 4th generation of fermions to the SM adds two new phases to the (now 4x4) CKM, which provide new sources of CP-violation [28]. Indeed, as was recently shown in [31], the SM4 may be a natural framework for accommodating the observed flavor and CP structure in nature, and for addressing the CPproperties associated with the 4th generation quarks. Understanding the new CP-violating sector of the SM4 may also shed light on CP-anomalies in K and b-quark systems [9,11,29] and on baryogenesis [30]. Effects of the new SM4 CP-violating phases can be searched for directly in t ′ and b ′ systems as was recently noted in [10,32]. In particular, studying CP-violation in these heavy 4th generation quark systems may help to pin down the single dominating CP-violating quantity associated with the 4th generation quarks at high-energies [33]. In this respect our single t ′ and b ′ production channels via a heavy Higgs resonance, gg → H → t ′ bW, b ′ tW , may be rich in exhibiting various types of CP-asymmetries in analogy to CP-violation in single-top production [34].