Search for t → ch at e + e − Linear Colliders

We study the rare top-quark decay t → ch , where h is a generic Higgs boson, at a linear collider. If kinematically accessible, all models contain this decay at some level due to quark ﬂavor mixing. Some models, such as Model III of the two-Higgs doublet model, have a tree-level top-charm-Higgs coupling, and the branching ratio is close to 0 . 5%. Others, such as the MSSM, have a coupling induced at one-loop, and can have a branching ratio in the range of 10 − 5 − 5 × 10 − 4 . We ﬁnd that a linear collider of √ s = 500 GeV and a luminosity of 500 fb − 1 will begin to be sensitive to this range of the coupling.


I. INTRODUCTION
Two of the most important unanswered puzzles of elementary particle physics are the nature of electroweak symmetry breaking and the origin of flavor.Both are implemented in the Standard Model of particle physics in an ad hoc manner.Often, extensions of the Standard Model (SM) treat these two problems separately; extended Higgs models do not address the flavor problem and flavor symmetry models do not substantively address the nature of electroweak symmetry breaking.Yet, there are no dynamical models satisfactorily addressing the two puzzles coherently.Since the top quark is so heavy, with a mass on the order of the electroweak symmetry breaking scale, it is tempting to consider the possible special role of the top quark in the electroweak symmetry breaking sector and in the quark flavor sector, and its interactions could shed light on both problems.Flavor-changing-neutral-current (FCNC) decays of the top quark [1] could be very sensitive to various extensions of the Standard Model that address both of these puzzles.
Most studies of flavor-changing-neutral-current top-quark decays have examined the decay of the top quark into a charmed quark and a gauge boson [2] (either a gluon, photon or a Z).However, the flavor-changing-neutral-current decay most likely to shed light on the nature of electroweak symmetry breaking is the decay of the top quark into a charmed quark and a Higgs boson [1,3] (or the decay of the Higgs boson into a top and a charmed quark if kinematically preferred).It is this interaction that we study in this paper.
All models will contain a top-charm-Higgs (TCH) vertex at some level 1 .Even in the Standard Model, a one-loop diagram gives a contribution, but it is GIM-suppressed and extremely small [3].In extensions of the Standard Model, however, such a vertex can occur either at tree level or at a non-GIM-suppressed one-loop level.
One of the simplest extensions of the Standard Model has a tree-level TCH vertex.This is the general two-Higgs doublet model, referred to as Model III, in which no discrete symmetries are imposed to avoid flavor-changing neutral currents.In this case, the quark Yukawa coupling matrices consist of two parts, y = y 1 + y 2 , where y i is the coupling to the i'th Higgs doublet.Since diagonalizing y will not, in general, diagonalize y 1 and y 2 , there will be tree-level flavor-changing neutral currents.Cheng and Sher [4] noted that if one wishes to avoid fine-tuning, then the observed structure of the mass and mixing hierarchy suggests that the FCNC vertex ξ ij qi q j h is given by the geometric mean of the Yukawa couplings of the quark fields2 1 Whether a TCH vertex exists does, in part, depend on nomenclature.Strictly speaking, one should consider the "Higgs" to be the other member of the isodoublet containing the longitudinal components of the weak vector bosons; after all, it is this field that is associated with spontaneous symmetry breaking.With that definition, the Yukawa couplings are necessarily flavor-diagonal.Here, we use the word "Higgs" to indicate a scalar field whose mass eigenstates contain some part of the field associated with spontaneous symmetry breaking.
where v = 246 GeV is the weak scale, and the λ ij are naturally all of O(1).This ansatz has no conflict with current phenomenological observation, and yet predicts rich physics in the heavy-flavor generations.Under this ansatz, the top-charm-scalar coupling ξ ct is the biggest in the model, and is approximately 0.05.Model III is fairly general.A number of more specific models have a large TCH coupling as well.Burdman [5] studied a topcolor model [6], which gave a large TCH coupling of the same order as Model III (although his "Higgs" boson had a mass larger than that of the top quark).A topflavor model [7] analyzed the flavor-changing coupling of the top and charm to a top-pion, finding a TCH coupling that is substantially larger than that of Model III.Recently, a bosonic topcolor model [8] was proposed that automatically has the TCH coupling as the largest fermionic coupling of the Higgs boson.An extensive analysis of models with additional singlet quarks (Q) by Higuchi and Yamamoto [9] gives a flavorchanging coupling which is proportional to the product of the cQ and tQ mixings; this can be larger than the Model III coupling.In general, in models which treat the top quark differently than other quarks motivated by its weak-scale mass, one might expect TCH coupling of Model III size or larger.
There have been numerous studies of Model III with the above ansatz, examining K, B, µ and τ decays, as well as K 0 , B 0 and D 0 mixing [10].Here, however, we will be concerned with the large TCH coupling only.
The rate for the decay t → ch was first calculated by Hou [11].For a top-quark mass of 175 GeV, the branching ratio of t → ch to t → bW + is This implies that the branching fraction of t → ch can be typically of 10 −3 or higher for λ ct ∼ O (1).If the Higgs is heavier than the top, one can have h → tc + ct, as first discussed in Ref. [11].This leads to the process e + e − → h 0 A 0 → tt cc + cctt at a linear collider [12], to like-sign top-pair production at the LHC [13], and to the linear collider process e + e − → νe ν e ( tc + ct) [14].These processes all assume a Higgs boson heavier than the top, however, and recent electroweak precision data [15], as well as hints from LEP [16], prefer the scenario of a rather light Higgs boson, making t → ch kinematically accessible.As one can see from Eq. ( 2), the branching ratio for λ ct = 1 and a 115 GeV Higgs mass is 3 × 10 −3 .
If one looks at one-loop processes, there are many more possibilities.Most extensions of the Standard Model will avoid the GIM suppression that makes the coupling so small in the Similarly, we have not worried about the running mass effects at different renormalization scales, and we simply take the pole masses for m t and m c for our analysis.Standard Model [3].The most popular extension is the MSSM.In this model, there must be a Higgs boson lighter than the top quark, and so the decay will occur.An extensive analysis of t → cH, where H refers to any one of the three neutral scalars in the MSSM, was carried out in Ref. [17].They show that the dominant contribution comes about from loops with gluinos and squarks, and the branching ratio ranges from 10 −5 to 5 × 10 −4 over the MSSM parameter space.In an R-parity violating SUSY model, a one-loop contribution can also give [18] a branching ratio as large as 10 −5 .
We see that models with a tree-level TCH coupling have branching ratios of the order of 10 −3 − 10 −2 , and models with a one-loop coupling can have branching ratios of the order of 10 −5 − 10 −4 .What level is experimentally detectable at high energy colliders?Recently, discovery limits at the LHC were calculated [20] and are approximately 5 × 10 −5 .While the results are quite encouraging, detailed Monte Carlo simulation would be needed to draw a definitive conclusion due to the complicated background issue at hadron colliders.In a linear collider, backgrounds are much more manageable, although signal cross sections are lower.In this paper, we assume that a light Higgs boson will be observed and examine in detail the discovery potential for its FCNC coupling at a linear collider.

A. Production Cross Section
Using the coupling constant given in Eq. ( 1), we wish to explore the limits obtainable on λ ct at a linear collider.We consider the process where the corresponding Feynman diagrams are shown in Fig. 1.The dominant contribution to process (3) is from e + e − → t t followed by t ( t) decay into ch (ch), namely σ(e The decay branching ratio is calculated and shown in Fig. 2(a), where we see that for m h = 120 GeV and λ ct = 1, t → ch has a branching ratio of 2.8 × 10 −3 .At tree level, top-pair production has a total cross section of 580 fb at the center-of-mass energy √ s = 500 GeV.Thus each channel will have a cross section of 1.6 fb.The total cross section of e + e − → tch and tch is presented in Fig. 2(b) as a function of ( The primary background to the above signal is from the SM top-pair production e + e − → t t → bW + bW − , with one W decaying hadronically and the other decaying either hadronically or leptonically, depending on which signal of Eqs. ( 4) and (5) we are looking at.In order to identify the signal from the background, we require that 3 b's be tagged.The efficiency for a single b tagging is taken to be 65% [19].This still does not eliminate the background: For the case that both W 's decay hadronically, one out of the four non-b jets from W decay to light quarks may be misidentified as a b-jet.We assume the misidentification to be 1% for each jet.Thus for four jets the total misidentification probability is 4%.Similarly, for the case that one of the two W 's decays leptonically, one of the two non-b jets from W decay to light quarks is misidentified as a b-jet.The SM background induced by one W decay into cb is negligible due to the small size of the CKM mixing matrix element V cb ∼ 0.04.The entries of the first row in Table I  GeV for signal (m h = 120 GeV and λ ct = 1) and hadronic, leptonic and total backgrounds before and after cuts.
including decay branching fractions and b-tagging efficiencies.We find that the background rate is still overwhelming.
We next try to make use of the distinctive signal kinematics by reconstructing h, t, t and W masses from the final state momenta.To simulate the detector response, we smear the jet energies according to a Gaussian spread where ⊕ denotes a sum in quadrature.For the signal, we first consider the case where W decays into two jets.Of the three tagged b's, we choose two b's whose invariant mass is closest to m h and identify them as the b's coming from h decay (M rec h ).Then from the three light jets we pick the jet that, together with b b selected above, produces an invariant mass (M rec t ) that is closest to m t .This jet is now identified as c from t → cb b decay.The remaining two jets can be used to construct the W mass M rec W and with the third b, we have M rec t .If W decays into lν, it is simpler.We simply need to choose two of the three b-jets to construct M rec h .The only non-b jet is identified as c.We get M rec W from the momenta of ℓ ± and missing energy assigned to ν, after adding the momentum of the third b, we have M rec t .For the background in which W decays hadronically, we randomly pick one from the 4 non-b jets and assume it to be misidentified as a b-jet.Then we follow the same construction as for the signals, we will get M rec h , M rec t , M rec t and M rec W .For the background when W decays leptonically, we randomly choose one of the two non-b jets to be misidentified as b and again follow the analysis for the signals to get the four reconstructed invariant masses.The normalized differential distribution with respect to these reconstructed masses are shown in Fig. 3 for signals and backgrounds.The characteristic difference between the signal and background is evident from these mass distributions.The first crucial reconstructed mass is M rec h .A peak structure in this variable provides the confidence of the signal observation and gives the discrimination power between the signal and background.The wide width is due to the jet-energy smearing discussed earlier.The other distinctive mass variable is M rec t = M bbc , which reconstructs m t for the signal and no structure for the background due to the incorrect jet combination.Similarly, M rec t ≈ m t for the signal but it spreads out for the background, as seen in Fig. 3(c).Those distributions motivate us to device more judicial kinematical cuts.Our basic kinematical cuts are listed in Table II based on these four reconstructed masses.
Additional cuts can be applied to further increase the signal-to-background ratio.We first notice that the charm-jet energy from the top decay is monotonic in the top-rest frame as a result of a two-body decay which is about 50 GeV for m h = 120 GeV.In contrast, the faked charm jet from W decay will have an energy more spread out.The normalized distributions are shown in Fig. 4. Figure 4(a) is the E c distribution in the e + e − −lab frame which presents a nearly clear range due to the Lorentz boost of the top quark motion, where the sharp end-point in E c is sensitive to m h , that may provide an independent kinematical determination for m h .Figure 4(b) shows the E c distribution in the top-quark rest frame based on the t → b bc reconstruction.We see that E c is monotonic near 50 GeV modulus to the jet-energy resolution.This provides a good discriminator for the signal and background.In fact, the reconstructed Higgs boson  .We also look at the two b-jets coming from h decay and examine the harder and softer ones of the two in energy, separately, as illustrated in Figs.4(c) and 4(d).The optimal cuts are given in Table II.After applying the basic and additional cuts the signal-to-background ratio improves significantly, as summarized in the last two rows of Table I.

C. Sensitivity to the tch coupling
Given the efficient signal identification and substantial background suppression achieved in the previous section, we now estimate the sensitivity to the FCNC tch coupling from this reaction using Gaussian statistics, which is applicable for large signal event samples.We define the statistical significance by with N S and N B being the number of signal and background events.We note that σ = 3 (called 3σ) approximately corresponds to a 99% confidence level.Figure 5 presents the 3σ (2σ) sensitivity to the FCNC couplings by the solid (dashed) curve as a function of integrated luminosity for √ s = 500 GeV and m h = 120 GeV.Recall that at λ ct = 1, the t → ch branching ratio is about 2.8 × 10 −3 .Such an order of magnitude can be anticipated in models with tree-level FCNC couplings, and can be easily observed at a LC with an integrated luminosity of less than 40 fb −1 .As λ ct varies, the branching ratio scales like 2.8 × 10 −3 λ 2 ct .For 1-loop induced FCNC decays such as in SUSY models, the branching ratios can be about 10 −5 − 5 × 10 −4 , corresponding to λ ct of 0.06 − 0.4.A linear collider with 500 fb −1 integrated luminosity will begin to be sensitive to this range of the coupling of λ ct ≈ 0.4 (0.3) at a 3σ (2σ) level, but a higher luminosity will be needed to extend the coverage of the parameter space to a level about 0.2.

III. CONCLUSIONS
Models with a top-charm-Higgs coupling come in two categories: Those with a treelevel coupling and those with a coupling induced at one-loop.In the former case, such as the Model III two-Higgs doublet model, one expects λ ct to be of O (1).If the Higgs is somewhat lighter than the top quark, then the t → ch decay will be detected within the first 40 fb −1 of running of a linear collider, as indicated in Fig. 5.In the latter case, such as supersymmetric models (in which we know that one of the Higgs bosons will be sufficiently light), the branching ratios can be in the range of 10 −5 to a few times 10 −4 .This corresponds to a λ ct of 0.06 − 0.4.A linear collider with 500 fb −1 integrated luminosity will begin to be sensitive to this range of the coupling at a 3σ level, and higher luminosity will be needed to substantially extend the coverage of the parameter space.

FIG. 2 .
FIG. 2. (a) Branching ratio of t → ch decay as a function of m h and (b) Cross section for tch + tch production as a function of center-of-mass energy in e + e − collisions for m h = 120 GeV.√ s, where and henceforth m h = 120 GeV is chosen for illustration.The total cross section scales as λ 2 ct and is of order of 1 fb for λ ct = 1.

FIG. 3 .
FIG. 3. Normalized distribution for the signal (solid), hadronic background (dashes) and leptonic background (dots) with respect to the reconstructed masses (a) M rec h , (b) M rec t , (c) M rec t

FIG. 5 .
FIG. 5. 3σ sensitivity (solid) to the FCNC tch couplings at √ s = 500 GeV for m h = 120 GeV as a function of integrated luminosity.The dashed curve is for 2σ sensitivity.

TABLE II .
The basic and additional kinematical cuts applied to our signal-to-background optimization (all values in GeV).