Impact of squark generation mixing on the search for gluinos at LHC

We study gluino decays in the Minimal Supersymmetric Standard Model (MSSM) with squark generation mixing. We show that the effect of this mixing on the gluino decay branching ratios can be very large in a significant part of the MSSM parameter space despite the very strong experimental constraints on quark flavour violation (QFV) from B meson observables. Especially we find that under favourable conditions the branching ratio of the the QFV gluino decay gluino ->c bar{t} (bar{c} t) + neutralino_1 can be as large as about 50%. We also find that the squark generation mixing can result in a multiple-edge (3- or 4-edge) structure in the charm-top quark invariant mass distribution. The appearance of this remarkable structure provides an additional powerful test of supersymmetric QFV at LHC. These could have an important impact on the search for gluinos and the determination of the MSSM parameters at LHC.


Introduction
The search for supersymmetric (SUSY) particles will have a very high priority at the Large Hadron Collider (LHC) at CERN. If weak scale SUSY is realized in nature, gluinos and squarks, the SUSY partners of gluons and quarks, will have high production rates for masses up to O(1 TeV). The main decay modes of gluinos and squarks are usually assumed to be quark-flavour conserving (QFC). However, the squarks are not necessarily quark-flavour eigenstates and they are in general mixed by a 6 × 6 matrix. In this case quark-flavour violating (QFV) decays of gluinos and squarks could occur.
The effect of QFV in the squark sector on reactions at colliders has been studied only in a few publications. The pair production of quarks with different flavours at the LHC is studied in [1]. The QFV effect can also be probed in the top quark decay [2].
In all of these studies the external particles of the reactions are Standard Model (SM) particles (or SUSY Higgs bosons). This means that the effect of QFV in the squark sector is induced only by SUSY particle (sparticle) loops.
In sparticle reactions, on the other hand, the effect of QFV in the squark sector may be especially strong as they already occur at tree-level. The QFV decayt 1 → cχ 0 1 [4] and QFV gluino decays [5] were studied in the scenario of minimal flavour violation (MFV), where the only source of QFV is the mixing due to the Cabibbo-Kobayashi-Maskawa (CKM) matrix. Note that the decayt 1 → cχ 0 1 is actually the standard Tevatron search mode for light top-squarks. In [6,7] squark pair production and their decays at LHC have been analyzed including also the effect of the squark generation mixing.
In the present Letter, we study the effect of mixing between the second and third squark generations in its most general form. More precisely, we study the influence of the mixing of charm squark and top squark on the gluino and squark decays. In particular, we calculate the branching ratios of the following gluino decays into two quarks plus neutralino via up-type squark decay (see Fig.1 1 As we always sum over the particles and antiparticles of the (s)quarks, we do not indicate if it is a particle or its antiparticle: qq ′ (with q = q ′ ) means qq ′ andqq ′ , and qq means qq, e.g.
We show that the QFV gluino decay branching ratio B(g → ctχ 0 1 ) can be very large (up to ∼ 50%) due to the squark generation mixing in a significant part of the MSSM parameter space despite the very strong experimental constraints from B factories, Tevatron and LEP 2 . We also study the effect of the squark generation mixing on the invariant mass distributions of the two quarks from the gluino decay at LHC. We show that it can result in novel multiple-edge structures in the distributions 3 .
These effects could have an important impact on the search for gluinos and the MSSM parameter determination at LHC.

Squark mixing with flavour violation
Here we summarize the MSSM parameters in our analysis. The most general up-type squark mass matrix including left-right mixing as well as quark-flavour mixing in the where the three 3 × 3 matrices read The indices α, β = 1, 2, 3 characterize the quark flavours u, c, t, respectively. M 2 Qu and M 2 U are the Hermitean soft-SUSY-breaking mass matrices for the left and right up-type squarks, respectively. Note that in the super-CKM basis one has M 2 Qu = K·M 2 Q ·K † due to the SU(2) symmetry, where M 2 Q is the Hermitean soft-SUSY-breaking mass matrix for the left down-type squarks and K is the CKM matrix. Note also that M 2 Qu ≃ M 2 Q as K ≃ 1. A U is the soft-SUSY-breaking trilinear coupling matrix of the up-type µ is the higgsino mass parameter. v 1,2 are the vacuum expectation values of the Higgs fields with v 1,2 / √ 2 ≡ H 0 1,2 , and tan β ≡ v 2 /v 1 . m uα (u α = u, c, t) are the physical quark masses.
(iii) The LEP limits on the SUSY particle masses [18]: mχ± where A 0 is the CP-odd Higgs boson and h 0 is the lighter CP-even Higgs boson.

Quark flavour violating gluino decays
We study the effect of the 2nd and 3rd generation squark mixing on the gluino decays.
We focus on the QFV gluino decays of Eq.(1) leading to the same final state c tχ 0 1 . We calculate the gluino and squark decay widths taking into account the following two-body decays: where u k = (u, c, t) and d k = (d, s, b). The squark decays into the heavier Higgs bosons are kinematically forbidden in our scenarios studied below. The formulae for the twobody decays in (13) can be found in [6], except for the squark decays into the Higgs bosons for which we take the formulae of [23] modified appropriately with the squark mixing matrix in the general QFV case.
We take tan β, m A 0 , M 1 , M 2 , mg, µ, M 2 Qαβ , M 2 U αβ , M 2 Dαβ , A U αβ and A Dαβ as the basic MSSM parameters at the weak scale. We assume them to be real. The QFV parameters are the squark generation mixing terms M 2 Qαβ , M 2 U αβ , M 2 Dαβ , A U αβ and A Dαβ with α = β. As a reference scenario, we take the scenario given in Table 1.
This scenario is within the reach of LHC and satisfies the conditions (i)-(v). For the  Table 2. We show the up-type squark compositions in the flavour eigenstates in Table 3.
For the important branching ratios of the gluino and squark two-body decays we 139 264 800 1000 10 800 138 261 1003 1007 261 1007 Table 2: Sparticles and corresponding masses (in GeV) in the scenario of Table 1.
We now study the basic MSSM parameter dependences of the QFV gluino and squark decay branching ratios for the reference scenario of Table 1. In Fig.2 Table 1. We see that the QFV decay branching ratio B(g → ctχ 0 1 ) quickly increases up to ∼ 50% with increase of the effectivec R −t R mixing angle tan(2θ ef f In Fig.3  show the corresponding branching ratio contours. All basic parameters other than M 2 Q23 and M 2 U 23 are fixed as in our reference scenario defined in Table 1. We see that the QFV decay branching ratio B(g → ctχ 0 1 ) increases quickly with increase of thẽ c R −t R mixing parameter |δ uRR 23 | and can be very large (up to ∼ 50 %) in a significant part of the δ uLL Studying the branching ratios of the gluino and up-type squark two-body decays separately allows for a better understanding of their contributions to the QFV gluino decayg → ctχ 0 1 . In Fig.4 we show the δ uRR

23
(i.e.c R −t R mixing parameter) dependences of the gluino and squark decay branching ratios, where all basic parameters other than M 2 U 23 are fixed as in the scenario of Table 1. We see that B(g → ctχ 0 1 ) increases quickly with increase of |δ uRR 23 | for |δ uRR 23 | < ∼ 0.1 and can be very large (∼ 50%) in a wide range of δ uRR 23 . This behaviour can be explained by an argument similar to that below Eq.(16). In Fig.4(b) [(c)] we see that B(g →ũ i c) and B(g →ũ i t) [B(ũ i → cχ 0 1 ) and B(ũ i → tχ 0 1 )] with i = 1, 2 are large in a wide range of δ uRR 23 , except for B(g →ũ 2 t) which is kinematically suppressed. This leads to the very large B(g → ctχ 0 1 ) in a wide range of δ uRR 23 (see Eq. (14)).
In Fig.5 we show the δ uRL 23 (i.e.c R −t L mixing parameter) dependences of the gluino decay branching ratios, where all basic parameters other than A U 32 are fixed as in the scenario of Table 1. We see that the QFV decay branching ratio B(g → ctχ 0 1 ) can be quite large (∼ 30-50%) in a wide range of δ uRL 23 . B(g → ctχ 0 1 ) decreases (down to ∼ 30%) and the quark-generation violating (QGV) decay branching ratio B(g → cbχ ± 1 ) increases (up to ∼ 20%) with increase of |δ uRL 23 |. Sizable δ uRL

Impact on collider signatures
Here we study the invariant mass distributions (i.e. the differential decay branching ratios) dBr(g →ũ i u j → u j u kχ 0 n )/dM u j u k , with M u j u k being the invariant mass of the two quark system u j u k in the final state. The kinematical endpoinds of the distributions are given in terms of the masses of the involved particles by [24] M i(min,max) with λ(x, y, z) = x 2 + y 2 + z 2 − 2(xy + xz + yz), whereũ i is the intermediate squark, u j is from the primary decay (i.e. the two-bodyg decay) and u k is from the secondary decay (i.e. theũ i decay). Note that M i(min,max) Note that the individual distribution dBr(g →ũ i u j → u j u kχ 0 n )/dM u j u k (dBr(g → u i u k → u k u jχ 0 n )/dM u j u k ), is proportional to M u j u k and its allowed range is given by In the following we show how QFV due to the 2nd and 3rd generation mixing of the up-type squarks influences the invariant mass distributions. We discuss two scenarios, one with gluino mass mg = 800 GeV and the other with mg = 1300 GeV.
We start from the QFV scenario with mg = 800 GeV given in Table 1. In this QFV scenario the squark mass eigenstatesũ 1 andũ 2 are a strong mixture of the flavour eigenstatesc R andt R . First we consider the invariant mass distribution for a final state including two top quarks. Fig.6 shows the invariant mass distributions of the top quark pairs for the QFV scenario, where one has B(g → ttχ 0 1 ) = 12.0%. Note that the invariant mass distribution of the two top quarks in the QFV scenario shows no additional edge structure. This is because only the lightest up-type squark,ũ 1 , can mediate this final state while the other squarks are too heavy.
Next we consider the invariant mass distribution for a final state including c and t quarks in the QFV scenario of Table 1, where one has B(g → ctχ 0 1 ) = 46.3%. Fig.6 shows the invariant mass distribution of ct. There are more edge structures due to the processesg →ũ 1 t → tcχ Next we consider the invariant mass distribution of final state quarks for a QFV scenario with a heavier gluino (mg = 1300 GeV) given in Table 4. This scenario is inspired by the mSUGRA scenario A of Ref. [25] and satisfies all of the conditions (i)- (v) in section 3. The resulting masses of squarks, neutralinos and charginos are given in Table 5. We show the corresponding up-type squark compositions in the flavour eigenstates in Table 6. In this scenario the squark mass eigenstateũ 1 (ũ 2 ) is dominated by a strong mixture of the flavour eigenstatest R andc R (t L andc L ). In Fig.7 we show the two invariant mass distributions of tt and ct, where one has B(g → ttχ 0 1 ) = 16.6%, and B(g → ctχ 0 1 ) = 31.4%. Note that the QFV decay branching ratio B(g → ctχ 0 1 ) is large.
The invariant mass distribution of two top quarks shows no additional edge structure for the same reason as in the scenario with mg = 800 GeV discussed above.
The decayg →ũ 2 t is kinematically allowed but phase-space suppressed. Moreover, u 2 → tχ 0 1 is strongly suppressed becauseũ 2 (∼t L +c L ) does not significantly couple tõ χ 0 1 (∼B 0 (Bino)) in this scenario. Hence, B(g →ũ 2 t → ttχ 0 1 )(=0.00035) is very small. As for the invariant mass distribution of c and t quarks in the QFV scenario of Table   4, there are more edge structures due to theũ 1 -mediated processesg →ũ 1 t → tcχ Finally, we briefly discuss the measurability of the QFV decayg → c tχ 0 1 at LHC. It is important whether one can discriminate between the QFV decayg → c tχ 0 1 and the QFC decayg → t tχ 0 1 . Therefore, it is necessary to identify the top quarks in the final states. This is possible by using the decay t → bW with the W decaying into 253 483 758 775 482 774 Table 5: Sparticles and corresponding masses (in GeV) in the scenario of Table 4.
two jets. For this purpose, a special method was proposed in [24], where it is assumed that the masses of the gluino and theχ 0 1 are known from other measurements. The signature of the decayg → c tχ 0 1 would be 'charm-jet + top-quark + missing-energy'. Therefore, charm-tagging also would be very useful. If this is not possible, one should search for the decayg → q tχ 0 1 (q = t), i.e. for the signature 'jet + top-quark + missing-energy'. In the scenarios discussed, the most important SUSY background would be due to the QFC decayg → t tχ 0 1 and the pair production of the lightest uptype squarks, pp →ũ 1 +ũ 1 + X, withũ 1 → cχ 0 1 andũ 1 → tχ 0 1 . The most important SM background would be top-quark pair production. For the measurement of the endpoints in the multiple edge structure a good energy/momentum resolution of the detector would be necessary. In any case, one should take into account the possibility of significant contributions from QFV decays in the gluino search. Moreover one should also include the QFV squark parameters in the determination of the basic

Conclusion
To conclude, we have studied gluino decays in the MSSM with squark mixing of the second and third generation, especiallyc L/R -t L/R mixing. We have shown that QFV gluino decay branching ratios such as B(g → c tχ 0 1 ) can be very large due to the squark mixing in a significant part of the MSSM parameter space despite the very strong experimental constraints from B factories, Tevatron and LEP with those of b → sγ and ∆M Bs being especially important.
We have also studied the effect of the squark generation mixing on the invariant mass distributions of the two quarks from the gluino decay at LHC. We have found that it can result in novel and characteristic edge structures in the distributions. In particular, multiple-edge (3-