Flavor Changing Heavy Higgs Interactions with Leptons at Hadron Colliders

In a general two Higgs doublet model, we study flavor changing neutral Higgs (FCNH) decays into leptons at hadron colliders, $pp \to \phi^0 \to \tau^\mp\mu^\pm +X$, where $\phi^0$ could be a CP-even scalar ($h^0$, $H^0$) or a CP-odd pseudoscalar ($A^0$). The light Higgs boson $h^0$ is found to resemble closely the Standard Model Higgs boson at the Large Hadron Collider. In the alignment limit of $\cos(\beta-\alpha) \cong 0$ for $h^0$--$H^0$ mixing, FCNH couplings of $h^0$ are naturally suppressed, but such couplings of the heavier $H^0, A^0$ are sustained by $\sin(\beta-\alpha) \simeq 1$. We evaluate physics backgrounds from dominant processes with realistic acceptance cuts and tagging efficiencies. We find promising results for $\sqrt{s} = 14$ TeV, which we extend further to $\sqrt{s} = 27$ TeV and 100 TeV future pp colliders.


I. INTRODUCTION
Recent 13 TeV studies at the LHC by ATLAS and CMS experiments confirm that the properties of the 125 GeV Higgs boson are in good agreement with the expectations from the Standard Model (SM) Higgs boson [1,2]. This is in sharp contrast to the persistent signs of significant deviation from SM in the flavor sector. The 3-4σ hints of lepton universality violation, in simple tree-level semi-leptonic B decays as well as in flavor-changing loop processes, have been very much in the news [3]. Moreover, for over a decade now the muon anomalous magnetic moment measurement at BNL [4] also seem to show about 3.5σ deviation from SM. While so far none of these constitutes compelling evidence against the SM, but even if just one of them pans out, it would be physics beyond SM. In particular, it is important to recall that lepton universality is purely an accidental symmetry of SM. These underpinnings prompt us to question lepton universality and lepton flavor violation in the Higgs sector itself.
Our investigation was motivated by the experimental 2σ hint for h 0 → τ µ from CMS [5]. While it has subsequently disappeared [6], it in fact motivates further the search for H 0 , A 0 → τ µ, involving heavy exotic Higgs bosons, as we shall explain. Note in particular that, in face of the current semileptonic anomalies in B decays, a general two Higgs doublet model (g2HDM) had been invoked [7] over the disfavored conventional Type II of two Higgs doublet models (2HDM-II). While the situation with the anomalies are as yet inconclusive, we adopt the g2HDM set up in this work, i.e. without the usual Z 2 symmetry to forbid flavor changing neutral Higgs (FCNH) couplings. Another mechanism may be at work instead of Z 2 or Natural Flavor Conservation [8]: alignment [9]. Removing interactions of the extra scalars with vector boson pairs (H 0 W W and H 0 ZZ), other than the SM-Higgs, is known as the alignment limit [10][11][12]. Influenced by the LHC results on the 125 GeV boson [1,2], we will assume that one must work close to this limit.
We seek the discovery of the leptonic flavor changing decay, specifically φ 0 → τ µ, where φ 0 = h 0 , H 0 , A 0 . In SM, h 0 → τ µ is highly suppressed at loop level by the extremely tiny neutrino masses, but in g2HDM without any Z 2 symmetry, this decay is in principle possible at tree level. We adopt the following interaction Lagrangian [13,14], where and α is the mixing angle between neutral Higgs scalars, in the notation [15] of 2HDM-II. The κ matrices are diagonal and fixed by fermion masses, κ F = √ 2m F /v with v ≃ 246 GeV, while ρ matrices are in general not diagonal. The off diagonal elements of ρ are tree level FCNH couplings. However, in the exact alignment limit of c β−α = 0, the h 0 boson approaches SM Higgs and couples diagonally, but H 0 and A 0 can still lead to φ 0 → τ µ at tree level.
In this paper, we study the discovery potential for the decays H 0 , A 0 → τ ± µ ∓ , followed by τ decays into an electron and neutrinos or into a τ -jet (π, ρ, or a 1 ) and neutrino. Imposing the current LHC Higgs data, CMS and B physics constraints, we calculate the full tree level matrix elements for both signals and backgrounds. We use realistic acceptance cuts to reduce the backgrounds with current b-tag, τ −tag, and mistag efficiencies. Some promising results are presented for 14 and 27 TeV center of mass (CM) energies for an integrated luminosity L = 300 and 3000 fb −1 , in sync with future High Luminosity (HL) and High Energy (HE) LHC [17][18][19][20].
We discuss experimental limits on relevant parameters from B physics and LHC Higgs data in Sec. II, and give in Sec. III the production cross sections for the Higgs signal and the dominant background with realistic acceptance cuts, as well as our strategy to determine the reconstructed masses for the Higgs bosons. Sec. IV presents the discovery potential at the LHC for √ s = 14 TeV, and also for future hadron colliders with √ s = 27 and 100 TeV.
Optimistic conclusions are drawn in Sec. V.

II. CONSTRAINTS ON RELEVANT PARAMETERS
The most relevant parameters are ρ τ µ , ρ µτ for the decay H 0 /A 0 → τ µ, and ρ tt for the production gg → H 0 /A 0 via the triangle-top loop. A potentially large ρ tc induces [21] H 0 /A 0 → tc, ct, which can dilute the H 0 /A 0 → τ µ branching ratio, while ρ ct is subject to tight constraints by B physics data. LHC data for the 125 GeV h boson [1,2] suggest | cos(β − α)| ≪ 1 in 2HDM-II. We take cos(β − α) = 0.1 for illustration, although larger values are allowed in the general 2HDM [21,22]. As for other ρ matrix elements, we set ρ f f = κ f = √ 2m f /v for diagonal elements except ρ tt , and ignore off-diagonal ones except ρ τ µ , ρ µτ and ρ tc . Degenerate extra scalar masses, i.e. M H 0 = M A 0 = M H ± , is assumed for simplicity. In this section, we consider phenomenological constraints on ρ τ µ , ρ µτ , ρ tt and ρ tc under these assumptions. In general ρ tt is complex and it may contribute to CP violation and Baryogenesis [23]. For simplicity, we will take it to be real in this work.
The FCNH couplings ρ τ µ and ρ µτ induce h 0 → τ µ decay, with branching ratio where M h 0 ≃ 125 GeV, and the τ + µ − and µ + τ − modes are added up. The total width Γ h 0 is estimated by the sum of h 0 → W W * , ZZ * , gg, bb, cc and τ + τ − partial widths obtained by rescaling of SM values [24] with Γ(h 0 → τ µ) added. We impose the 95% C.L. limit B(h 0 → τ µ) < 0.25% by CMS [6]. Constraints on ρ τ µ and ρ µτ by various low-energy processes containing tau and muon are discussed in the literature (see, e.g. Ref. [25][26][27][28]). It is found that τ → µγ is most relevant. Its branching ratio is given by [28] where we take B(τ → µν µ ν τ ) = (17.39 ± 0.04)% [29], and A L,R gives the strength of the τ → µγ amplitude with different chiral structure. In addition to the one-loop contribution mediated by the neutral and charged scalar bosons, we also include the two-loop Barr-Zee type contribution in A L,R , following Ref. [28]. The latter contribution can be obtained by the obvious translation of the expression for µ → eγ [30], and we include the dominant contribution from the effective φ 0 γγ (φ 0 = h 0 , H 0 , A 0 ) vertex, which brings in dependence on ρ tt via the top loop. Current limits on τ → µγ are B(τ → µγ) < 4.5 × 10 −8 by Belle [31] and 4.4 × 10 −8 by BABAR [32], both at 90% C.L. Belle II may improve the limit by a factor of 100 [33]. We conservatively take B(τ → µγ) = 10 −9 to illustrate future sensitivity. ρ tt is also constrained by B physics, in particular by the B q (q = d, s) meson mixings and b → sγ [21]. We update the results of Ref. [21] with the latest experimental and theoretical values as summarized below. We adopt the Summer 2018 result by UTfit [34] for values of CKM parameters and constraints on the B q -B q mixing amplitude (M q 12 ): where C Bq e 2iφ Bq ≡ M q 12 /M q 12 | SM . As for b → sγ, we adopt a recent world average B(B → X s γ) exp = (3.32 ± 0.15) × 10 −4 [35], which includes the recent Belle result [36], and the updated SM prediction B(B → X s γ) SM = (3.36 ± 0.23) × 10 −4 [37,38] for the photon energy E γ > 1.6 GeV. We then use the ratio [39] based on our LO calculation, allowing the 2σ experimental uncertainty of R b→sγ exp with the theoretical uncertainty linearly added. Note that the new experimental and theoretical values result in rather strong limits on M H ± in 2HDM-II [40]: M H ± > 570-800 GeV at 95% CL, depending on the method used to extract the limit. Our method gives M H ± 710 GeV in 2HDM-II at large tan β We ignore effect of ρ tc on B q mixings and b → sγ as it enters via the charm loop, making its impact minor [39] compared with ρ tt and ρ ct entering via the top loop. But ρ tc induces t → ch decay [41,42], and the recent ATLAS limit [43] of directly constrains ρ tc if c β−α is nonzero. The t → ch 0 width is given by and we definẽ ρ tc = |ρ tc | 2 + |ρ ct | 2 / √ 2 as a convenient FCNH coupling [21,44]. Combining with the LO t → bW width to obtain the total top width, we recast the ATLAS limit [43] of Eq. (5) to obtain λ tch = |ρ tc c β−α | = |ρ tc c β−α |/ √ 2 0.064 for ρ ct = 0. In our numerical calculations of this section, we take the latest PDG values [29]  for τ → µγ by BABAR, pink-shaded regions for the B s mixing (C Bs ) and green-hatched regions for b → sγ. The other three observables in Eqs. (4) give weaker limit than C Bs and are not shown in the figures. The dashed contours with B(τ → µγ) = 10 −9 are shown as future Belle II sensitivity. We note that the constraints by h 0 → τ µ and b → sγ are highly sensitive to the choice of parameters: the h 0 → τ µ constraint gets weaker for a smaller c β−α and eventually loses sensitivity if c β−α = 0; the b → sγ constraint is relaxed for a smaller |ρ bb |, and becomes weaker than the B s mixing constraint if ρ bb = 0. In passing, the effect [28] on the muon g − 2 is insignificant (|δa µ | < O(10 −12 ) in the shown parameter regions) due to small ρ τ µ /ρ µτ and c β−α values, which suppress the one-loop h 0 contribution, and the H 0 -A 0 mass degeneracy, which leads to cancellation of the one-loop H 0 and A 0 contributions. Combining experimental limits from LHC Higgs data and B physics, we consider ρ τ µ = ρ µτ < 0.01, and |ρ tc c β−α | = λ tch < 0.064 [43]. To be consistent with B physics constraints, we choose for φ 0 = H 0 or A 0 , which always satisfies the b → sγ constraint for the heavy Higgs scalar mass considered in our study.

III. HIGGS SIGNAL AND PHYSICS BACKGROUND
In this section, we discuss the prospect of discovering FCNH interactions from heavy Higgs bosons H 0 and A 0 decaying into τ ± µ ∓ . There are several parameters that can affect the signal cross section in the 2HDM. We use the experimental results and constraints to optimize the parameter range. Recent data from LHC point toward a Higgs sector in which the light CP even Higgs state is the SM-like Higgs [1,2]. This constraint suggests that c β−α is very small. For case studies in our analysis we set c β−α = 0.1.

A. The Higgs Potential and Decay Final States
For the heavy CP-even H 0 boson, the most important SM decay channels are bb, tt, W W , and ZZ. In addition, tc and h 0 h 0 channels might become dominant in some regions of parameter space. The CP-odd pseudoscalar A 0 boson has significant decays into bb, tt, as well as possible dominant contributions from tc and Zh 0 channels.
To study heavy boson H 0 or A 0 decays involving the light Higgs boson h 0 , let us consider a general CP-conserving Higgs potential [10] Applying minimization conditions, we can express the triple Higgs coupling g Hhh in terms of physical masses and mixing angles [10,45] g which vanishes in the alignment limit, as the self coupling is proportional to c β−α . For simplicity, we take the heavy Higgs states H 0 , A 0 and H ± to be degenerate and we set λ 6,7 = 0. As a sample study, we choose three values of λ 5 = ±1 and 0, to maintain tree level unitarity. For Yukawa couplings, except ρ tt (Eq. 7), we set ρ ii = κ i , which is in good agreement with the current constraints from B Physics and LHC. For off-diagonal elements ρ ij , we perform case studies forρ tc = 0.1 and 0.5, and set all the remaining off-diagonal terms to be 0 except ρ τ µ . Figure 2 shows all major two body decays for the heavy H 0 and A 0 , withρ tc = 0.1 and 0.5. Note that, in keeping ρ τ µ = ρ µτ as in Fig. 1 and fixing the value to 0.01, H 0 , A 0 → τ µ dominates over τ τ , which is interesting by itself. For the H 0 boson,ρ tc , λ 5 and tan β play crucial roles in affecting the H 0 → τ µ branching ratio. We use 2HDMC [46] to scan over 150 GeV ≤ M H ≤ 500 GeV and 1 ≤ tan β ≤ 10 for λ 5 = 0. For M H > 2m t , H 0 → tt, h 0 h 0 and tc channels might become predominant. This suggests M H values close to 1 TeV may not be visible in the τ µ channels, so we limit our case study to M H < 500 GeV.
The pseudoscalar A 0 decays mostly into fermions, as shown in Figs. 2(c) and 2(d). Its decay is independent of tan β and λ 5 in a general 2HDM. Only ρ tc has significant impact on the branching fractions. Forρ tc 0.5, A 0 → tc +tc becomes dominant. Furthermore, for M A > 220 GeV, A 0 → Zh 0 also makes significant contribution. For M A > 2m t , the tt channel starts to dominate, hence we limit our study to M A < 500 GeV to ensure significance.

B. Higgs Signal
Our main signal channel is the production and FCNH decay of a heavy Higgs boson (φ 0 = H 0 , A 0 ) via gluon fusion, pp → φ 0 → τ µ+X [47][48][49][50][51][52][53]. With the τ decaying leptonically, we are looking for a final state of two opposite sign, different flavor leptons and missing energy. With a hadronically decaying τ , a final state with a τ -jet (j τ ), a muon, and missing energy is needed. We have evaluated the FCNH signal cross sections with analytic matrix element and leading order CT14 parton distribution functions [54,55]. To include higher order corrections we calculate K-factors with Higlu [56] for pp → φ 0 + X.

C. Standard Model Backgrounds
The dominant background for leptonic final states comes from pp → τ τ → eµ + E / T + X, pp → W + W − + X and pp → h → τ τ + X. For hadronic channel, we have considered pp → W ± j → µj + E / T + X as the most dominant background along with the τ τ channel. For hadronic channel, tt contribution is highly suppressed, when we veto any event with more than one b jet, with p T > 20 GeV and |η| < 4.7. We have used MADGRAPH [57] and HELAS [58] to generate matrix elements for the backgrounds. To include the NLO corrections, we have employed MCFM [59,60] to evaluate higher order cross sections.

D. Realistic acceptance cuts
To study the discovery potential for the FCNH signal, we apply realistic acceptance cuts proposed by CMS [5,6] at √ s = 13 TeV as shown in Table I. In addition, we apply Gaussian smearing for particle momenta [61, 62] to simulate detector effects based on ATLAS [63] and  CMS [64] specifications.
We note that, as the Higgs boson mass increases, M col (τ µ) cut becomes more effective, and for M H > 180 GeV, pp → h 0 → τ τ + X, pp → h 0 → W + W − + X are almost completely vetoed. For leptonic channel, pp → W + W − +X becomes more dominant than pp → τ τ +X.

IV. DISCOVERY POTENTIAL
To estimate the discovery potential, we require that the lower limit on the signal plus background should be larger than the corresponding upper limit on the background with statistical fluctuations, which leads to [69] where σ S and σ B are the signal and background cross sections, respectively, and L is the integrated luminosity. Choosing N = 2.5, we obtain a 5σ significance. For a large number of background events, it simplifies to the statistical significance where N S and N B are the number of signal and background events.

A. Discovery Reach for Pseudoscalar A 0
The pseudoscalar A 0 has higher production cross section, and with no suppression coming from A 0 → h 0 h 0 , which is forbidden, it is more promising than the heavy scalar H 0 . Fig. 3 shows the discovery region for pp → A 0 → τ µ + X in the (M A ,ρ τ µ ) plane, forρ tc = 0.1 and 0.5, including both the leptonic channel τ → eνν (upper panels) and the hadronic channel τ → j τ ν (lower panels). Because of high QCD backgrounds, performance for hadronic τ decay is worse than leptonic decay, despite its higher branching ratios.
We show our results for √ s = 14, 27 and 100 TeV. At low masses, M A < 180 GeV or roughly the tc threshold, the entire range of ρ τ µ is detectable at 3000 fb −1 , independent of the center-of-mass energy. For an intermediate range (200 GeV < M A < 300 GeV), our discovery region starts shrinking because of A 0 → tc predominance (plus a milder effect from A 0 → Zh 0 turn-on), which is more striking for the largerρ tc = 0.5 value as shown in the right panel plots of Fig. 3. For higher mass range (M A > 300 GeV), we see a slight increase in the 5σ region before and around M A ∼ 2m t , owing to the rise in production cross section for gg → A 0 , before the turn-on of A 0 → tt decay further suppresses our signal towards higher masses beyond M A 360 GeV. Note thatρ tc = 0.5 is actually larger than ρ tt for our mass range (see Eq. (7)), which is constrained by B physics.
B. Discovery Reach for Heavy CP-even Scalar H 0 For the heavy CP-even boson H 0 , the situation is quite different. The branching fraction for H 0 → τ µ is affected by ρ tc , tanβ and λ 5 . The latter Higgs sector parameter affects the H 0 → h 0 h 0 decay, where in Fig. 2 we illustrated with λ 5 = 0. In order to understand the effect of λ 5 , we perform a case study for pp → H 0 → τ µ → eµ + X with M H = 300 GeV, ρ τ µ = 0.01, and scan over −1 ≤ λ 5 ≤ 1 for tanβ = 1. The results are shown in Fig. 4 for √ s = 14, 27 and 100 TeV for the leptonic channel andρ tc = 0.1. The hadronic channel is similar except it will have higher QCD background.
We observe that for a fixed value of tan β, increasing λ 5 from −1 to 0 lowers the cross section of pp → H 0 → τ µ + X while increasing the trilinear Higgs coupling, g Hhh , which enhances the branching fraction of H 0 → h 0 h 0 , with λ 6 = λ 7 = 0. As a case study, let us choose the values of λ 5 = −1, 0, with tan β = 1 to preserve tree-level unitarity and stability for a general 2HDM, which resembles the generic case more closely, and perform a scan for 0.001 ≤ ρ τ µ ≤ 0.01 and 150 GeV ≤ M H ≤ 500 GeV. The results are shown in Fig. 5. There is a large discoverable region in the low mass regime (M H < 180 GeV). However, as we start increasing M H , first H 0 → tc, then H 0 → h 0 h 0 , then H 0 → tt become dominant. The discovery potential is improved somewhat around the M H ∼ 2m t threshold because of rise in gg → H 0 production cross section, appearing as 'dips' of 5σ contours in Fig. 3 and Fig. 5. Beyond that region, we still have some parameter space that can be probed, and a 100 TeV high energy collider can probe to lower couplings. The likelihood of detection increases as we reduce the value of λ 5 , from 0 to −1. The situation forρ tc 0.5 would be worse than A 0 → τ µ, the right panels of Fig. 3.

V. CONCLUSION
The general two Higgs doublet model offers a very rich phenomenology for flavor changing neutral Higgs interactions with fermions, because of the absence of any symmetry to suppress them. Strong experimental constraints exist for these FCNH interactions, but third generation fermions might offer promising signatures for new physics at the LHC and future hadron colliders. Experimental data from LHC Run 1 had shown some hints for the light CP-even Higgs boson h 0 → τ µ, but became insignificant with 2016 CMS data at Run 2. However, in the general 2HDM, the coupling probed is λ hτ µ = ρ τ µ cos(β − α), which is expected to be small in the alignment limit of cos(β − α) → 0, where the light CP-even Higgs boson h 0 approaches the standard Higgs boson.
We have investigated the prospects of discovering H 0 , A 0 → τ µ for the high luminosity (HL) and high energy (HE) LHC and future high energy pp colliders. With gluon fusion being the dominant mode of production for both heavy scalars because of finite ρ tt , we find promising results for LHC with cos(β − α) = 0.1,ρ tc = 0.1, when H 0 , A 0 → tc +tc is not yet overwhelming for M H up to 300 GeV. The choice of h 0 -H 0 mixing parameter cos(β − α) and ρ tc = (|ρ tc | 2 + |ρ ct | 2 )/2 values are meant as illustrative. Having taken degenerate H 0 , A 0 and H + , M H 0 < 300 GeV can still evade b → sγ constraint and should be taken seriously. It should be noted that A 0 is more promising than H 0 because of its higher production cross section and fewer decay channels affecting its decay to τ µ, but H 0 decay depends also on Higgs potential due to h 0 h 0 mode. Ifρ tc is considerably larger than 0.1, H 0 , A 0 decay to tc would suppress τ µ observability, and a higher energy collider would be needed.
Our study has focused on discovering H 0 , A 0 in τ µ final state, but companion final states such as Zh 0 (for A 0 ), h 0 h 0 (for H 0 ) and tc, tt are worthy pursuits in their own right, some of which have been studied elsewhere.