Entangling Higgs production associated with a single top and a top-quark pair in the presence of anomalous top-Yukawa coupling

The ATLAS and CMS collaborations observed a mild excess in the associated Higgs production with a top-quark pair tt¯h\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \left(t\overline{t}h\right) $$\end{document} and reported the signal strengths of μtthATLAS = 1.81 ± 0.80 and μtthCMS = 2.75 ± 0.99 based on the data collected at s=7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \sqrt{s}=7 $$\end{document} and 8 TeV. Although, at the current stage, there is no obvious indication whether the excess is real or due to statistical fluctuations, here we perform a case study of this mild excess by exploiting the strong entanglement between the associated Higgs production with a single top quark (thX) and tt¯h\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ t\overline{t}h $$\end{document} production in the presence of anomalous top-Yukawa coupling. As well known, tt¯h\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ t\overline{t}h $$\end{document} production only depends on the absolute value of the top-Yukawa coupling. Meanwhile, in thX production, this degeneracy is lifted through the strong interference between the two main contributions which are proportional to the top-Yukawa and the gauge-Higgs couplings, respectively. Especially, when the relative sign of the top-Yukawa coupling with respect to the gauge-Higgs coupling is reversed, the thX cross section can be enhanced by more than one order of magnitude. We perform a detailed study of the influence of thX production on tt¯h\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ t\overline{t}h $$\end{document} production in the presence of the anomalous top-Yukawa coupling and point out that it is crucial to include thX production in the analysis of the tt¯h\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ t\overline{t}h $$\end{document} data to pin down the sign and the size of the top-Yukawa coupling in future. While assuming the Standard Model (SM) value for the gauge-Higgs coupling, we vary the top-Yukawa coupling within the range allowed by the current LHC Higgs data. We consider the Higgs decay modes into multileptons, bb¯\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ b\overline{b} $$\end{document} and γγ putting a particular emphasis on the same sign dilepton events. We also discuss the prospects for the LHC Run-2 on how to disentangle thX production from tt¯h\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ t\overline{t}h $$\end{document} one and how to probe the anomalous top-Yukawa coupling.


JHEP04(2017)138
the Higgs decay modes into multileptons, bb and γγ putting a particular emphasis on the same sign dilepton events. We also discuss the prospects for the LHC Run-2 on how to disentangle thX production from tth one and how to probe the anomalous top-Yukawa coupling.

Introduction
The Higgs boson was discovered at the Large Hadron Collider (LHC) [1,2]. After analyzing almost all the Run-1 data, the measured properties of the Higgs boson are the best described by the standard model (SM) Higgs boson [3,4], which was proposed in 1960s [5][6][7]. The most constrained is the Higgs coupling to the massive gauge bosons normalized to the corresponding SM value (gauge-Higgs couplings) C v = 0.94 +0. 11 −0.12 , which is very close to the SM value [8,9]. On the other hand, the top-and bottom-Yukawa couplings cannot be determined as precisely as C v by the current data. Currently, they are within 30-40% of the SM values [8,9], yet, the negative regime of the top-Yukawa coupling is still allowed at 95% confidence level (CL). 1 On the other hand, one of the most exciting results from both ATLAS and CMS in their Run-1 data was the excess in the same-sign dilepton events with b-jets and missing transverse energy [10][11][12][13][14][15][16]. The ATLAS collaboration reported a significance of about 2σ in the exotic search [10] and the CMS collaboration a significance of about 2.5σ in the tth Higgs search [11]. Some people have taken them as the twilight of new physics beyond the SM (BSM) [17][18][19].
In this work, we focus on the excess observed in Higgs boson production in association with a top-quark pair (tth). In the same sign dilepton channel (ss2 ), the best-fit signal strengths are: µ ATLAS tth ,ss2 = 2.8 +2.1 −1.9 [20] and µ CMS tth ,ss2 = 5.3 +2.1 −1.8 [11]. The CMS excess is about 2.5σ above the SM prediction while the ATLAS result is still consistent with the SM. While, the best-fit signal strengths for combined channels are: µ ATLAS tth = 1.81 ± 0.80 and µ CMS tth = 2.75 ± 0.99 at √ s = 7 and 8 TeV [8,9]. Even though the data do not show a significant deviation from the SM predictions and there is no obvious indication yet whether the excess is real, there are still enough rooms for the implication of new physics beyond the SM. Here we attempt to interpret the mild excess by exploiting the strong entanglement between the associated Higgs production with a single top quark (thX) and tth production in the presence of anomalous top-Yukawa coupling.
As well known tth production only depends on the absolute value of the top-Yukawa coupling at the leading order (LO), see figure 1, which is similar to gluon-gluon fusion production. Therefore, the tth cross section is insensitive to the sign of the top-Yukawa coupling at LO. Meanwhile, in thX production, this degeneracy is lifted through the strong interference between the two main contributions, which are proportional to the top-Yukawa and the gauge-Higgs couplings, respectively. Note that we include both th + X andth + X production when we refer to thX with X denoting the accompanied particle(s) produced together with t(t) and h. The Feynman diagrams contributing to thX production with X = j (qb → thq ) is depicted in figure 2. The left diagram is proportional to the gauge-Higgs coupling while the right one to the top-Yukawa coupling. 2 The interference between the two diagrams was shown to be significant and induces large variations in the total cross section with the size and the relative sign of the Higgs couplings to the gauge bosons and the top quarks. It was shown in literature  that the cross section can be enhanced by more than an order of magnitude when the relative sign of the top-Yukawa coupling to the gauge-Higgs coupling is reversed.
In this work, we perform a detailed study of the influence of thX production on tth production in the presence of the anomalous top-Yukawa coupling. While assuming the Standard Model (SM) value for the gauge-Higgs coupling, we vary the top-Yukawa coupling within the allowed range by the current LHC Higgs data. We consider the Higgs decay modes into multileptons, 3 bb and γγ putting a particular emphasis on the same sign dilepton events. We show that the current ATLAS and CMS analyses of tth could be significantly contaminated by the thX processes. Moreover, the thX processes contribute (or contaminate) at quite different levels in various detection modes of tth, depending on the value of top-Yukawa coupling, on the cuts used in each experiment, and on the decay mode of the Higgs boson. We shall illustrate such behavior in section III, which is far more complicated than simply assuming a small constant level of contamination in all channels. In addition to explaining the apparent mild excess in tth production by entangling thX production, we also propose how to disentangle thX production from tth one at the LHC Run-2. The main objective of this work is to further pin down the sign and the size of the top-Yukawa coupling. To achieve the objective, we point out that it is crucial to consider the entanglement between thX and tth.
Note that the ∼ 2σ excesses were seen in the channels of multileptons and bb of ATLAS and in the channels of multileptons and γγ of CMS, but not in the others. It may as well be due to statistical fluctuations, but could also be due to some specific forms of new physics. Only more data can tell. In this work, we perform a case study in which, through the thX processes, the contributions of the anomalous top-Yukawa coupling to tth production manifest non-trivially depending on the value of top-Yukawa coupling, on the cuts used in each experiment, and on the decay mode of the Higgs. Our case study shows that the (future) observations related to tth production should be carefully made without simply assuming a small constant level of contamination in all channels which is common to both the ATLAS and CMS experiments.  Figure 3. Feynman diagrams contributing to thX production with X = jb.  The organization is as follows. In the next section, we lay down the formalism and the calculation method. In section 3, we show the influence of thX with the anomalous top-Yukawa coupling on tth for both the ATLAS and CMS Run-1 data. In section 4, we propose some scenarios to further disentangle thX from tth for the LHC Run-2. Finally, we discuss and conclude in section 5.

Processes and Higgs couplings involved
We consider two types of production processes for the Higgs boson and the top quark. The first one is the associated production of the Higgs with a pair of top quarks, see figure 1. The second one is the associated Higgs production with a single top quark plus anything else: thX production with X = j (qb → thq ), jb (qg → thq b), W (gb → thW ), 4 see figures 2-4 in which we have marked the vertices of hW W and htt with squares. In tth production, the production cross section only depends on the square of the top-Yukawa coupling. However, in thX production, the cross sections depend on the size of the gauge-Higgs and top-Yukawa couplings and the relative sign between them.

JHEP04(2017)138
In fact, the process qg → thjb is a part of the NLO QCD corrections to qb → thj when the momentum of the final b quark in thjb is integrated out. In our work, using Mad-Graph5@NLO, we calculate the cross section for the qg → thjb process at NLO adopting the four-flavor scheme. And then, we define thj and thjb productions by introducing a set of separation cuts: p j T > 10 GeV, |η j | < 5, p b T > 30 GeV, |η b | < 2.5. Naturally, the low (high) p b T region is taken for thj (thjb) production. We obtain σ(thj) = 11.2 (43.0) fb and σ(thjb) = 5.77 (23.6) fb at the LHC with √ s = 8 (13) TeV. We note that the sum σ(thj) + σ(thjb) = 17.0 (66.6) fb agrees well with the NLO cross sections found in the literature. As will be shown, the contributions of thj and thjb to the accumulated signal strengths strongly depend on the Higgs decay channels and experiment cuts chosen. In this way, we properly reflect the different kinematic signatures of thj and thjb which could be lost if we do not introduce the separation. On the other hand, since the NLO QCD corrections for both pp → tth and pp → thW are relatively large compared with pp → thj, we multiply the corresponding K factors to the LO cross sections for each of them.
Without loss of generality, one can write the gauge-Higgs and Yukawa couplings of the Higgs boson h as 5 Here only the gauge-Higgs coupling g hW W and the top-Yukawa couplings are relevant to the tth and thX production processes shown in figures 1-4. We note g hW W = g hZZ = g S hf f = 1 in the SM.
In order to calculate the event rates we have to consider the decay branching ratios of the Higgs boson, which depend on g hW W , g hZZ , g S htt,hbb and a few more couplings, including hτ τ , hcc, hγγ, and hgg. The amplitude for the decay process h → γγ can be written as where k 1,2 are the momenta of the two photons and 1,2 the wave vectors of the corresponding photons with µ Retaining only the dominant loop contributions from the third-generation fermions and W ± , and including some additional loop contributions from new particles, the scalar form factor is given by where τ x = m 2 h /4m 2 x , N C = 3 for quarks and N C = 1 for tau leptons, respectively. For the loop functions of F sf,1 (τ ), we refer to, for example, ref. [45]. The additional contributions JHEP04(2017)138 ∆S γ are due to additional particles running in the loop. In the SM, g S hf f = g hW W = 1 and ∆S γ = 0. Similarly, the amplitude for the decay process h → gg can be written as where a and b (a, b = 1 to 8) are indices of the eight SU(3) generators in the adjoint representation. Including some additional loop contributions from new particles, the scalar form factor is given by In the SM, g S hf f = 1 and ∆S g = 0. In the decays of the Higgs boson, we can see that the partial width into bb depends on g hbb , that into W W * and ZZ * depends on g hW W,hZZ , and that into γγ and gg depends implicitly on all g hW W , g S htt , g S hbb , and g S hτ τ . The dependence of the production cross sections and the decay branching ratios on g hW W and g S hf f has been explicitly shown in the above equations. Since we are primarily interested in size of the gauge-Higgs and top-Yukawa couplings and the relative sign between them, for bookkeeping purpose, we use the following simplified notations We shall show the anomalous top-Yukawa coupling effects on tth and thX production at the LHC in the next section.

Signal strengths
First we note that signal strengths depend on the decay modes of the top quark and the Higgs boson, as well as their production mechanisms. For a choice of experimentallydefined decay mode D, and taking into account the thX production processes, we define the signal strength µ(tth) with respect to the SM tth production as follows where σ(tth) = σ(pp → tth) and σ(thX) = σ(pp → thX) + σ(pp →thX) are understood. The detection efficiencies η's depend on the experimental apparatuses and cuts for the specific production and decay mode. By introducing the cross-section ratios and the D-dependent detection-efficiency ratios Table 1. The best-fit values for the category-dependent signal strengths µ CMS tth and µ ATLAS tth coming from the CMS [11] and ATLAS [20,46,47] searches, respectively, for the associated production of the Higgs boson with a top quark pair at √ s= 7 and 8 TeV for m h = 125.6 GeV (CMS) / 125 GeV (ATLAS). one may have (2.11) We note that 1 = R(tth) = 1 in the SM limit of C v = 1 and C S t = +1 and µ(tth) is always larger than 1 due to the entanglement of thX production. Our main task is to calculate the cross section ratios R's in the presence of anomalous top-Yukawa coupling and the detection-efficiency ratios 1,2,3,4 for various top-quark and Higgs-boson decay modes.

thX production with the anomalous top-Yukawa coupling
Both the CMS [11] and ATLAS [20,46,47] collaborations have published the results of their searches for the associated production of the Higgs boson with a top-quark pair via different Higgs decay channels at √ s= 7 and 8 TeV. We summarize their best-fit results in table 1. Since the experimental uncertainties in the hadronically-decaying τ and 4 categories are too large at this stage, we shall focus only on the ss2 , 3 , γγ and bb categories in our analysis below. In the γγ category for h → γγ, both CMS [11] and ATLAS [46] included all the decay modes of a top-quark pair: semileptonic (tt → lνjjbb), leptonic (tt → lνlνbb), and hadronic (tt → jjjjbb) modes. On the other hand, in the bb category for h → bb, both CMS [11] and ATLAS [47] considered only the semileptonic and leptonic decay modes of the top-quark pair. Finally, in the categories of ss2 and 3 for h → multileptons, both CMS [11] and ATLAS [20] included only the semileptonic decay mode of the top-quark pair.
In order to perform a detailed study of the influence of thX production with anomalous top-Yukawa coupling on tth production, we simulate both the thX and tth processes and generate events by MadGraph5 [48], perform parton showering and hadronization by Pythia 8.1 [49], and employ the detector simulations by Delphes 3 [50]. We use NN23LO1 for parton distribution functions with different renormalization/factorization scales which we shall show below. We follow the selection cuts and detector efficiencies of the CMS [11] JHEP04(2017)138 Table 2. The signature of the search channels used in the tth analysis for CMS.
Same-Sign Dilepton (sub)leading lepton: 2 e/µ, PT > 25 (20) GeV Table 3. The signature of the search channels used in the tth analysis for ATLAS. and ATLAS [20,46,47] tth searches. We summarize the signatures of the search channels used in the tth analysis for CMS in table 2 and for ATLAS in table 3.
We calculate the tth production cross section with the factorization (µ F ) and renormalization (µ R ) scales set at m t + m h /2 in the four-flavor scheme. On the other hand, in computing the production cross sections for thX, we include the t-channel thj and thjb processes and the thW process, but ignore the s-channel thb process due to its much smaller JHEP04(2017)138 cross section. In calculating the production cross sections for thj and thjb, µ F = µ R are set at 75 GeV in the four-flavor scheme. For thW , we are employing the dynamic factorization and renormalization scales in the five-flavor scheme.
As shown in refs. [3,4] in which the model-independent fit to the current Higgs data is performed, the negative C S t = −1 is ruled at 95%CL if only the gauge-Higgs coupling C v and the top-Yukawa coupling C S t vary. However, C S t = −1 is still allowed at 95%CL when the gauge-Higgs C v , top-Yukawa C S t , bottom-Yukawa C S b , and tau-Yukawa C S τ couplings are all allowed to vary. Furthermore, if some sizable contributions to ∆S γ and ∆S g due to additional new particles running in the loop are assumed, a broad range of C S t between −2 and +2 is still consistent with the current Higgs data.
In the following, we show the results of our numerical analysis in each of categories of Leptons (ss2 and 3 ), γγ, and bb for the Higgs boson decaying into multileptons, two photons, and two b quarks, respectively. Note that, in our numerical analysis, we vary the top-Yukawa coupling C S t within the range allowed by the current LHC Higgs data while taking the SM value for the gauge-Higgs coupling, C v = 1. For the bottom-Yukawa C S b and tau-Yukawa C S τ couplings, one may freely take either +1 or −1 since their signs would have negligible effects on the production cross sections and decay branching ratios.

Category Leptons for h → multileptons
In the category Leptons which includes leptonic decays of h → W W, ZZ, τ τ → multileptons, we focus on the subcategories of ss2 and 3 modes. We shall use several different values of C S t to show the possibly strong entanglement between thX production and tth production for both the ATLAS and CMS analyses. Note that CMS used the so-called Multivariate Analysis (MVA) method in their analysis, however, we only follow their set of preselection cuts and event selection requirements to perform the cut-based analysis.
First, we note that the CMS and ATLAS collaborations were adopting different signatures and preselection cuts to analyze the category Leptons as shown in table 2 and  table 3. 6 The CMS analysis was performed in the ss2 and 3 subcategories while the ATLAS analysis was carried out in the subcategories of 2 + 4j, 2 + ≥ 5j, and 3 . Without knowing an appropriate way to combine the two sets of data, we present our results handling the CMS and ATLAS cases separately to make full use of the existing data. Further, in the CMS and ATLAS analyses of the 3 subcategory, also required was a low-mass invariant-mass cut M > 12 GeV to remove the J/Ψ background and a Z-pole mass veto cut |M + − − M Z | > 10 GeV to suppress the Z background. Some additional cuts on the scalar sum of the transverse momenta (P T ) of the two leptons and the missing energy (E miss T ) were also applied in the CMS case.
To quantify the effects of different values of C S t on tth and thX, we use the signalstrength formula for µ(tth) in eq. (2.8), which consists of the sum of the products of the cross section ratios R's and the D-dependent detection efficiency ratio 's, which are in JHEP04(2017)138  Table 4. The cross-section ratios R(tth) and R(thX) with X = j, jb, W defined in eq. (2.9). We are taking √ s = 8 TeV (LHC-8) and C S t = ±1 , ±1.5.
turns given by eq. (2.9) and eq. (2.10), respectively. Explicitly, we have In table 4, we show the cross section ratios R(tth) and R(thX) with X = j, jb, W at the 8 TeV LHC (LHC-8) taking C S t = ±1 and ±1.5. Note R(tth) = 1 (2.25) for |C S t | = 1 (1.5) and the thX cross sections can be largely enhanced for the negative values of C S t . In table 5, we show the D-dependent detection efficiency ratios 1,2,3,4 with the CMS cuts in the ss2 (upper) and 3 (lower) subcategories for C S t = ±1 , ±1.5. By using the cross section ratios given in table 4, one can obtain the CMS tth signal strengths µ CMS tth . We observe that µ CMS tth ,ss2 ∼ 2 (3) for C S t = 1.5 (−1.5) and the signal strengths are larger for the negative values of C S t . One may make similar observations for µ CMS tth ,3 . Recently, the CMS collaboration has also reported a possible excess in the decay process h → τ ∓ µ ± , B(h → τ ∓ µ ± ) = 0.84 +0. 39 −0.37 %, with a significance of 2.4σ in the search for the lepton-flavor violation (LFV) [51]. If we take into account this LFV decay of the Higgs boson, we can slightly enhance the production rate of h → multileptons mode by a few percents. We estimate the h → τ ∓ µ ± contribution by rescaling h → τ + τ − channel with the branching ratios and the τ detection efficiency. The CMS tth signal strengths µ CMS tth after taking account of h → τ ∓ µ ± are also presented in table 5.
Similarly, we calculate the D-dependent detection efficiency ratios 1,2,3,4 with the ATLAS cuts in the 2 + 4j (upper), 2 + ≥ 5j (middle), and 3 (lower) subcategories for several values of C S t and present them in

LHC-8
With CMS Analysis Cuts

Category γγ for h → γγ
In the category γγ for h → γγ, we include all the decay modes of the top-quark pair. We consider two subcategories of leptonic selection and hadronic selection. The lepton-selection subcategory is for the semileptonically and leptonically decaying top-quark pair while the hadronic-selection one for the hadronically decaying top-quark pair. To single out the effect of anomalous top-Yukawa coupling on thj and tth production in this category, we assume a non-vanishing ∆S γ due to additional particles running in the h-γ-γ loop, see eq. (2.4). In fact, one may have B(h → γγ) = (2.3, 5.4, 1.53, 5.68) × 10 −3 for C S t = (1, −1, 1.5, −1.5) using, for example, HDECAY [52]. We are using B(h → γγ) = 2.3 × 10 −3 independently of C S t assuming a non-zero ∆S γ which cancels out the the effect of anomalous top-Yukawa coupling on B(h → γγ). This assumption also helps to avoid the constraint on S γ (m h ) from the current LHC Higgs data [3,4].
To repeat the CMS analysis, we follow their selection cuts listed in table 2, which are used in the cut-based analysis [11]. Also, we further impose the Higgs-mass window cut: 100 GeV ≤ m γγ ≤ 180 GeV.
For the ATLAS analysis, we follow ref. [46] with preselection cuts listed in table 3. We further impose the Higgs-mass window cut (105 GeV < m γγ < 160 GeV) and the

LHC-8
With ATLAS Analysis Cuts > 20 GeV and the eγ invariant-mass cut M eγ > 94 GeV or < 84 GeV are also applied in the leptonic-selection category. In the hadronic-selection subcategory, we adopt the selection 1 in ref. [46] using the working point with efficiency of 70% for identifying b-jets.
Before we present the results of our numerical study of the effects of thX on tth in the category γγ, we would like to make some remarks on a few noticeable aspects from the ATLAS tth search. It has been shown that there was no significant excess over the background in the h → γγ mode, and thus the 95% CL upper limit is set at 6.7 × σ SM (tth). Especially, ATLAS took into account the dependence of the tth and thX cross sections as well as the branching ratio B(h → γγ) on the top-Yukawa coupling. The ATLAS tth search sets the lower and upper limits on C S t : −1.3 ≤ C S t ≤ 8.0 at 95% CL. In table 7, we show the D-dependent detection efficiency ratios 1,2,3,4 with the CMS cuts in the hadronic-selection (upper) and leptonic-selection (lower) subcategories for C S t = ±1 , ±1.5. By using the cross section ratios given in table 4, one can obtain the CMS tth signal strengths µ CMS tth ,γγ(lep) and µ CMS tth ,γγ(had) using eq. (3.1). In the leptonic-selection subcategory, we observe that µ CMS tth ,γγ(lep) > 2 for |C S t | = 1.5. In the hadronic-selection subcategory, we obtain the larger values for negative C S t : µ CMS tth ,γγ(had) ∼ 1, 2.5; 2.5, 5 for C S t = (+1, −1; +1.5 − 1.5). Also presented is the combined signal strength µ CMS   Table 7. Category γγ: the D-dependent detection efficiency ratios i 's defined in eq. (2.10) with the CMS cuts for the category γγ taking C S t = ±1 , ±1.5 and √ s = 8 TeV. The resulting signal strengths µ CMS tth ,γγ(lep) and µ CMS tth ,γγ(had) are also shown. Note that the CMS cuts N (j) ≥ 4, N (b) ≥ 1 for γγ hadronic channel are not strong enough to separate tth from thX processes.

LHC-8
With ATLAS Analysis Cuts

Category bb for h → bb
In the category bb for h → bb, we consider the semileptonic and leptonic decay modes of the top-quark pair which leads to the two subcategories of single lepton (1 ) and dilepton (2 ). The CMS preselection cuts shown in table 2 and the ATLAS ones in table 3 are first applied. And we further impose P T j > 40 GeV for the leading 3 jets in the single-lepton subcategory. In the dilepton subcategory, we select the events with exactly two oppositely charged leptons e + e − , e ± µ ∓ , µ + µ − with P T l 1 > 25 GeV and P T l 2 > 15 GeV. For e ± µ ∓ events, we further require H T , scalar sum of transverse momenta of leptons and jets, to be larger than 130 GeV. For e + e − and µ + µ − events, we impose two more conditions: (i) more than 2 b-jets and M ll > 15 GeV to reduce the J/Ψ background and (ii) exactly 2 b-jets, M ll > 60 GeV to remove the events in the low-mass region with large error bars, and |M ll − M Z | > 8 GeV to veto the Z background. We then combine these selections to complete the dilepton selection.
In table 9, we show the D-dependent detection efficiency ratios 1,2,3,4 with the CMS cuts in the single-lepton (upper) and dilepton (lower) subcategories for C S t = ±1 , ±1.5. By using the cross section ratios given in table 4, one can obtain the CMS tth signal strengths µ CMS tth ,bb(1 ) and µ CMS tth ,bb(2 ) using eq. (3.1). We observe that µ CMS tth ,bb(1 ) > 2 for C S t = −1 , ±1.5 and µ CMS tth ,bb(2 ) > 2 for C S t = ±1.5. The combined signal strength µ CMS tth ,bb Similarly, in table 10, we show the D-dependent detection efficiency ratios 1,2,3,4 with the ATLAS cuts and the signal strengths µ ATLAS tth ,bb(1 ) , µ ATLAS tth ,bb (2 ) , and µ ATLAS tth ,bb . Similar observations can be made as in the CMS case.   Table 9. Category bb: the D-dependent detection efficiency ratios i 's defined in eq. (2.10) with the CMS cuts for the category bb taking C S t = ±1 , ±1.5 and √ s = 8 TeV. The resulting signal strengths µ CMS tth ,bb(1 ) and µ CMS tth ,bb (2 ) and the combined one µ CMS tth ,bb are also shown.

LHC-8
With ATLAS Analysis Cuts

Disentangling thX from tth
In this section, we show kinematic distributions for the tth and for thX processes in the presence of anomalous top-Yukawa coupling in an attempt to disentangle thX production from tth one using specific selection cuts. We focus on the h → γγ channel at the LHC with √ s = 13 TeV (LHC-13) adopting the Delphes ATLAS fast detector simulation. We closely follow the analysis in a previous work [21]. Here we use the thj process for illustration while the other thX processes have similar features.

LHC-13
In table 11, we show the cross sections ratios R(tth) and R(thX) with X = j, jb, W at the 13 TeV LHC taking C S t = ±1 and ±1.5. Comparing the ratios at √ s = 8 TeV presented in table 4, we observe the LHC-13 ratios are more or less similar to the LHC-8 ones.
We show the p Tγ and η j distributions for the tth and thX processes in figure 8 and figure 9, respectively, taking C S t = ±1 , ±1.5. With C S t = 1, the p Tγ distribution of the thX process, especially, that of the thW process becomes harder relative to the tth distribution. In the η j distributions, the thj and thjb processes have more forward pseudorapidity. We therefore come up with a set of selection cuts summarized in table 12, in which we order the jets according to their energy since most of the time the forward jet is the most energetic one. It is in general correctly chosen as shown in the η j distribution. Note that we require to tag one forward jet and apply the Higgs-mass window cut on the diphoton invariant mass M γγ .
The accumulated thj signal strength µ(thj) 7 is shown in the left panel of figure 10 with C S t = ±1 , ±1.5. To obtain the signal strength µ(thj) in the h → γγ decay at LHC-13, we impose the thj-specific cuts listed in table 12. In the right panel of figure 10, we show the accumulated tth signal strength µ(tth) obtained by using the tth-specific cuts in the same table. We observe that µ(thj) (left) is dominated by thj (green) for the negative 7 Similarly as µ(tth) given by eq.   values of C S t , implying that our thj-specific cuts are working very efficiently when the thj production cross section is much enhanced with R(thj) > ∼ 1. On the other hand, µ(tth) (right) is dominated by tth (blue) independently of C S t and we observe that our tth-specific cuts are working reasonably well as in the LHC-8 case (the left panel of figure 6). We can further draw a few observations from figure 10 as follows.
1. When the experiment is targeting at tth production using the tth-specific cuts, there are contaminations from the thX processes. For positive C S t , the thX contaminations JHEP04(2017)138 Figure 10. Accumulated signal strengths µ(thj) (left) and µ(tth) (right) at LHC-13 obtained by stacking the various thX contributions on the tth one for C S t = +1 , −1 , +1.5 , −1.5 from left to right. We use the Delphes ATLAS template for detector simulations. are small. But, for negative C S t , they can be as large as the tth signals. For C S t = −1, for example, µ(tth) ∼ 2 and only half of which comes from tth.

2.
From the left panel, we can see that the thX processes dominate the signal strength µ(thj) for negative C S t , which means that the thj-specific selection cuts we employed indeed can single out the thj process from the tth one.
3. The large values of µ(thj) ∼ O(10) when C S t deviates from its SM value 1 imply that the direct thj searches are also important as complementary channels. Current LHC constraints on the thj searches at √ s = 8 TeV in are still weak [60][61][62][63][64][65][66], so that more data are needed at √ s = 13 TeV in the future to probe the anomalous top-Yukawa coupling through this channel.

Conclusions
Usually, the associated Higgs production with a single top quark dubbed as thX with X = j, jb, W makes only small contributions to the overall experimental signal strength of µ(tth). In this work, however, we have demonstrated explicitly that the thX processes can significantly increase the experimentally measured signal strength µ(tth) when the relative sign of the top-Yukawa coupling to the gauge-Higgs coupling is reversed. Furthermore, we have shown explicitly that the thX processes contaminate at quite different levels in various detection modes of tth, depending on the value of top-Yukawa coupling, on the cuts used in each experiment, and on the decay mode of the Higgs boson. Such behavior is far more complicated than simply assuming a small constant level of contamination in all channels. The signal strengths can be as large as 2 − 4 in the category Leptons for h → multileptons, 2 − 4.5 in the category γγ for h → γγ, and 2 − 4 in the category bb for h → bb. Assuming the mild excesses observed in tth production at the LHC are real, we note that all go in the right direction to match them.

JHEP04(2017)138
When more data are collected at √ s = 13 TeV, we can choose more specific cuts to single out the thX processes, which can effectively determine the size and the sign of the top-Yukawa coupling.
We offer the following comments on our findings.
1. The current data on tth production showed mild excesses at some level. 8 Although they may be simply due to statistical fluctuations, in this work, we have taken the liberty of interpreting the mild excesses by exploiting the strong entanglement between thX and tth. Our case study would be very useful if the future data support the excesses.
2. When the top-Yukawa coupling is kept at the SM value, i.e. C S t = 1, the contamination from all the thX processes is small, only about 5-15%, and that can be regarded as a sort of small higher-order corrections.
3. However, when the sign of the top-Yukawa coupling is reversed, i.e. C S t = −1, the thX contributions are significantly enhanced. And the resulting signal strengths can be as large as 1.4-2.0 (category Leptons), 1.0-2.5 (category γγ), and 1.4-2.0 (category bb), explaining the experimental excesses shown in table 1.

5.
In the approach adopted in this work, the dominant thX processes are thj and thjb both of which contain a very forward energetic jet. Also, as shown in figure 8, the thW process has a harder p T photon. Therefore, we successfully come up with a set of selection cuts to single out the thX processes from the tth process. It has been shown clearly in the left panel of figure 10.
6. One very useful observation in our work is that the contributions from various production processes of tth, thj, thjb, and thW to the accumulated signal strengths strongly depend not only on the Higgs decay channels but also on the experiment (ATLAS or CMS), as can be seen in figures 5-7.