Resolving the degeneracy in top quark Yukawa coupling with Higgs pair production

The top quark Yukawa coupling ($y_t$) can be modified by two dimension-six operators $\mathcal{O}_H$ and $\mathcal{O}_y$ with the corresponding Wilson coefficients $c_H$ and $c_y$, whose individual contribution cannot be distinguished by measuring $y_t$ alone. However, such a degeneracy can be resolved with Higgs boson pair production. In this work we explore the potential of resolving the degeneracy of the unknown Wilson coefficients $c_H$ and $c_y$ at the 14~TeV LHC and the 100~TeV hadron colliders. Combining the information of the single Higgs production, $t\bar{t}h$ associated production and Higgs pair production, the individual contribution of $c_H$ and $c_y$ to $y_t$ can be separated. Regardless of the value of $c_H$, the Higgs pair production can give a strong constraint on $c_y$ at the 100 TeV hadron collider. We further show that it is possible to differentiate various $c_y$ and $c_t$ values predicted in several benchmark models.

Introduction. Top quark Yukawa coupling (y t ) is the only coupling with the magnitude of order one in the Standard Model (SM). As the largest Yukawa coupling, it is important for vacuum stability and cosmology [1,2]. Besides, in many new physics (NP) scenarios [3][4][5][6][7], top quark plays an important role in triggering the electroweak symmetry breaking (EWSB) and is directly connected to new physics beyond the SM. Therefore, it is highly motivated to understand the top quark Yukawa sector better, both theoretically and experimentally. The parameter y t can be measured directly by the tth associated production. Recently, this process is confirmed by both the ATLAS and CMS collaborations with signal strengths µ tth = 1.32 +0. 28 −0.26 and 1.26 +0.31 −0.26 [8,9], respectively, at the Large Hadron Collider (LHC) with √ s = 13 TeV. Besides, y t can also be measured in loopinduced single Higgs boson production [10,11], t(t)hj associated production [12] and multi-top production processes [13,14]. With higher luminosity being accumulated, one expects the accuracy on y t can be further improved. It is thus timely to study what kind of NP can modify y t .
In general, we can parameterize NP effects on y t by several higher dimensional operators in a model independent way. Out of the complete set of dimension-6 operators listed in Ref. [15], we consider in this work the two operators which can modify y t at tree level: in the so called Strongly-Interacting Light Higgs (SILH) basis [16,17], where the dimension-six operators Here, c H and c y are the corresponding Wilson coefficients with c y being assumed to be real, Q L is the left-handed third-family quark doublet, v = 246 GeV is the vacuum expectation value, y SM t = m t /v is the SM top Yukawa coupling, and m t is the top quark mass. As O H modifies the Higgs boson wave function, it can universally shift all the single Higgs couplings, hence, affects y t .
Theoretically, the operators O H and O y can be induced by several different NP scenarios. For example, scalar singlets interacting with the Higgs doublet can induce the universal O H operator [18][19][20][21][22], while additional vector-like fermions can induce the operator O y via mixing with the top quark [23,24]. As both the operators in Eq. (2) can induce deviations in y t , one cannot differentiate their individual contributions if we only measure the top Yukawa coupling. Even if y t is measured to be consistent with the SM prediction, one cannot exclude the possibility of having cancellation among different NP operators. Or, if the deviation in y t is established, we still need to separate the effects of O H and O y for better understanding the origin of NP. Since the effect of the O H operator is to simply rescale any amplitude involving a single Higgs boson by a factor 1/ √ 1 + c H after renormalizing the Higgs boson field, its effect is universal and can be measured from studying the hV V (V = W ± , Z) couplings. However as shown in Ref. [25], in case that Higgs boson is a pseudo Nambu-Goldstone boson, other operator (at the order of O(p 2 )) can mimic the effect induced by O H on hV V couplings. Hence, it requires novel method to separately measure the coefficients of those two types of operators [25]. Likewise, in this work, we explore the possibility of separately measuring the coefficients of O H and O y , both contributing to top Yukawa coupling y t , through Higgs boson pair production gg → hh. In addition to modifying the single Higgs effective coupling of htt, both O H and O y can also contribute to the effective coupling hhtt, but with different combinations, which can be measured via Higgs boson pair production. Namely, studying Higgs boson pair production can be utilized not only for measuring Higgs self interactions, but also for discriminating new physics scenarios in the top sector. Only after the individual contribution of each effective operator is extracted can we further solidify the SM or otherwise establish NP models. Higgs boson pair production. The gluon-initiated Higgs pair production can be used to measure the trilinear Higgs self-coupling . It is also sensitive to various NP models [20,21,[48][49][50][51][52][53][54][55]. After the EWSB, the effective Lagrangian related to the non-resonant Higgs pair production is [56][57][58][59][60][61] where a is the color index, α s = g 2 s /4π with g s being the strong coupling strength, and m h is the Higgs boson mass. The SM, at tree level, corresponds to c 3 = c t = 1 and c 2t = c g = 0. Then the squared amplitude of gg → hh, after averaging over the gluon polarizations and colors, is [61] h , m 2 t ) are the form factors [62] withŝ andt being the canonical Mandelstam variables. The first term inside the bracket contributes to s-wave and the G term to d-wave component whose contribution to total cross section is numerically negligible [63].
To avoid any momentum dependent contributions to the Higgs self couplings induced by O H , we adopt the generalized canonical normalization of the Higgs filed and perform the field redefinition [59,64]: Applying this shift of the Higgs field throughout the Standard Model Lagrangian L SM leads to multiple Higgs couplings to any pair of massive gauge bosons or massive fermions. The effective couplings c t and c 2t are derived as We note that c t and c 2t have different dependence on the Wilson coefficients c H and c y . Also, c t = 1 − c H /2 + c y and c 2t = 3c y /2 − c H /2, when c y , c H 1 [59,60]. In Fig. 1 the dependence of c t and c 2t on c H and c y is shown. It is clear that the slope of c 2t (dashed lines) is different from c t (solid lines), especially when c 2t < 0 and c t < 1. Precise study on the effective coupling c 2t  therefore offers the possibility to discriminate the effects of c H and c y .
As shown in Eqs. (3) and (4), c 3 and c g also contribute to the Higgs pair production cross section. However, c g is already constrained to be within c − g and c + g by the signal strength measurements of single Higgs production gg → h [10,11] where R h ≡ σ(gg → h)/σ SM (gg → h), the sign "±" refers to the cases c t F > −2/3c g and c t F < −2/3c g , respectively. The combined fit to the single Higgs production, which depends on both c t and c g [56], and decay using 13 TeV LHC data gives rise to R h = 1.07 ± 0.09 (ATLAS [10]) and R h = 1.23 ± 0.13 (CMS [11]). By fixing R h , we can replace c g with a function of c t . Further- The green and gray bands correspond to the constraints, at the 2σ C.L., from the measurements of tth and single Higgs production cross section, respectively, at the 13 TeV LHC. The red box denotes (cH = 0, cy = 0), while the red dashed line denotes ct = 1. c ± g refers to the sign choices "±" in Eq. (7) with R h being fit from single Higgs production and decay data. more, the coupling c 3 in Eq. (3) is approximately equal to 1 − 3 2 c H + c 6 , as shown in Refs. [59,60], and is weakly constrained, −5.0 < c 3 < 12.1, by present data [65,66]. Here, c 6 is the coefficient of the dimension-6 operator O 6 = λ/v 2 |H| 6 . As the Higgs pair production cross sec-tion is more sensitive to the sign of c 3 , rather than its magnitude [67], we will choose c 3 = ±1 as our benchmark value in the following numerical analysis.
To compare σ(gg → hh) with the SM prediction, we define a ratio R hh as In Fig. 2, we show the contours of R hh at the 14 TeV LHC in the plane of c t and c 2t , with other parameters chosen as c g = 0 and c 3 = 1. R hh can be enhanced largely for both the positive and negative c 2t , but R hh is more sensitive to negative c 2t when c t is of order one, which can be understood with Eq. (4). In the large top quark mass limit, F → 2/3 and F → −2/3 [62]. Besides, the c 2 t F term dominates over the c t F term for c t 1. Therefore, a negative c 2t can enhance R hh more easily than a positive c 2t , for this choice of c g and c 3 [57].
With the information from Figs. 1 and 2, we conclude that it is hopeful to discriminate the effects of c H and c y through Higgs pair production, especially for the negative c 2t region. Furthermore, we could translate the above results in the plane of (c H , c y ). In Fig. 3, we show the contours of R hh with respect to the Wilson coefficients c H and c y , where various choices of the effective couplings c g and c 3 are considered. They correspond to For the cases (a) and (b), the hgg effective coupling c g is assumed to vanish while the trilinear Higgs self-coupling c 3 is assumed to be +1 and −1, respectively. In these cases, we include the constraints on the parameter space of c H and c y from the single Higgs production [10,11]; cf. the gray band of Fig. 3 (a, b). It amounts to c t = 1.03 ± 0.04 (ATLAS) and c t = 1.11 ± 0.06 (CMS). For the cases from (c) to (h), c g is derived from a given R h value, cf. Eq. (7), while c 3 is fixed to be identical to the SM value. For comparison, we also show the parameter space constrained by the tth measurements at the 13 TeV LHC, with an integrated luminosity of 79.8 fb −1 from ATLAS [8] and of 35.9 fb −1 from CMS [9]. Here, we take the combined best fit tth signal strength normalized to the SM prediction, such that c t = 1.15 +0.12 −0.11 (ATLAS) and c t = 1.12 +0.14 −0.12 (CMS). With the result depicted in Fig. 3, several comments are in order: • R hh is enhanced in some parameter space of c H and c y , the magnitude of the enhancement strongly depends on the choice of c 3 and c g .
• In case (b), the cancellation between the triangle diagram and box diagram does not happen, because of the negative c 3 which further enhances R hh [30].
• In case of c + g or c − g , the value of c g is extracted from the signal strength of single Higgs production process R h . We find that the variation of R h = 0.9, 1, or 1.1 can only slightly change the value of R hh .
• R hh is more sensitive to c − g than c + g . According to Eq. (7), c + g is close to zero for a positive c t , and accordingly the contours of R hh are similar to the contours in case (a). On the other hand, negative c − g can significantly deviate from zero and R hh is largely enhanced (cf. Eq. (4)).
• In cases (a), (c), (e) and (g), the possible enhancement of R hh can only come from the deviations of c t and c 2t . With the information of Figs. 1 and 2, R hh can be largely enhanced when c 2t < 0, which corresponds to the region c y < 0 in Fig. 3.
Sensitivity at the 14 TeV LHC and the 100 TeV hadron collider. Now we discuss the potential of discriminating the Wilson coefficients c H and c y at the 14 TeV LHC and the 100 TeV proton-proton hadron collider. As a concrete example, we examine the bbγγ channel, which has been studied by the ATLAS collaboration at the High-Luminosity LHC (HL-LHC), operating at the center-of-mass energy of 14 TeV with an integrated luminosity of 3 ab −1 [68]. As being discussed in Refs. [56,57], an analytical function can be used to describe the fraction of signal events passing through the kinematic cuts. Since the Higgs boson is a scalar particle and the gg → hh process is dominated by the s-wave contribution, the acceptance of the kinematic cuts, in the inclusive Higgs pair production, will mainly depend on the invariant mass of the Higgs boson pair (m hh ). The cross section of gg → hh, after imposing the kinemati cuts, can be written as [56] where the efficiency function A(m hh ) has been given in Refs. [56,57], both for the 14 TeV LHC and the 100 TeV hadron collider. In Ref. [57], it is demonstrated that the results obtained from the analytic cut efficiency functions agree very well with other results with more detailed simulations [60]. The SM backgrounds for the process of gg → hh production include bbγγ, ccγγ, bbγj, jjγγ, bbjj, tt( 1 ± ), ttγ, Z(→ bb)h(→ γγ), tth(→ γγ) and bbh(→ γγ), etc. At the 14 TeV LHC, with the integrated luminosity of L = 3 ab −1 [68], and the 100 TeV hadron collider, with L = 30 ab −1 [69], the signal (n s ) and background (n b ) The 2σ or 5σ discovery potential for gg → hh in the plane of cH and cy at the 14 TeV LHC. The green bands correspond to the constraints, at the 2σ C.L., from the measurements of tth at the 13 TeV LHC. The gray and yellow bands represent the projected 2σ errors in the single Higgs production and tth measurement at the HL-LHC, respectively. The red box denotes (cH = 0, cy = 0), while the red dashed line corresponds to ct = 1. c ± g refers to the sign choices "±" in Eq. (7) with R h being fit from single Higgs production and decay data.
With the event numbers listed above, the discovery potential for the signal process can be evaluated by using [70] Z = 2 (n s + n b ) log In Figs. 4 and 5, we show the contours of discovery potential for gg → hh → bbγγ at the 14 TeV LHC and the 100 TeV hadron collider, with various integrated luminosities. The 2σ and 5σ discovery potentials correspond to Z = 2 and Z = 5, respectively. As mentioned earlier, the signal strength R h is measured with an accuracy of about 10%, and R hh is not sensitive to the value of R h (for 0.9 ≤ R h ≤ 1.1), we therefore take R h = 1 as the benchmark to show the discovery potential in those figures. Several comments are in order regarding the discovery potential of gg → hh with respect to the Wilson coefficients c H and c y . In Fig. 4, only the parameter space The 5σ discovery potential for gg → hh in the plane of cH and cy at the 100 TeV hadron collider. The green bands correspond to the constraints, at the 2σ C.L., from the measurements of tth at the 13 TeV LHC. The gray and yellow bands represent the projected 2σ errors in the tth measurement at the 100 TeV hadron collider, with an integrated luminosity of 3 ab −1 at 2σ C.L., respectively. The red box denotes (cH = 0, cy = 0), while the red dashed line corresponds to ct = 1. c ± g refers to the sign choices "±" in Eq. (7) with R h being fit from single Higgs production and decay data.
on the left of the curve, or below the curve, labeled by a specified integrated luminosity at the HL-LHC, can be reached at the 2σ or 5σ C.L.. For case (a), the SM point cannot be probed by measuring only the gg → hh production. A larger parameter space is reachable at 2σ C.L. for case (b), as compared to cases (a) and (c), due to the large enhancement of R hh with negative c 3 . For case (d), with negative c g , the Higgs pair production cross section can be enhanced by a factor of about 10, as compared to the SM value (cf. Fig. 3(f )). Consequently, with an integrated luminosity of 1 ab −1 at the HL-LHC, most of the considered parameter space (for |c H | < 1 and |c y | < 1) can already be reached at the 5σ C.L.. For comparison, in the same figure, we also show the constraints imposed by the projected 2σ errors in the single Higgs production and tth measurement at the HL-LHC, with an integrated luminosity of 3 ab −1 . It amounts to c t = 1 ± 0.02 (single Higgs with c g = 0) and c t = 1±0.05 (tth production) [71] which have included the statistical and experimental systematic uncertainties.
In Fig. 5, we see that the discovery potential for the gg → hh measurement is much improved at the 100 TeV hadron collider, for all the benchmark cases. The 5σ discovery significance can be easily reached. For cases (a) and (c), large region of c y , c H < 0.2 can be discovered with the integrated luminosity of 0.5 ab −1 . For case (b), with negative c 3 , the region of c y , c H < 0.5 can be almost discovered with only 0.1 ab −1 . For case (d), with negative c g , the currently allowed region can be explored with 2 fb −1 . For comparison, in the same figure, we also show the constraints imposed by the projected 2σ errors in the single Higgs production and tth measurement at the 100 TeV hadron collider, with an integrated luminosity of 3 ab −1 . It amounts to c t = 1 ± 0.0072 (single Higgs production, with c g = 0) and c t = 1 ± 0.013 (tth production) [72,73]. Note that both the statistical and systematic uncertainties are included in the single Higgs analysis, while only statistical error is discussed in tth production. Here, we have scaled down the error in tth measurement by the inverse of the square root of integrated luminosity.
To estimate the expected accuracy for measuring (c H , c y ) with the Higgs pair production at the 14 TeV LHC and the 100 TeV hadron collider, respectively, we perform a log likelihood ratio test [70] for the hypothesis with non-zero c H , c y against the hypothesis with c H = c y = 0. The test ratio is defined as [70] where the likelihood function L(c H , c y ) is Here P (k|λ) is the usual Poisson distribution function, P (k|λ) = λ k e −λ /k!. We assume the observed data is generated under the hypothesis with c H = c y = 0 [70,74] and calculate the two-sided p-value. For convenience, we convert the p-value into the equivalent significance [70,75], where Φ the cumulative distribution of the standard Gaussian and Erf is the error function. In our case, Z = 2 n 0 ln n 0 n 1 + (n 1 − n 0 ) with n 0 = n b + n s (0, 0) and n 1 = n b +n s (c H , c y ). The discrimination between the hypothesis with arbitrary (c H , c y ) and the hypothesis with c H = c y = 0 is shown in Figs. 6 and 7. The hypothesis with (c H , c y ) outside of the blue bands is rejected at 1σ level for the HL-LHC and the 100 TeV hadron collider, respectively. After combining the measurements of single Higgs production (gray band) and tth production (yellow band) at the HL-LHC, it is possible to differentiate c H and c y at the HL-LHC for cases (b) and (d), but it becomes challenging for cases (a) and (c), cf. Fig. 6. The situation will be much improved at the 100 TeV hadron collider, cf. Fig. 7. It is obvious that the Higgs pair production is more sensitive to c y than to c H . We find that at the 1σ C.L., the combined constraint from single Higgs production, tth production and the Higgs pair production measurements, at the 100 TeV hadron collider with the The gray, yellow and blue bands represent the 1σ constraint from the measurements of single Higgs production, tth production and Higgs pair production at the HL-LHC, respectively. Both the statistical and experimental systematic uncertainties have been included in the single Higgs production and tth production. The red box denotes (cH = 0, cy = 0), while the red dashed line coresponds to ct = 1. integrated luminosity of 3 ab −1 , yields −0.04 < c y < 0 and −0.08 < c H < 0.08. To further constrain c H , it is necessary to improve the measurement of tth associated production. However, if the Higgs boson is a SM-like particle, c H could be constrained by the hV V coupling measurement to 1% ∼ 2% level [73].
Given the good sensitivity of differentiating c H with c y at the 100 TeV hadron collider, it is worthwhile clarifying the specific values of (c H , c y ), as induced by several generic classes of NP models [21,23,76]. Table I lists the Wilson coefficients c H and c y predicted by various NP models [21,23,76]. Both heavy scalars and vectors could contribute to c H and c y , while the additional heavy vector-like quarks (VLQs) could contribute to c y and c g . The sign of c y is arbitrary in two Higgs doublet (2HDM) [21] and VLQ models [23]. Those NP models can be easily discriminated if they modify c H or c y by a sizable amount, cf. Fig. 7.
Conclusions. Both the Wilson coefficients c H and c y of dimension-six operators can contribute to the top quark Yukawa coupling simultaneously, thus their individual contributions cannot be separated with the measurement of htt coupling alone. In this work, we demonstrate that c H and c y also contribute to the tthh effective coupling Expected accuracy for measuring (cH , cy) with single Higgs production, tth production and the Higgs pair production at the 100 TeV collider, with the integrated luminosity of 3 ab −1 . The gray, yellow and blue bands represent the 1σ constraint from the measurements of single Higgs production, tth production and Higgs pair production at the 100 TeV collider, respectively. Both the statistical and experimental systematic uncertainties have been included in the single Higgs production. The red box denotes (cH = 0, cy = 0), while the red dashed line coresponds to ct = 1. For comparison, several benchmark points of NP models are also shown: A) singlet scalar, B) 2HDM or VLQs with cy > 0, C) real triplet scalar, D) complex triplet scalar, E) vectors, F) 2HDM or VLQs with cy < 0.  [21,23,76].
A) singlet scalar [21,76] B) 2HDM [21] or VLQs [23] cH > 0, cy = 0 cH = 0, cy > 0 C) real triplet scalar [76] D) complex triplet scalar [76] cH = 2cy < 0 cH = cy < 0 E) vectors [76] F) 2HDM [21] or VLQs [23] cH = 3cy > 0 cH = 0, cy < 0 c 2t , whose information can be well extracted out from the Higgs pair production. Thus this process can be used to distinguish the effects of c H and c y at the 14 TeV LHC and the 100 TeV hadron collider. Regarding the discovery potential for the process gg → hh, it shows that the 2σ confidence level is reachable for some parameter space at the 14 TeV LHC in general, and the sensitivity can be much improved at the 100 TeV hadron collider. Regrad-ing the sensitivity of measuring c t and c H , we find it is challenging to differentiate c H and c y at the HL-LHC, except for some special scenarios. The situation will be much improved at the 100 TeV hadron collider. After combing the single Higgs production, tth production at the 100 TeV hadron collder, the Higgs pair production can give a strong constraint on c y regardless of the value c H . The precise measurement of both c H and c y enable us to discriminate various new physics models.