New theoretical constraints on scalar color octet models

We study theoretical constraints on a model whose scalar sector contains three $SU(2)_L$ doublets, two of which are color singlets, and one a color octet. The model possesses two limiting cases: the Two-Higgs-Doublet Model (2HDM) and the Manohar-Wise (MW) model leading to us calling it the 2HDMW. We use the unitarity of the theory to constrain the parameters of the scalar potential at next-to-leading order in perturbation theory. We also derive conditions guaranteeing the stability of the potential. Using the HEPfit package, we single out the viable parameter regions at the electroweak scale and test the stability of the renormalization group evolution up to the multiple TeV region. Additionally, we set upper limits on the scalar mass splittings. All results are also given for the MW limiting case.


I. INTRODUCTION
The discovery of the 125 GeV Higgs boson at the LHC [1,2] exemplifies the success of the Standard Model (SM). However, considering the experimental precision, possibilities of physics beyond the SM (BSM) are not excluded yet. It is therefore important to determine to what extent different BSM scenarios are still viable. One of the simplest and most commonly studied extension of the SM is the Two-Higgs-Doublet Model (2HDM), which adds an additional Higgs field to the SM. Manohar and Wise (MW) proposed an alternative extension of the SM by adding a color-octet electroweak doublet scalar [3]. The phenomenology of the model has been studied in detail [4][5][6][7][8][9][10][11][12][13][14][15][16][17], including aspects such as production of scalars, the lower limit of the scalar masses and possible constraints on the parameter space. Following this direction, Ref. [18,19] recently considered extending the MW model in a 2HDM context. Given the null results at the LHC, any BSM physics that is discovered is likely to be more exotic in nature. In particular, the scalar sector of the model considered consists of two color-singlet electroweak doublets, Φ 1,2 , and one color-octet electroweak doublet, S. The MW model is the limiting case with Φ 1 → 0 or Φ 2 → 0, whereas the 2HDM is recovered in the limit S → 0. Due to the existence of these two limiting cases we will refer to this model as the 2HDMW.
Ref. [18] investigated tree-level constraints on the 2HDMW arising from symmetries and perturbative unitarity. A study of LHC phenomenology was also performed, and found that the color-octet scalar added to the 2HDM could produce large corrections to the one-loop couplings of the Higgs boson to two gluons or photons. Ref. [19] derived the one-loop beta functions for the scalar couplings in the 2HDMW, and the evolution of the renormalization group equations (RGEs) was then used to place upper limits on the parameters of the model. Similar practices were applied in studies of the the SM [20,21], the MW model [14], and the 2HDM [22][23][24]. The parameter space was further constrained in Ref. [19] by requiring no Landau poles (LPs) below a certain high energy scale Λ, the scalar potential being stable and perturbative unitarity satisfied at all scales below Λ. The perturbative unitarity constraints imposed on the model in Ref. [18,19] are leading order (LO), and a considerable region in the parameter space survives. It is a reasonable expectation that supplementing these constraints with corrections at higher orders, though complicated, can result in noticeable modifications to the surviving parameter space. In this paper we utilize the generic tool provided by Ref. [25][26][27] to explore these perturbative unitarity bounds at next-to-leading order (NLO). This is the first work that imposes next-to-leading order unitarity bounds on scalar color octets, as well as the first to derive the complete set of positivity conditions for both, MW and 2HDMW models. More generally this work focuses on theoretical constraints on the 2HDMW. An investigation of experimental bounds on the model is saved for future work. The rest of the paper is organized as follows: The 2HDMW model is defined in Section II. The theoretical constraints are explained in Sec. III. Following that our results for the surviving parameter space are presented in Sec. IV. Concluding remarks are given in Sec. V.

II. THE MODEL
As stated above, the scalar sector of the model consists of two color-singlet electroweak doublets Φ 1,2 , and one color-octet electroweak doublet S. The most general renormalizable potential of the scalar sector is [18,28]: All interactions between S, Φ 1 , and Φ 2 and the self-interactions are included. In Eq. (1), we use i, j as SU (2) indices; the notation S i = S A i T A , where A is color index. The trace is taken over the color indices.
The physical parameters of this model are the masses of the Φ 1 and Φ 2 fields, which we denote like in the 2HDM as m h , m H and m A for the neutral bosons and as m H ± for the charged Higgs particles, as well as the octet masses m R , m I and m S ± for the neutral scalar, the neutral pseudoscalar and the charged octet scalar of the MW model. Moreover, we will call the two angles of the diagonalization of the mass matrix in the 2HDM sector α and β, according to the convention in the literature.
We apply the following conditions to reduce the number of the parameters in the scalar potential and the Yukawa potential, defined below in Eq. (2) and Eq. (3), respectively.
• We restrict the 2HDM sector to be CP -conserving.
The mass degeneracies m H ± = m A and m S ± = m I result from custodial symmetry.
• We impose a Z 2 symmetry, which is only softly broken by quadratic terms. This prevents tree level flavor changing neutral currents (FCNCs), and further reduces the number of free parameters. The charge assignments we consider are given in Table I. Note that the original MW paper was motivated by the principle of minimal flavor violation [32,33]. This is in contrast with our approach of imposing Z 2 symmetry, which is motivated by the practicality of reducing the number of parameters in the scalar potential while still maintaining some ability to generate flavor effects.  The scalar potential of the model with the aforementioned constraints imposed reads where ν 4 = 0 in the Type I and the Type IIu 2HDMW and ω 4 = 0 in the Type IId 2HDMW, leaving us with four massive and twelve massless parameters. The masses of scalars and their mixing angles are obtained by diagonalizing the mass matrices of this model; the expressions of those physical parameters were presented in Eq. (6) and (7) in Ref. [18]. For an overview over all assumptions, a comparison with the limiting cases of the 2HDM and the MW model and an account of the free parameters, we refer to table II. The general Yukawa potential of the 2HDMW in the flavor eigenstate basis is given by where the η i are complex constants. In the Type I 2HDMW we have η D 1 ≡ 0 and in Type IIu (IId) η D 2 ≡ 0 and η D S ≡ 0 (η U S ≡ 0). We use the conventionH i = ε ij H * j , where H = Φ 1,2 , S, and a, b are flavor indices.

A. Priors
For our analysis we make use of the open source package HEPfit [34], which is linked to the Bayesian Analysis Toolkit [35]. Even if we will not apply experimental constraints and thus  not necessarily rely on a fitting tool, we chose this set-up for the following reasons: BAT can also deal with flat likelihood distributions and HEPfit is optimized for a fast evaluation of the constraints. The sampling covers the whole parameter space, so we cannot miss relevant regions. This is not guaranteed if we use a random scattering approach. Furthermore, the presented HEPfit implementation of the 2HDMW as well as the MW and 2HDM limiting cases are available for everyone [34] and can be used in future HEPfit studies on these models including experimental data.
In our Bayesian fits we use the following ranges for the 2HDMW parameters as priors: We fix β − α to π/2 in order to align the light Higgs h with the SM Higgs and reproduce its signal strength values at tree-level; and we set m 2 12 = 0 because its value is not relevant here. Note that we do not require m h = 125 GeV in the 2HDMW. This constraint can always be accomplished by adjusting m 2 12 . Only in the MW limiting case with Φ 2 = 0, we impose that the SM-like Higgs has a mass of 125.18 ± 0.16 GeV [36,37], which results in an almost fixed λ 1 , like in the SM.

B. Unitarity
The unitarity of the S-matrix can be used to place constraints on the parameters of a theory [38] (see also [14,25,[39][40][41][42][43]). If a certain combination of parameters becomes too large, an amplitude will appear to be non-unitary at a given order in perturbation theory. We will refer to these constraints as perturbative unitarity bounds, or just unitarity bounds for short, even though the more accurate statement is that perturbation theory is breaking down.
Considering only two-to-two scattering these constraints take the following forms at various orders in perturbation theory is the contribution at the th order in perturbation theory to the jth partial wave amplitude. The NLO+ inequality includes the square of NLO correction, and thus contains some, but not all of the NNLO contributions to the partial-wave amplitude. When considering the scattering of scalars at high energy only the j = 0 partial wave amplitude is important. The matrix of partial wave amplitudes is given by and we use a 0 to indicate the eigenvalues of a 0 . The two-to-two scattering matrix at tree level in the neutral, color singlet channel of the 2HDMW model was recently derived in Refs. [18,19]. The NLO unitarity bounds are then computed approximately using the algorithm of Ref. [27]. A virtue of this approach is its simplicity as it only relies on knowledge of the LO partial wave matrix and the one-loop scalar contributions to the beta functions of the theory. This algorithm is built on previous work in Ref. [25,26], and results for the special case of the 2HDM can be found in those references. The approximation is valid when the center-of-mass energy is much greater than the other scales in the problem. 1 As such we only start enforcing the unitarity bounds for RGE scales above 750 GeV ≈ √ 10 v, and do not impose unitarity bounds when running from the EW scale to 750 GeV.
We also enforce the smallness of higher order corrections to the partial wave amplitudes with the following constraint [25,26,44,45] for each eigenvalue of the partial wave matrix as long as a (0) 0 > 0.01.
In our simplified potential V fit , the positivity conditions reduce to

D. Positivity of the mass squares
Additional bounds are derived from requiring the masses of the colored scalars to be real: with v = v 2 1 + v 2 2 ≈ 246 GeV, and where s β and c β are sine and cosine of β, respectively with tan β = v 1 /v 2 . We must have m 2 S > 0 so that the vacuum preserves SU (3) C . Note that the mass splitting between the colored states is and m 2 S ± = m 2 I due to custodial symmetry.

E. Renormalization group stability
So far we only discussed theory constraints at the electroweak scale. Assuming the validity of the model up to some higher scale imposes bounds on the parameters: Scenarios that define a viable model at m Z could feature one (or more) quartic couplings with an unstable behavior under the renormalization group evolution to a higher scale. This could be due to a Landau pole, but also the boundedness-from-below criteria described in Section III C should be fulfilled at any scale. Furthermore, the unitarity conditions should be applied at least above some scale, µ u , as they are computed in the limit µ u √ λ i v with λ i being a quartic coupling of the theory. Here, we chose to use µ u = 750 GeV like in [26]. We only take into account the quartic coupling terms from [19] and neglect the contributions of Yukawa and gauge couplings to the RGEs. In Ref. [19] it was shown how the parameter space is constrained in three cases where log 10 (Λ/1 GeV) = 10, 13, 19 if one uses LO unitarity and 2HDM stability.

IV. RESULTS
While the boundedness-from-below constraints are trivial, we want to discuss the different unitarity constraints in the 2HDMW, before we consider higher scales and the effect of the theory constraints on the physical parameters for both, the 2HDMW and the MW model.

A. Different unitarity constraints
In Figure 1 we show the effects of LO, NLO and NLO+ criteria on the λ 4 vs. λ 3 and µ 4 vs. µ 3 planes as well as the impact of the R conditions explained in Section III B at the electroweak scale. We observe that-contrary to the 2HDM case [25,26]-the quartic couplings enjoy more freedom if we apply NLO(+) or the R criteria instead of the LO unitarity. The reason for this is that the LO unitarity conditions only depend on few quartic couplings and disallow extreme values for them, while in the NLO(+) case, large quartic couplings can be compensated by tuning some of the other quartic couplings. Along the diagonal of the left hand panel of Figure 1 we can observe the consequence of not applying the R criteria if the LO unitarity condition is accidentally small: In the small strip with |λ 4 + λ 3 | ≤ 0.01 the quartic coupling λ 4 can be larger than 11 in magnitude. If we compare all sets of unitarity constraints with the region that is stable at least up to 1 TeV and compatible with NLO+ unitarity and the R conditions, we observe that the latter is a very strong bound. We would like to stress that we recommend to use the NLO(+) unitarity conditions only at scales significantly larger than the electroweak vev because beyond LO the quartic couplings are running couplings evaluated at an energy much larger than v. Tree-level unitarity constrains the quartic couplings to the beige areas; the two sets of one-loop conditions NLO and NLO+ force the couplings to stay within the pink and light blue regions, respectively. The purple contour delimits the area compatible with the R conditions. The different unitarity bounds at the electroweak scale need to be compared to the regions with stable running including unitarity up to a scale of 1 TeV (red).

B. Combination of all theoretical constraints
In Figure 2 we illustrate the combination of the theory constraints with stability up to a certain scale in the λ 4 vs. λ 3 and ω 4 vs. ω 2 planes as representative examples of the 2HDMW. The limits obtained from the global fit to all quartic couplings of the 2HDMW and the MW limiting case can be found in Table III. In this section, we analyze three different scenarios: In the first case, we run all quartic couplings to the stability scale of µ st = 1 TeV, controlling at each iteration of the RG evolution if the potential is bounded from below and if all quartic couplings are in the perturbative regime, that is smaller than 4π in magnitude. We find that with these constraints, the absolute value of the quartic couplings at the electroweak scale cannot exceed limits between 3.3 and 8.5 (1.7 and 7.5) without applying any unitarity bound to the 2HDMW (MW). The has to be confronted with the second scenario, for which we add the NLO+ unitarity constraints as well as the R criteria at scales above 750 GeV to the previous fit. The impact on the parameters is quite sizable: In Figure 2 we see that the allowed regions shrink by a factor of 1.5 to 2. The maximally allowed values for the quartic couplings range from 2.2 to 5.7 in the 2HDMW and from 1.3 to 5.6 in the MW, see Table  st = 1 TeV without  III. Finally, we impose that the scalar potential with all discussed theory bounds is stable up to even higher scales Λ. Originally, we wanted to test high scales of 10 4.8 , 10 7.6 , 10 12 and 10 19 GeV, going in evenly spaced steps in the logarithm of log 10 Λ towards the Planck scale, but our fitting set-up turned out to become unstable beyond 10 4.8 GeV. If we choose 10 4.8 GeV≈ 63 TeV as our high scale example, all parameters have to be between −1.8 and 2.2 in the 2HDMW and between −1.6 and 2.2 in the MW. Hence, the limits at 63 TeV are stronger by a factor of about 2/5 with respect to the ones obtained with stability at 1 TeV.
In the MW limiting case the role of λ 1 is different, as it is the only parameter on which the mass of the SM-like Higgs depends; it is thus basically fixed by the Higgs mass measurements. Also, we do not impose any Z 2 symmetry on the MW model, that is we treat ν 4 as a free parameter.
Comparing our results with those of Ref. [19], we find that our allowed ranges for the quartic couplings assuming stability and NLO unitarity up to 63 TeV are more or less of the same size as previous limits using LO unitarity and no MW positivity up to 2 · 10 4 TeV.
The limits on the quartic couplings can be translated into bounds on the physical model parameters. Like in the 2HDM we observe strong restrictions of the differences between m 2 H and m 2 A [26,45], but also m 2 I cannot deviate very much from m 2 R , see Figure 3. The former mass square difference depends on the values of the λ i , while m 2 R − m 2 I is proportional to    Table III. Even if Fig. 2 and 3 were obtained for the Type I 2HDMW, the 2HDMW limits in Table III hold for all three types, only that either ν 4 or ω 4 have to be set to zero, depending on the type.

V. CONCLUSIONS
We have studied the NLO unitarity bounds on the 2HDMW, which extends the scalar sector of 2HDM with an additional color octet scalar. In contrast to the 2HDM case, the NLO unitarity bounds allow for larger quartic couplings than the LO. A color-octet scalar, therefore, is advantageous to the existence of heavier Higgs bosons.
In addition, we have derived a set of necessary conditions to bound the 2HDMW potential from below for the first time. These conditions constrain most of the quartic couplings except a few. They are also applicable to the MW model, for which all the quartic couplings are bounded in this limit.
Finally, we have combined all theoretical constraints and found limits of the couplings assuming stability at different scales. Requiring a stable potential at a higher scale favors smaller difference between pairs of neutral scalars, such as m A − m H and m R − m I . The next obvious step would be a combination with experimental constraints, for which our publicly available HEPfit implementation could be used.