Common origin of radiative neutrino mass, dark matter and leptogenesis in scotogenic Georgi-Machacek model

We explore the phenomenology of the Georgi-Machacek model extended with two Higgs doublets and vector fermion doublets invariant under $SU(2)\times U(1)_Y\times \mathcal {Z}_4$. The $\mathcal {Z}_4$ symmetry is broken spontaneously resulting an accidental $\mathcal {Z}_2$ which forbids triplet fields to generate any vacuum expectation value and leading to an inert dark sector providing a viable candidate for dark matter and generate neutrino mass radiatively. Another interesting feature of the model is leptogenesis arising from decay of vector-like fermions. A detailed study of the model is pursued in search for available parameter space consistent with the theoretical and experimental observations for dark matter, neutrino physics, flavour physics, matter-antimatter asymmetry in the Universe.


Introduction
Evidences from cosmology and astrophysics claim that about a quarter of the Universe is made up of dark matter [1].
However, the nature of dark matter remains unknown as the Standard Model (SM) of particle physics fails to provide a viable dark matter (DM) candidate. Despite being celebrated as the most successful theory after the discovery of Higgs boson at collider experiments, various theories beyond the SM are proposed in order to explain the dark matter.
Simple extensions of SM with DM candidate are probed considering the stability of dark matter is protected by an additional discrete symmetry (such as Z 2 or Z 3 etc.). Apart from the dark matter, the origin of neutrino mass also remains unexplained by the SM of particle physics despite various neutrinos oscillation experiments has confirmed that neutrinos are massive [2]. This discrepancy in SM also calls for theories beyond the SM. Neutrino masses can indeed be generated by various see-saw mechanisms [3][4][5][6][7][8][9][10][11][12][13] at tree level with new heavy fermions and scalar fields, which can also generate matter-antimatter asymmetry in the Universe through the leptogenesis mechanism [14][15][16][17][18].
Among various extensions beyond the SM, two Higgs doublet model (2HDM) is one of the most simplest extensions where an additional scalar doublet similar to SM Higgs doublet is added [29][30][31][32]. In conventional models of 2HDM, the newly added doublet is assumed to be odd under a Z 2 symmetry which is broken spontaneously after electroweak symmetry is broken. After spontaneous symmetry breaking (SSB) doublet scalar fields obtain vacuum expectation values (VEVs) and mix up, resulting new physical Higgs particles. In the present work, we explore the phenomenology of a 2HDM extended Georgi-Machacek (GM) model [33][34][35]. In the GM model, new scalar triplets with hypercharge Y = 0 and Y = 1 are added to the SM scalar sector in a way that the electroweak ρ-parameter remains unaffected by simply assuming both the triplets develop same VEVs after spontaneous breaking of symmetry [36][37][38][39][40]. The hypercharge Y = 1 triplet in GM model also produces tiny neutrino masses at tree level. We propose an extension of the GM model with an additional scalar doublet and new vector-like fermions charged under SU (2) × U (1) × Z 4 symmetry. After spontaneous breaking of the symmetry, both the doublet scalar fields develop VEVs resembling the usual 2HDM while the triplet fields and vector fermion doublets remain protected by residual Z 2 which constitutes the dark sector. The neutrino mass generation at tree level via triplet field is prohibited. However, new vector-like fermions provide a window to radiatively generate neutrino mass in one-loop level. In addition, the lightest neutral scalar particle arising from triplet fields being charged under the remnant Z 2 , is stable and serves as a dark matter candidate. Finally, CP violating decays of heavy vector fermions can explain the matter-antimatter asymmetry in the Universe via leptogenesis. With new scalar and fermion fields the modified GM model is referred as scotogenic Georgi-Machacek (sGM) model.
The paper is organized as follows, we firstly describe the sGM model in detail. The phenomenologies of the model are then performed with considering various theoretical and experimental limits. Study of leptogenesis within the model is presented in the next section. Finally we conclude the paper.

sGM Model
The scalar sector of the GM model [33] contains one Higgs doublet φ with Y = 1/2, one complex triplet ∆ with Y = 1 and a real scalar triplet T with Y = 0 fields.
where the neutral components are parametrized as with v φ , v ∆ and v T being the VEVs for φ 0 , D 0 and T 0 , respectively. The most general form of the Higgs potential, invariant under the SU (2) L × U (1) Y gauge symmetry, is given by whereφ = iτ 2 φ * . In the above Eq. (3), µ 2 and κ can be complex.
In the present work we extend the model in the framework of two Higgs doublet with vector-like fermions (VLF's), We consider a Z 4 symmetry associated with the model instead of standard Z 2 symmetry in 2HDM. Charges of different particles and fields are given in Table I. The general form of the Higgs potential, invariant under the SU (2) L × U (1) Y × Z 4 symmetry, is parametrized as whereφ 1,2 ≡ iσ 2 φ * 1,2 , and the 2HDM potential is given by The m 2 12 term in Eq. (6) breaks the Z 4 symmetry of the potential softly similarly to the case of standard two Higgs doublet model. The Z 4 symmetry is broken spontaneously and the doublet fields φ 1 , φ 2 acquires VEVs. Interestingly, although the Z 4 symmetry is broken, the fact that T and ∆ fields do not generate any VEVs indicates a hidden residual Z 2 symmetry. Therefore, this accidental Z 2 symmetry provides feasible candidates for dark matter arising from the triplet fields in the model. After spontaneous breaking of symmetry we get new physical scalar in visible 2HDM sector and a dark sector originating from triplet scalars. The neutral scalar particles and singly charged particles of the triplets T and ∆ mix with each other and provides two neutral physical scalars and two physical charged scalars in dark sector. However, since the VEV of ∆ field is zero due to the residual symmetry Z 2 , the neutrino mass is vanishing at tree level in the model. This issue can be resolved by the newly added vector-like fermions which also respects the unbroken Z 2 symmetry.
Gauge invariant Yukawa interactions of the fermions with triplet scalars in the present framework are given as We will later show that these new interaction terms provide necessary ingredients to radiatively generate neutrino masses in one-loop.

Scalar Sector
After spontaneous breaking of SU (2) L × U (1) Y × Z 4 symmetry, Higgs fields φ 1 and φ 2 acquire vacuum expectation The visible sector is identical to 2HDM which contains two neutral physical scalars (h, H), one pseudo-scalar particle A and a pair of charged scalar H ± . Conditions for the minimization of the potential are Different couplings λ i (i = 1, ..., 5) are expressed in terms of physical masses m h , m H , m A , m H ± and parameters α, β and m 12 , where we denote s α,β = sin α, sin β, c α,β = cos α, cos β, with tan β = v 2 /v 1 . Here m h = 125 GeV is the mass of SM Higgs and m H is denoted as the mass of heavy Higgs boson. After the SSB, the scalar fields T and ∆ acquire zero VEVs which resemble an unbroken residual Z 2 symmetry. Therefore, the scalar fields T and ∆ are inert in nature.
Defining a new mixing angle δ, we write physical charged scalars as Masses of physical charged scalars are with y = tan 2δ =

Vacuum stability
We adopt the criteria of copositivity of symmetric matrices to get the conditions of vacuum stability [41]. Only the quartic terms in the potential should be considered, since these terms dominate at large field value. It is to be noted that vacuum stability conditions don't give any constraints to couplings κ 41 and κ 42 . This is because we can make these couplings positive by applying a phase rotation of a field or field redefinition. We parameterize the fields as, where f, g, m, n ∈ [0, 1]. According to the definition above, considering the quartic terms out and ignoring κ 41 and κ 42 terms, the potential of Eq. (5) can be expressed as The symmetric matrix of quartic couplings can be represented in basis (h 2 1 , h 2 2 , δ 2 , t 2 ) as where λ 345 = λ 3 + f 2 (λ 4 − |λ 5 |). If (λ 4 − |λ 5 |) ≥ 0, the minimum of the potential is obtained for f = 0, whereas for (λ 4 − |λ 5 |) < 0 the minimum obtained assuming f = 1. Similar convention can be employed to parameters g, m, n to apply the copositivity criteria upon the matrix M . Copositive conditions for which vacuum of the scalar potential in Eq. (22) becomes stable are: where we used a theorem of copositivity criteria to get copositive matrix conditions [42]. The conditions mentioned above are derived assuming ρ 4 ≥ 0 and ρ 5 ≥ 0.
In addition, scalar and fermion couplings must also remain within the perturbative limit for which following conditions must be satisfied

Neutrino mass
The neutrino masses are generated via one-loop diagram as shown in Fig. 1. The Yukawa interaction terms in Eq. (7) and scalar terms with κ 41 and κ 42 appearing in Eq. (5) are responsible for generation of light neutrino mass.
The Z 4 symmetry in the model assures that two different scalar doublets are necessary in order to generate tiny neutrino mass in one-loop. Neutrino mass in the present model can further be realized as self-energy corrections. However, there will be contributions from two different diagrams involving neutral scalars and charged scalars as shown in Fig. 2. The expression of neutrino mass is given as It is to be noted that, in the present scenario if the mixing angle between neutral physical scalar sin γ → 0, then mixing angle between charged scalars (sin δ) also becomes zero. The conditions for which those mixing angles becomes zero are Therefore, in order to generate tiny neutrino mass one must have κ 41,42 = 0 and κ 41 = −κ 42 .
We consider vector fermions to be much heavier than scalar particles M Σ m S1,2 , m S + 1,2 , and further assume With the above simplified choice, following Eqs. (14), (17) and Eq. (20) we can rewrite the neutrino mass matrix elements as Therefore, from Eq. (28), one can realize seesaw like radiative neutrino mass expressed as where we assumed that the mass matrix of vector fermion ζ = diag{ζ 1 , ζ 2 , ζ 3 } and the diagonal matrix elements are given as Therefore, if one considers ( where In the present work we include three vector fermion doublets. However, since vector fermions do not mix with each other and mass of charged and neutral fermions are degenerate (i.e.; M Σ + = M Σ 0 = M Σ for a single generation), the contributionT VLF = 0.
Let us now discuss how electroweak precision observables modify in presence of new scalars. In case of 2HDM within alignment limit (β − α) = π/2, contribution toT parameter reads as [44][45][46] where From Eq. (34), it can be concluded thatT 2HDM vanishes for m A = m H ± or m H = m H ± . Therefore, if one considers above conditions, new contribution toT parameter will arise from triplet scalar fields T and ∆ only. It is to noted that, in the present formalism there exists mixing between neutral and charged components of the triplet fields, which must be taken into account to calculateT . Considering the effects of mixing, additional contribution toT parameter In the above Eq. (36), s θ = sin θ and c θ = cos θ where θ = γ, δ. Using the above expression, bounds on the mixing angles or mass splittings between scalar particles can be obtained in the present model. We use the value T =T new α = 0.07 ± 0.12 [2] to constrain the model parameter space.

Collider bounds
In this section, we briefly discuss collider constraints on different visible and dark sector particles of the model.
LEP excludes a new charged scalar of mass less than 80 GeV from charged scalar decay into cs and ντ final state [47].
Similarly bound on charged fermion mass is 101.2 GeV [48,49] from the decay of charged fermion into νW ± final state. Let us now consider the bounds obtained from LHC. In the present model we have singly charged scalars S ± 1,2 and one doubly charged scalar D ±± in dark sector. These particles can contribute to the Higgs to diphoton decay process. The decay rate for the process h → γγ is given as Here G F is the Fermi coupling constant, α is the fine-structure constant, N c = 3(1) for quarks (leptons), Q f is the electric charge of the fermion in the loop, and Couplings of SM Higgs h with different charged scalars in Eq. (37) are listed in Appendix. The relevant loop functions are given by and the function f (τ ) is given by One can calculate the quantity called signal strength R γγ given as Using the latest and future constraints from LHC on the R γγ signal strength [50,51], masses of charged particles in our model can be constrained. It is to be noted that for heavier masses of charged inert scalar particles with mass around TeV, one can safely work with 2HDM constraints. In the present framework, we consider only one Higgs doublet φ 2 couples to the SM sector which resembles the standard type-I 2HDM. This is possible because Z 4 charge of scalar doublets are +1 and -1 (as in Table I) which is equivalent to standard 2HDM with Z 2 charge. For type-I 2HDM, Higgs signal strength does not provide any limit on tan β with alignment limit (β − α) = π/2. However, deviation from the alignment limit puts stringent bound on tan β for type-I 2HDM [52][53][54][55]. Apart from Higgs signal strength measurement, the most stringent bound on charged scalar mass comes from flavor physics when the B → X s γ decay is taken into account [56,57]. It is found that for all types of 2HDM including type-I 2HDM, inclusive decay b → sγ excludes charged Higgs mass below 650 GeV for tan β < 1 [57]. Therefore, in the present framework, with the alignment limit (β − α) = π/2, we conservatively work with following conditions

Dark Matter
As mentioned before, the lightest neutral scalar S 1 or S 2 can serve the role of dark matter candidate in the model. We implement the model in LanHEP-3.3.2 [58] and calculate relic density and direct detection of the dark matter using micrOMEGAs-4.3.5 [59]. We use the dark matter relic density observed by PLANCK [1] and constrain the model with the dark matter-nucleon scattering cross-section limits from direct search experiments XENON1T [60,61] and PandaX-II [62,63]. κ41 κ42 κ21 κ22 DM candidate --+ + S1 ----S1 + + --S2 + + + + Lightest inert particle is charged Apart from 2HDM parameters (as in Eq. (43)), there are dark sector parameters in the model. We consider the following parameters to be independent parameters which constrain the dark sector in the present formalism which are given as m S2 , sin γ, κ 1i , κ 2i , κ 3i , κ 4i (i = 1, 2).
With the above choice of parameters the mixing between charged scalar particles (δ) becomes a dependent parameter.
With the above mentioned choice of scalar mixing, we consider a parameter space depending on the choice of κ 4i and κ 2i (i = 1, 2) in order to obtain a viable DM candidate in our model. We tabulate how the choice of couplings determines whether DM is neutral or charged as shown in Table II. It is to be noted that our conclusion in Table II is independent of the choice of couplings κ 1i and κ 3i (i = 1, 2). Therefore, in the present work, we consider only six relevant parameters to explore DM phenomenology, are given as m S2 , sin γ, κ 2i , κ 4i (i = 1, 2). (45) As observed in Table II, we exclude the case with κ 21,22 > 0 and κ 41,42 > 0 values. Therefore, we are left with two cases, I) neutral DM candidate is represented by S 1 and II) DM is represented by S 2 . We will discuss both the cases separately in details in this section. Before we discuss the DM phenomenology, we first consider the model parameter space in agreement with T parameter results. We consider a set of parameters following Table II Table II resulting S 1 (S 2 ) as the DM candidate. Since T parameter depends on the mass splitting, we observe that it is independent of the choice of sign of couplings mentioned in Table II and thus we ignore the second row of Table II Case:I, S1 dark matter In this section we study DM phenomenology for the case when DM is represented by S 1 candidate and resembles the T 0 candidate of Y = 0 triplet. We work with the conditions set by Table II for this purpose.
• Effects of sin γ Right panel: DM mass plotted against direct detection cross-section for the same set of parameters.
In Fig. 4(a), we show the variation of DM relic density with mass of DM for three different values of sin γ = (0.1 , 0.3 , 0.5). Fig. 4(a) is plotted for tan β = 1, κ 1i = κ 2i = κ 3i = 0.05 and κ 4i = −0.05 where i = 1, 2. In Fig. 4(b) the variation of dark matter direct detection against DM mass is depicted for same set of parameters and compare with the experimental bounds on DM-nucleon scattering cross-section with PandaX-II and XENON1T. It can be observed that changes in the mixing angle sin γ does not affect DM relic density very much and a 1.2 TeV dark matter candidate is in agreement with the relic density bound from PLANCK (red horizontal lines in left panel of Fig. 4).
However, with increase in the mixing angle, the DM direct detection cross-section tends to decreases considerably.
Therefore, large mixing angle sin γ is favorable for a viable DM candidate in the present framework.
• Effects of tan β In Fig. 5, we repeat the results for dark matter with same set of coupling parameters as used in case of Fig. 4 for different values of tan β = (1 , 5 , 10) with scalar mixing fixed at sin γ = 0.1. We observe in Fig. 5(a) that for different values of tan β, DM relic abundance does not change significantly with DM mass. However, DM-nucleon scattering cross-section increases with larger tan β values as seen in Fig. 5(b).
• Effects of κ 41,42  Fig. 4. In Fig. 6 we observe that as  in mass splitting between charged and neutral eigenstates S + 1,2 and S 1,2 . With increased mass splitting, contribution from co-annihilation channels decreases which reduces DM relic density for a given mass of DM as observed in Fig. 6.
However, no significant change in DM direct detection is observed for changes in coupling κ 41,42 .
• Effects of κ 11,12 and κ 31,32 Finally, we consider changes of couplings κ 1i and κ 3i , i = 1, 2 and study how these couplings affects dark matter phenomenology. It is found from the study that couplings κ 1i and κ 3i does not alter DM relic density significantly but can affect the DM direct detection cross-sections. This is obvious since, these couplings allow Higgs portal interactions with SM sector particles. However, since gauge interactions and co-annihilation channels dominate over Higgs portal interactions, these couplings are less sensitive to DM relic density. For the purpose of demonstration, we consider tan β = 1, sin γ = 0.1 and couplings κ 21,22 , κ 41,42 etc. used in previous set of plots shown in Fig. 4-5. From Fig. 7, we observe that changing value of κ 11,12 = −0.05 from κ 11,12 = 0.05, does not affect direct detection results significantly. In the previous section, we have presented the results for DM dominated by S 1 . In this section we consider the case when DM is S 2 dominated which resemble the neutral candidate of Y = 1 triplet. For this purpose, we follow the conditions mentioned in Table II. • Effects of sin γ Similar to the previous section, we now repeat our study for the case considering DM candidate is S 2 with positive values of κ 41,42 following Table II. In Fig. 8(a)-(b), we show the variation of DM relic density and direct detection cross-section against DM mass for three different values of sin γ plotted with set of coupling parameters κ 11,12 = κ 31,32 = κ 41,42 = 0.05 and κ 21,22 = −0.05 using tan β = 1. We observe that, even DM candidate is S 2 -like, with different sin γ values, DM mass versus relic density plots does not undergo any significant changes and 1.2 TeV dark matter satisfies DM relic density. However, significant changes in DM mass versus DM-nucleon scattering cross-section plot can be observed with changes in sin γ values as depicted in Fig. 8(b). It is clearly shown that with increasing sin γ, the DM direct detection cross-section gains large enhancement. This is quite different when compared with the case of S 1 DM case presented in Fig. 4(b).

• Effects of tan β
We now repeat the study with three different tan β = 1, 5, 10 values for sin γ = 0.1, keeping other parameters fixed as taken in previous case (Fig. 8), and present our findings in Fig. 9. From Fig. 9(a), we conclude that changes in  tan β is less significant to DM relic density. However, Fig. 9(b) predicts that for larger tan β, DM-nucleon cross-section tends to decrease.
• Effects of κ 41,42 In Fig. 10, we repeat our study following Fig. 6 by varying κ 41,42 from 0.05 to 0.5 for sin γ = 0.1 and tan β = 1 keeping the rest of the parameters unchanged as considered in Fig. 8. Similar to the case of dark matter in Fig. 6, here we also observe large changes in DM relic density which predicts that DM can have mass around 1.2 TeV to 1.6 TeV. This can be explained by the rise in mass splitting between dark sector particles which causes reduction in DM annihilation cross-section happening from DM co-annihilation. Note that, for different values of κ 4i (i = 1 − 2), DM direct detection cross-section remains unchanged similar to the case of S 1 dark matter.
• Effects of κ 11,12 and κ 31,32  Similar to the study of S 1 dark matter phenomenology performed in Fig. 7, we investigate the effects of κ 11,12 and κ 31,32 in Fig. 11 on DM-nucleon scattering cross-section. As we have mentioned before, due to the effects of gauge annihilations and co-annihilation of dark matter candidate, these couplings are not significant to DM relic abundance.
Looking into Fig. 11, we conclude that dark matter direct detection results remains almost unaffected with change in sign of κ 31,32 from positive to negative. However, for the case κ 11,12 = −0.05, there is significant enhancement in DM direct detection cross-section when compared with the case κ 11,12 = 0.05, which can be probed by next generation DM direct search experiments and constrain the available model parameter space. κ41 κ42 κ21 κ22 DM candidate + + + + S1 --+ + Lightest inert particle is charged It is to be noted that apart from the region of parameter space considered in Table II, there also exists another region of available parameter space as given in Table III for − 1 √ 2 ≤ sin γ ≤ 0. However, the DM phenomenology does not alter significantly for negative values of neutral scalar mixing sin γ and results are similar to cases discussed in Figs. 4-10. Therefore, we will not discuss the combination of parameter space mentioned in Table III. Therefore, DM phenomenology of 2HDM extension of inert GM model suggests that a viable TeV scale dark matter can achieved within the framework. Since the masses of dark sector particles are around few TeVs, their contribution to the process h → γγ are negligible compared to the contribution from W ± , H ± and fermion (top and bottom quarks) and thus can be ignored safely. In other words, conditions described in Eq. (43) for type-I 2HDM remains unaltered. In the present model, Yukawa interactions mentioned in the Eq. (7) will result in charged lepton flavor violation. In Fig. 12, we show possible diagrams that contribute to flavor violating decay. The Feynman amplitude for the process l i → l j γ is given as

Lepton flavor violation
where the form factors F and G are expressed as where, In Eq. (48), we have dropped the indices of M Σ k for simplicity. It is to be noted that for a vector fermion doublet Stringent bound on flavor violating process µ → eγ is obtained from MEG experiment [64].
Upper limit on the decay branching ratio for µ → eγ decay is B(µ → eγ) < 4.2 × 10 −13 . The decay rate for the process µ → eγ in the present model is given by In Fig. 13(a) GeV. Using the allowed range of M Σ1 and Yukawa coupling values, we plot the corresponding decay branching ratio B(µ → eγ) against M Σ1 , as shown in Fig. 13(b). It is clearly observed that within the specific range of λ(y) − M Σ1 , µ → eγ decay branching ratio is negligible compared to the experimental limit B(µ → eγ) < 4.2 × 10 −13 [64].
Therefore, lepton flavor violation do not impose any stringent limit in the present framework.

Leptogenesis in sGM model
Apart from the generation of one-loop neutrino mass and providing a feasible candidate for dark matter, the scotogenic GM model can also generate lepton asymmetry which can explain baryon asymmetry in the Universe (BAU). In the present model, heavy vector-like fermions can decay into leptons and triplets. A net asymmetry in lepton number can be generated from the CP violating decay of heavy vector-like fermions.
The net amount of asymmetry generated from CP violating decay of VLF's coming from its decay into leptons with two different scalar triplets are expressed as and In the above Eq. (50)-(51), total decay width of vector fermion is given as The asymmetry 1∆ can be redefined as Similarly, the asymmetry 1T becomes As mentioned before, in the following study we consider the mass matrix of vector-like fermions to be diagonal and assume hierarchical structure of vector-like fermion masses such that M Σ1 < M Σ2 < M Σ3 and M Σ1 : M Σ2 : M Σ3 = 1 : 3 : 30. Therefore, Boltzmann equations for leptogenesis consists of three equations (which is governed by decay of lightest vector fermion) and where Eq.
It is to be noted that Boltzmann Eqs. (56)- (57) are based on the assumption that the transfer of asymmetry between Y T L and Y ∆ L is negligible and their evolution are independent of each other. This is possible when the condition for the narrow width approximation is satisfied which is given as [65] The final lepton asymmetry obtained by solving Boltzmann equations for leptogenesis is given as In the above equation the additional factor of two is due to the fact that asymmetry is generated from both neutral and charged fermions of the vector fermion doublet. Eq. (61) reveals that, there can be a cancellation in net asymmetry Y L (z → ∞). Similar to the case of standard leptogenesis with right handed neutrinos, here the final lepton asymmetry is also partially transferred into baryon asymmetry via sphaleron transition [17] Baryon asymmetry of the Universe as measured by PLANCK is Y B = (8 − 9.5) × 10 −11 , hence the required lepton asymmetry is Y L ∼ (2.5 − 2.9) × 10 −10 [2].  We now investigate the possibility whether the present model can generate the required BAU. In order to do so, we first consider Yukawa couplings and mass of vetor-like fermions that will provide sub-eV light neutrino mass. For this purpose, we use the allowed M Σ1 and Yukawa coupling λ = y parameter space derived in Fig. 13(a) for neutrino mass scale m ν = 0.02 − 0.20 eV. Moreover, the Boltzmann equations for lepton asymmetry used is valid only when the condition for narrow width approximation in Eq. (60) is respected. It is found that the washout and transfer of asymmetry becomes significant for Γ 1 2 /M Σ1 H 1 ≥ 0.1 [65]. To circumvent this issue, we further restrict the allowed M Σ1 vs λ = y parameter space in Fig. 13(a) with the condition Γ 1 2 /M Σ1 H 1 < 0.1. In Fig. 14(a), we present the variation of vector fermion mass M Σ1 against the narrow width approximation parameter using the neutrino mass constraints from Fig. 13(a). The upper limit for narrow width approximation Γ 1 2 /M Σ1 H 1 < 0.1 is represented by the red horizontal line shown in Fig. 14(a). With the given range of neutrino mass scale m ν = 0.02 − 0.20 eV, Fig. 14(a) provides an upper limit on vector fermion mass for which narrow width approximation and the Boltzmann equations (Eqs. (56)-(57)) for lepton asymmetry remains valid. Using the allowed range of M Σ1 values obtained from Fig. 14(a) and Yukawa coupling from Fig. 13(a), we solve Boltzmann equations Eqs. (55)- (57) and evaluate the amount of lepton asymmetry in the present formalism. As to obtain the limits in Fig. 13(a) and Fig. 14(a), taking the condition λ = y we get 1∆ = 1T and Br T L = Br ∆ L = 1/2. Therefore, for λ = y, lepton asymmetry generated from Eqs. (56)-(57) will be equal, i.e.; Y ∆ L (z → ∞) = Y T L (z → ∞). In Fig. 14(b), we plot the total lepton asymmetry Y L versus Γ 1 2 /M Σ1 H 1 using the available M Σ1 parameter space obtained for aforementioned m ν values. Horizontal red lines in Fig. 14(b) exhibit the Y L value required to generate observed baryon asymmetry in the Universe. For demonstrative purpose, in Table IV, we tabulate lepton asymmetry Y L for few sets of benchmark parameters obtained from the solutions of Boltzmann equations for leptogenesis. We observe that for benchmark set-I, yield of lepton asymmetry Y L is above the required amount of matter-antimatter asymmetry. On the other hand, for benchmark set-II with m ν ∼ 0.06 eV, net lepton asymmetry Y L successfully generates observed baryon abundance. However, set-III result in Table IV suggests that yield Y L fails to explain baryon asymmetry in the Universe.
It is worth mentioning that solution to Boltzmann equations in this work is based on the assumption that the Universe is radiation dominated at the era of leptogenesis. However, this situation can alter if one considers a modified thermal history of Universe [66]. Recent studies [67][68][69][70] reveal that non-standard cosmological effects (such as a new scalar field active near the temperature T ∼ M Σ1 ) on leptogenesis can reduce the effects of washout significantly and enhance the yield of Y L by one or two order depending on the new parameters. We expect similar changes in lepton asymmetry Y L if non-standard cosmological effects are taken into account. In such scenarios, benchmark set-III of Table IV can also explain matter-antimatter asymmetry in the Universe. Therefore, non-standard leptogenesis scenarios can relax constraints and broaden the available model parameter space considered in the model. However at lower temperature T ∼ 10 2 −10 3 TeV, the non-standard effect is completely washed out and Universe becomes radiation dominated. Therefore, thermal freeze-out of dark matter remains unaffected and hence dark matter phenomenology discussed in the work remains unharmed.

Conclusions
In this work we present a study of common origin of neutrino mass, dark matter and leptogenesis by extending Georgi-Machacek model with two Higgs doublet and new vector fermions. The composition of Georgi-Machacek model includes two additional SU (2) triplet scalar with different hypercharge along with the SM Higgs doublet. Although the GM model preserves custodial symmetry and can generate neutrino mass in tree level, the model cannot account for a stable dark matter candidate. We show that, a new Z 4 symmetry is assigned to scalar and fermion fields can provide a feasible dark matter candidate upon breaking of the SU (2) × U (1) Y × Z 4 symmetry. The Z 4 symmetry is broken by the VEVs of doublet Higgs fields while triplet scalar fields do not generate any VEVs. Thus a residual Z 2 symmetry ensures the stability of neutral scalar originating from the mixing of scalar triplets which serves the purpose of dark matter resulting a scotogenic Georgi-Machacek (sGM) model where neutrino mass is generated in one-loop.
Moreover, the sGM model also exhibit feature of leptogenesis, capable of generating matter-antimatter asymmetry in the Universe from decaying vector fermions.
We put constraints on the model parameter space from various theoretical and experimental observations, such as vacuum stability, dark matter relic abundance, dark matter direct detection cross-section. Detailed study of dark matter phenomenology reveals that the model predicts a TeV scale dark matter candidate. In the present model, matter-antimatter asymmetry originates from CP violating decay of vector-like fermions. Using the limits from light neutrino mass, we found that massive vector fermions around 10 10−11 GeV can produce the experimentally observed baryon abundance. We have also observed that lepton flavor violation do not impose any significant constraint in the model parameter space. Therefore, a simple extension of Georgi-Machacek model with two Higgs doublet and heavy vector-like fermions accompanied by Z 4 can provide answers to the puzzles of dark matter, matter-antimatter asymmetry in a single framework with radiative generation of neutrino mass.