SU(5) grand unified theory with A4 modular symmetry

We present the first example of a grand unified theory (GUT) with a modular symmetry interpreted as a family symmetry. The theory is based on supersymmetric $SU(5)$ in 6d, where the two extra dimensions are compactified on a $T_2/\mathbb{Z}_2$ orbifold. We have shown that, if there is a finite modular symmetry, then it can only be $A_4$ with an (infinite) discrete choice of moduli, where we focus on $\tau = \omega=e^{i2\pi/3}$, the unique solution with $|\tau|=1$. The fields on the branes respect a generalised CP and flavour symmetry $A_4\ltimes \mathbb{Z}_2$ which is isomorphic to $S_4$ which leads to an effective $\mu-\tau$ reflection symmetry at low energies, implying maximal atmospheric mixing and maximal leptonic CP violation. We construct an explicit model along these lines with two triplet flavons in the bulk, whose vacuum alignments are determined by orbifold boundary conditions, analogous to those used for $SU(5)$ breaking with doublet-triplet splitting. There are two right-handed neutrinos on the branes whose Yukawa couplings are determined by modular weights. The charged lepton and down-type quarks have diagonal and hierarchical Yukawa matrices, with quark mixing due to a hierarchical up-quark Yukawa matrix.


I. INTRODUCTION
The flavor puzzle, the question of the origin of the three families of quarks and leptons together with their curious pattern of masses and mixings, remains one of the most important unresolved problems of the Standard Model (SM). Following the discovery of neutrino mass and mixing, whose origin is fundamentally unknown, there are now almost 30 undetermined parameters in the SM, far too many for any complete theory. The lepton sector in particular involves large mixing angles that suggest an explanation in terms of discrete non-Abelian family symmetry [1,2]. Furthermore, such discrete non-Abelian family symmetries have been combined with grand unified theories (GUTs) in order to provide a complete description of all quark and lepton (including neutrino) masses and mixings [3].
It is well known that orbifold GUTs in extra dimensions (ED) can provide an elegant explanation of GUT breaking and Higgs doublet-triplet spitting [4]. Similarly, theories involving GUTs and flavor symmetries have been formulated in ED [5][6][7][8][9][10][11][12]. These EDs can help us to understand the origin of the discrete non-Abelian group symmetry such as A 4 and S 4 which may be identified as a remnant symmetry of the extended Poincaré group after orbifolding. Some time ago it was suggested that modular symmetry, when interpreted as a family symmetry, might help us to provide a possible explanation for the neutrino mass matrices [13,14]. Recently it has been suggested that neutrino masses might be modular forms [15], with constraints on the Yukawa couplings. This has led to a revival of the idea that modular symmetries are symmetries of the extra-dimensional spacetime with Yukawa couplings determined by their modular weights [16]. However to date, no attempt has been made to combine this idea with orbifold GUTs in order to provide a unified framework for quark and lepton masses and mixings.
In this paper we present the first example in the literature of a GUT with a modular symmetry interpreted as a family symmetry. The theory is based on supersymmetric SUð5Þ in 6d, where the two extra dimensions are compactified on a T 2 =Z 2 orbifold, with a twist angle of ω ¼ e i2π=3 . Such constructions suggest an underlying modular A 4 symmetry with a discrete choice of moduli. This is one of the main differences of the present paper as compared to recent works with modular symmetries which regard the modulus τ as a free phenomenological parameter [15,16]. We construct a detailed model along these lines where the fields on the branes are assumed to respect a flavor and generalized CP symmetry A 4 ⋉ Z 2 which leads to an effective μ − τ reflection symmetry at low energies, implying maximal atmospheric mixing and maximal leptonic CP violation. The model introduces two triplet flavons in the bulk, whose vacuum alignments are determined by orbifold boundary conditions, analogous to those used for SUð5Þ breaking with doublet-triplet splitting. There are also two right-handed neutrinos on the branes whose Yukawa couplings are determined by modular weights. The charged lepton and down-type quarks have diagonal and hierarchical Yukawa matrices, with quark mixing due to a hierarchical up-quark Yukawa matrix.
The remainder of the paper is organized as follows. In Sec. II we discuss the orbifold T 2 =Z 2 and its symmetries, as follows. In Sec. II A, we give a review of modular transformations while in Sec. II B we describe how the orbifold T 2 =Z 2 is only consistent with modular A 4 symmetry and a choice of modulus. In Sec. II C, we explicitly show the orbifold T 2 =Z 2 with twist angle ω ¼ e i2π=3 and modular A 4 symmetry. In Sec. II D we study the remnant symmetry after compactification on the T 2 =Z 2 orbifold, while Sec. II E connects this remnant symmetry and the modular symmetry. In Sec. II F we discuss the enhanced A 4 ⋉ Z 2 on the branes. In Sec. III, we present the field content of the SUð5Þ GUT with A 4 modular symmetry and a Uð1Þ shaping symmetry, including the Yukawa sector and the specific structure for the effective alignments that the modular symmetry can generate, resulting in the low energy form of the SM fermion mass matrices which we show can lead to a very good fit to the observables. Finally in Sec. IV we present our conclusions. In order to make the paper self-contained, some necessary background information is included in the Appendixes. In Appendix A we show the explicit proof that only the A 4 modular symmetry is consistent with the branes, with specific choices of modulus. We supplement the general A 4 group theory in Appendix B, the consistency conditions for generalized CP symmetry consistent with A 4 in Appendix C and the general theory for modular forms in Appendix D. Finally we show sample fits of the observed data in Appendix E.

A. Review of modular transformations
In this subsection we present the general theory of modular transformations. The structure of the extra-dimensional torus is defined by the structure of the lattice by where ω 1;2 are the lattice basis vectors. The variable z refers to the complex coordinate z ¼ x 5 þ ix 6 , where x 5 and x 6 are the two extra-dimension coordinates. The torus is then characterized by the complex plane C modulo a two-dimensional lattice Λ ðω 1 ;ω 2 Þ , where Λ ðω 1 ;ω 2 Þ ¼ fmω 1 þ nω 2 ; m; n ∈ Zg, i.e., T 2 ¼ C=Λ ðω 1 ;ω 2 Þ . The lattice is left invariant under a change in lattice basis vectors described by the general transformations or equivalently if a; b; c; d ∈ Z and ad − bc ¼ 1. These are called modular transformations and form the modular group Γ [15]. Without loss of generality, the lattice vectors may be rescaled as such that the torus is equivalent to one whose periods are 1 and τ ¼ ω 2 =ω 1 and we can restrict τ to the upper halfplane H ¼ Imτ > 0. The modular transformations on the rescaled basis vectors which leave the lattice invariant are given by 1 A SLð2; ZÞ transformation on the modulus parameter τ and its negative are equivalent, as can be seen from Eqs. (2) and (4). Therefore, we can use the infinite discrete group PSLð2; ZÞ ¼ SLð2; ZÞ=Z 2 , generated by to describe the transformations that relate equivalent tori. This is also called the modular groupΓ satisfyinḡ Γ ¼ Γ=fAE1g. 2 The generators of the infinite-dimensional modular group can also be written as They satisfy the presentation where S; T ∈ SLð2; ZÞ. We will be considering the finite-dimensional discrete subgroups by imposing an additional constraint on T M , where M is a positive integer, where S; T ∈ SLð2; Z M Þ. These groups, with M ≤ 5, are isomorphic to the known discrete groups asΓ 2 ≃S 3 ,Γ 3 ≃A 4 , We now introduce a convenient (if nonunique) representation for the modular transformations consistent with the presentation in Eq. (8), which satisfies the presentation of theΓ M group, for any integer M > 2. This representation will be useful in the following discussion.
B. Why the orbifold T 2 =Z 2 suggests modular A 4 symmetry with modulus τ = ω In this subsection we present an argument which shows that a particular T 2 =Z 2 orbifold (as assumed in this paper) suggests an underlying modular A 4 symmetry with specific modulus parameters.
We begin by defining the orbifold T 2 =Z 2 in terms of two arbitrary lattice vectors ω 1 and ω 2 , The action of the orbifold in Eq. (10) leaves four invariant 4d branes given by 3 After compactification, the symmetries of the branes remain unbroken; therefore it is relevant to study any possible symmetry among the branes which will affect the fields localized on them. Therefore, we want to check if the modular transformations in Eq. (9) leave an invariant set of branes for some value of M. At this stage the modulus τ ¼ ω 2 =ω 1 can apparently take any value. However we present a proof in Appendix A that only the A 4 symmetry is consistent, meaning that M ¼ 3, when the basis vectors are related by The p and q are integers satisfying that is an integer, which has infinitely many discrete solutions. Furthermore, since the modular forms restrict τ to be in the upper complex plane, then q < 0. This paper's approach is to focus on the orbifold first and then derive the modular symmetries, instead of going directly into the modular symmetries. We will restrict ourselves to the case where jω 1 j ¼ jω 2 j, which happens when p ¼ −1, q ¼ −1. This way we focus on studying the effects only of the angle between both vectors. We can, without loss of generality, choose ω 1 ¼ 1. Furthermore, the modular symmetries require τ to lie in the upper complex plane; in this case the only solutions to Eqs. (12) and (13) are ω 2 ¼ ω ¼ e i2π=3 . This uniquely fixes the modulus coming from the orbifold T 2 =Z 2 . We emphasize that this is one of the main differences of the present paper as compared to recent works with modular symmetries which regard the modulus τ as a free phenomenological parameter [15,16]. In our work, we assume a specific orbifold T 2 =Z 2 , for which we have shown that one consistent choice for a surviving modular symmetry is A 4 with fixed modulus τ, although we shall not address the problem of moduli stabilization [17].
and modular A 4 symmetry Following the argument of the previous subsection, we henceforth focus on the orbifold T 2 =Z 2 with particular twist angle denoted as ω ¼ e i2π=3 , identified as the modulus τ associated with a particular finite modular symmetry A 4 , where A 4 is the only choice consistent with this orbifold.
This orbifold then corresponds to the identification where the first two equations are the periodic conditions from the torus T 2 and the third one is the action generated by the orbifolding symmetry Z 2 . The twist corresponds to ω ¼ e i2π=3 . The orbifold symmetry transformations leave four invariant 4d branes as shown in Fig. 1: The transformations permute the branes and leave invariant the set of four branes in Eq. (15). These transformations satisfy where the first line is the presentation of the group A 4 and both lines complete the presentation of S 4 [1]. In Fig. 1 we show how these transformations act on the extra-dimensional space and how the "remnant A 4 symmetry" is realized. Fixing M ¼ 3, the set of branes is invariant under the modular transformations on the lattice vectors ð1; ωÞ T . These transform the basis vectors as (noting that 1 þ ω þ ω 2 ¼ 0) leaving the lattice invariant as can be seen from Fig. 2. The matrices S; T ð3Þ fulfill the presentation of the group they generate to be where S; T ð3Þ ∈ SLð2; Z 3 Þ, so that the branes are indeed invariant under the discrete modular groupΓ 3 ≃ A 4 . As we will see in Sec. II F, this symmetry will be enlarged.
D. Remnant brane symmetry for T 2 =Z 2 with ω = e i2π=3 So far we have shown that the choice of orbifold T 2 =Z 2 is consistent with the finite modular symmetry A 4 with a discrete choice of moduli, where we focus on τ ¼ ω ¼ e i2π=3 . Now we will take a step back, forget about modular symmetries for a while, and just consider the symmetries of the branes with a twist angle ω ¼ e i2π=3 . We will discover an S 4 symmetry that has apparently nothing to do with modular symmetry, which we refer to as "remnant S 4 symmetry." In the next subsection we shall show how the subgroup remnant A 4 symmetry is related to the previous A 4 finite modular symmetry.
In this section, we will find that the branes are invariant under an S 4 and its subgroup A 4 symmetry which can be identified as a remnant symmetry of the spacetime symmetry after it is broken down to the 4d Poincaré symmetry through orbifold compactification. Here, we assume that the spacetime symmetry before compactification is a 6d Poincaré symmetry. The compactification breaks part of this symmetry. However, due to the geometry of our orbifold with twist angle ω ¼ e i2π=3 , a discrete subgroup is left unbroken. This group may be generated by the spacetime transformations (which belong to the extradimensional part of the 6d Poincaré). The orbifolding leaves four invariant branes, and this specific orbifold structure leaves them related by the group S 4 . This symmetry, together with 4d Poincaré transformations, is a subgroup of the extra-dimensional Poincaré symmetry that survives compactification. This is the standard "remnant symmetry" [6,18].
Any field located in the branes will transform under the 4d Poincaré group as usual. Since the branes transform into each other by the remnant symmetries, the fields on the brane should also transform under them. The four branes transform under the remnant S 4 symmetry and we choose the embedding of the representation 4 → 3 þ 1 so that the fields in the branes can only transform under those irreducible representations [7,11]. The fields in the branes do not depend on z but are permuted into each other by the S, T, U transformations.

E. The connection between remnant A 4 symmetry and finite modular A 4 symmetry
We have shown that the set of branes is invariant under a remnant S 4 (and its subgroup A 4 ) subgroup of the extradimensional Poincaré symmetry. We shall return to the S 4 symmetry in the next subsection and we now show that the remnant A 4 symmetry can be identified with the finite modular A 4 symmetry discussed earlier. Essentially, if we impose a modular symmetry A 4 on the whole space, its action on the branes is the same action as the remnant spacetime symmetry; i.e., it permutes the branes but leaves invariant the whole set. The modular symmetry acts on the basis vectors of the torus while the remnant symmetry is a spacetime symmetry and acts on the fixed points; therefore from the point of view of the branes, the "remnant A 4 symmetry" is an active transformation while the finite modular A 4 symmetry is an equivalent passive transformation. This way we may identify the remnant A 4 symmetry of the branes as a modular symmetry since the effect on the branes of each type of transformation is identical; it is just a choice of "picture" (active or passive) which we choose.
The action of the modular symmetry on the branes behaves as a "normal" symmetry (i.e., modular forms are not relevant) since fields located on the brane do not depend on the extra-dimensional coordinate. The modular symmetry can therefore be imposed as any usual symmetry. In the orbifold T 2 =Z 2 , the branes are only consistent with the modular groupΓ 3 , as shown in Appendix A. Any theory with this orbifold and fields allocated on the branes will only be consistent with the modular symmetryΓ 3 . In such a setup, the branes see the finite modular symmetry as simply equivalent to a remnant symmetry, a subgroup of the extradimensional Poincaré group.
We can see in Eq. (18) that the S, T transformations (and therefore theΓ 3 modular transformations) correspond to specific passive reflections, rotations and translations. In this way theΓ 3 must be a subgroup of the 6d Poincaré group. All modular groups are. However not all modular groups are consistent with the invariant branes, as we have shown.
On the other hand, fields in the bulk, which feel the extra dimensions, will also transform under some representation of the 6d Poincaré; however in this case they will transform under a nonlinear realization of thisΓ 3 symmetry, and this is precisely what are referred to as the modular forms [15].
We conclude that the modular symmetryΓ 3 acts either as a linear or nonlinear realization of the remnant symmetry A 4 , depending on whether we are concerned with brane fields or bulk fields.

F. Enhanced A 4 ⋉ Z 2 symmetry of the branes
We now recall that, in our setup, the brane fields enjoy a larger S 4 symmetry than the remnant A 4 symmetry, as shown in Sec. II D. However this larger S 4 symmetry is not related to a finite modular symmetry, since the branes can only be invariant under the modular transformations corresponding toΓ 3 ≃ A 4 , and is not enjoyed by the fields in the bulk. SUð5Þ GRAND UNIFIED THEORY WITH A 4 MODULAR … PHYS. REV. D 101, 015028 (2020) We note here that S 4 ≃ A 4 ⋉ Z 2 . The symmetry generated by U from Eq. (16) is a remnant symmetry of the orbifolding process, but it cannot be interpreted as a modular transformation. We conclude that the remnant symmetry of the branes isΓ 3 ⋉ Z 2 ≃ A 4 ⋉ Z 2 . The Z 2 symmetry is generated by C · U where U is the usual matrix representation of the generator from S 4 and C stands for complex conjugation of the complex coordinate, which is equivalent to a change of sign in x 6 , i.e., the parity transformation of the sixth dimension P 6 . The Z 2 is not a modular symmetry while the A 4 is. The product of both symmetries is not direct since the generator U does not commute with all A 4 generators and is the corresponding S 4 generator.
After compactification, the remnant Z 2 (which is not a subset of A 4 ) acts on the brane fields generalized CP transformation where the transformations P 1 ; …; P 5 are trivial whileP 6 ¼ P 6 U, where P 6 is the trivial parity transformation, while the U is a family transformation [19]; thus a field transforms as Under CP the fields transform as shown in Appendix C. As stated before, this effective symmetry transformation only affects nontrivially the brane fields and the fields in the bulk are unaffected, transforming under canonical CP and not forced to preserve it. Thus in our approach the generalized CP is a remnant symmetry in a particular sector of the theory, corresponding to the fields on the branes. We have shown that the remnant Z 2 symmetry on the branes behaves as an effective generalized CP transformation. In Appendix C we check its compatibility with the A 4 flavor symmetry and find that it is consistent, as indeed it must be.

A. The model
In this section we construct a supersymmetric SUð5Þ GUT model on a 6d orbifold T 2 =Z 2 with twist ω ¼ e i2π=3 , with an A 4 modular symmetry as a flavor symmetry, extended by the Z 2 symmetry on the branes. Furthermore we impose a global Uð1Þ as a shaping symmetry. We impose different boundary conditions at each invariant brane, which break the original symmetry into the minimal supersymmetric standard model (MSSM). The Uð1Þ as a shaping symmetry forbids any higher order terms, while a discrete Z N would allow them, the smaller the N, the corrections would appear at lower order.
The A 4 modular symmetry will require the Yukawa couplings to be specific modular forms, while the Z 2 symmetry will further restrict the possible mass matrix structure so that the theory has strong predictions for leptons [20]. As we shall see later, the up quarks will lie in different A 4 singlets with modular weight zero, so that only the subgroup Z 3 is a remnant while the Z 2 behaves trivially. This forces stringent relations for the lepton mass matrices but not for the quarks.
All the fields in the bulk ψ will transform under the modular transformations where ρ is the usual matrix representation of the corresponding A 4 transformation. Each field has a weight −k, with no constraint in k since the fields are not modular forms. The superfields that are located on the brane do not depend on the extra dimensions and therefore they must have weight zero [15]. We arbitrarily choose a weight for each of the bulk fields. The whole field content is listed in Tables I and II. The fields that do not have weight or parity under the boundary conditions are located on the branes and feel the symmetry A 4 ⋉ Z 2 ; see Table I. The transformations of the fields   TABLE I. Fields on the branes, including matter and righthanded neutrino superfields. A working set of charges is fq 1 ; q 2 ; q 3 g ¼ f2; 0; 1g. Note that the 3 representations on the brane transform under A 4 ⋉ Z 2 as shown in Table VI and Eq. (C4).

Representation Localization
under this symmetry are discussed in Appendix B. The 3 representations on the brane transform under A 4 ⋉ Z 2 are as shown in Table VI and Eq. (C4). The field F contains the MSSM fields L and d R and is a flavor triplet. It is located on the brane. The fields T AE i contain the MSSM u R , e R , Q; they are three flavor singlets. There are two copies of each T with different parities under the boundary conditions; as we shall see in the next section, this allows different masses for down quarks and charged leptons. There are only two right-handed neutrinos N c a;s . The MSSM Higgs fields h u;d are inside the H 5;5 respectively. We have two flavons ϕ 1;2 that help to give structure to the fermion masses. Finally, the field ξ generates the hierarchy between the massesà la Froggat-Nielsen [21].

B. GUT and flavor breaking by orbifolding
Since the orbifold has the symmetry transformations of Eq. (14), the fields must also comply with them. However since we are in a gauge theory, the equations need not be fulfilled exactly but only up to a gauge transformation, so any field complies with where the G's are gauge transformations that must fulfill where the first equation comes from the fact that it belongs to the parity operator, the second is due to the fact of the commutativity of the translations and the third one denotes the relation between parity and translations.
Since the branesz i are invariant under the orbifold symmetry transformations of Eq. (14), they act as boundaries which, due to the G's gauge transformations, impose the boundary conditions which correspond to a reflection at each of the branes. These boundary conditions are related to the gauge transformations as For simplicity, we choose all G's to commute, meaning that G 5;6 ¼ G −1 5;6 , and therefore all boundary conditions become matrices of order 2. The boundary conditions imply an invariance at each brane under some A 4 × SUð5Þ transformation, and they are chosen to break the symmetry in a particular way as follows: where I 3 ; T 1;2 ∈ SUð3Þ, while I 5 ; diagð−1; −1; −1; 1; 1Þ ∈ SUð5Þ, and explicitly and the last boundary condition is fixed and defined by the other boundary conditions as The boundary condition P 0 breaks the effective extended N ¼ 2 → N ¼ 1 SUSY. The boundary conditions P 1=2;ω=2 leave their corresponding Z 2 symmetry invariant and together break A 4 completely and SUð5Þ → SUð3Þ × SUð2Þ × Uð1Þ.
The fields F; N c a;s ; ξ lie on the brane and are unaffected by the boundary conditions. The fields T AE are A 4 singlets and do not feel the A 4 breaking conditions. They have different parities and feel the SUð5Þ breaking condition. The fields T þ contain the light MSSM u R , e R fields, while T − contains the light field Q. This allows for independent masses for charged leptons and down quarks since they come from different fields. The Higgs fields feel the SUð5Þ breaking condition leaving only the light doublets, solving the doublet triplet splitting problem [7] (for a recent discussion see e.g., [11]). The flavons ϕ 1;2 feel the A 4 breaking conditions. They have different parities under the conditions and this fixes their alignments to be We may remark that these flavon vacuum expectation value (VEV) alignments do not break the Z 2 symmetry generated by U, even though they are in the bulk. We see that the orbifolding breaks the symmetry SUð5Þ × A 4 ⋉ Z 2 → SUð3Þ × SUð2Þ × Uð1Þ × Z 2 while solving the doublet triplet splitting, separating charged lepton and down-quark masses and completely aligning flavon VEVs.
We do not show an explicit driving mechanism for the VEVs v 1;2;ξ . We assume that they are driven radiatively [22].

C. Yukawa structure
In 6d, the superpotential has dimension 5 while each superfield has dimension 2. A 6d interacting superpotential SUð5Þ GRAND UNIFIED THEORY WITH A 4 MODULAR … PHYS. REV. D 101, 015028 (2020) is inherently nonrenormalizable. We work with the effective 4d superpotential, which happens after compactification. We assume the compactification scale is close to the original cutoff scale. We use Λ to denote both the compactification scale and the GUT scale, which is taken to be the cutoff of the effective theory. Assuming this makes the Kaluza-Klein modes to be at the GUT scale so that they do not spoil standard gauge coupling unification or any of the current precision tests. With the fields in Tables I and II, we can write the effective 4d Yukawa terms where i, j ¼ 1, 2, 3. Due to the stringent Uð1Þ shaping symmetry, there are no higher order terms. The field ξ has a VEV and generates hierarchies between familiesà la Froggat-Nielsen [21]. The first line in Eq. (30) gives the two right-handed (RH) neutrino Majorana masses without any mixing. The fields in both terms have zero weight so the modular symmetry does not add anything new. The second line generates Dirac neutrino masses. They have nontrivial weights and their structure will be discussed in Sec. III D. The third line gives masses to charged leptons. They are all weight zero automatically and the mass matrix is diagonal. The fourth line generates a diagonal down-quark mass matrix. Since it involves a different field (T − instead of T þ ) the coupling constants are independent. Finally the fifth line gives masses to the up quarks. It is a general nonsymmetric mass matrix with complex entries. Since the fields in these terms have a nontrivial weight but the T AE are singlets, the modular symmetry does not change the matrix structure. We remark that the top-quark mass term is renormalizable.
At the GUT level, the μ term is forbidden, so it should be generated by another mechanism at a much smaller scale [23].

D. Effective alignments from modular forms
In Eq. (30) we have a few terms involving nontrivial weights under the modular symmetry. This implies that the couplings y ν s ; y ν a ; y u ij ð31Þ are modular forms with a positive even weight [24]. They involve the Dedekind η function and its exact form can be found in Appendix D.
The modular forms are functions of the lattice basis vector parameter τ from Eq. (3). Usually this parameter is chosen to give a good fit to the flavor parameters. In our case, the specific orbifold of our model is set to fix and the modular form structure is fixed up to a real constant due to the extra condition coming from the generalized CP symmetry.
The modular form y ν s must be a triplet under A 4 to construct an invariant singlet with the triplet field F. Furthermore, it has weight α to compensate the overall weight of the corresponding term. We show the effective triplet alignments it can have in Table III for different weights α. The possibilities are very limited since many modular forms vanish when τ ¼ ω, as shown in Appendix D. Larger weight modular forms repeat the same structure so that this table is exhaustive, as discussed in Appendix D.
The modular form y ν a must have weight β. It multiplies the flavon ϕ 2 , so that they must be contracted into a triplet ðy ν a hϕ 2 iÞ 3 which will generate the effective alignment. In the case of y ν a being a singlet under A 4 , the effective alignment is simply given the flavon VEV hϕ 2 i in Eq. (29), which was fixed by the orbifold boundary conditions. When y ν a is a triplet under A 4 , it must be contracted with ϕ 2 as shown in Appendix B, 3 × 3 → 1 þ 1 0 þ 1 00 þ 3 a þ 3 s . This gives different possible products for the effective triplet. The actual effective alignment is an arbitrary linear combination of all possibilities and can be found in Table IV. For β ¼ 0 the only modular form is a singlet, so the only triplet that can be built is hϕ 2 i. For β ¼ 2, the only modular form is the triplet Y ð2Þ 3 shown in the Appendix D. The effective triplet is the linear combination TABLE III. The effective alignments of the modular form y ν s as a triplet, depending on its weight α. The parameter y is an arbitrary real constant to comply with the extended symmetry 3;2 , so that the actual alignment comes from the linear combination of hϕ 2 i × Y 1;1 0 → 3 and hϕ 2 i × Y 3 → 3 a þ 3 s .
By choosing the weights α, β, the structure of the neutrino mass matrix is completely defined. In principle, the y in Table III and y 1 , y 2 , y 3 in Table IV correspond to general complex numbers; however as we will see below they are constrained to comply with the nontrivial CP symmetry of the model. We have obtained all the possible A 4 invariant modular forms. However we have to comply with the extended symmetry A 4 ⋉ Z 2 . The U generator only transforms nontrivially the triplet field F which is contracted to a triplet modular form. A U transformation of the field F can be reabsorbed by transforming the modular form by where the C stands for complex conjugation. Invariant terms under the full symmetry must involve modular forms that are also invariant under the Z 2 transformation. From Table III, the only invariant case is when α ¼ 6 with a real y. From Table IV, the only invariant cases happen when β ¼ 0 with real y 1 or β ¼ 6 with y 1;2 real and y 3 imaginary. The triplet field F is not only taking part in the Dirac neutrino mass terms but also in the down-quark and charged lepton mass terms; therefore they also must be invariant under the enhanced symmetry A 4 ⋉ Z 2 . In this case, the field F is contracted with the flavon field ϕ 1 and it is easy to check that the transformation in Eq. (33) leaves the VEV invariant when real and therefore the charged lepton and down-quark mass terms when the parameters y d i and y e i involved are real. Finally, the modular form y u ij must have weight α þ 2γ to build an invariant. All the fields in the corresponding terms are singlets, so these modular forms must be singlets also and will not change the structure. Depending on i, j, the modular form y u ij must be a different type of singlet. The weight α þ 2γ has to be large enough so that the space contains the three types of singlets. This modular form does not add anything to the structure of the up-quark matrix but allows us to build the A 4 invariants for all T i T j combinations. The smallest weight that allows modular forms of all three types of singlets is 20, as discussed in Appendix D. These modular forms y u ij are in general complex. The case β ¼ 0 does not have enough freedom to fit the neutrino data. We conclude that the smallest phenomenologically viable choice for weights is

E. Mass matrix structure
We are now able to express the mass matrices following Eq. (30) and the effective alignments given in Sec. III D. First, we define the dimensionless parameters where Λ is the original cutoff scale. The down-quark and charged lepton mass matrices are diagonal: where the parameters y d i and y e i are real due to the enhanced symmetry on the branes A 4 ⋉ Z 2 while y u ij are in general complex.
The down-quark and charged lepton mass matrices in Eq. (36) are diagonal so the fit to the observed masses is straightforward. The hierarchy between the masses of the IV. The effective alignments of the modular form y ν a contracted with hϕ 2 i into a triplet, depending on its weight β. The parameters y i are dimensionless constants constrained by the A 4 ⋉ Z 2 symmetry. β ðy ν a hϕ 2 iÞ 3 =v 2 0 (2020) 015028-9 different families is understood through the powers ofξ and can be achieved assuming the dimensionless couplings to be of order Oð1Þ. All the contributions to quark mixing come from the up sector. The complex parameters in the up-type mass matrix [see Eq. (37)] fix the up, charm and top quark masses as well as the observed Cabibbo-Kobayashi-Maskawa (CKM) mixing angles. We can obtain a perfect fit for weight γ ¼ 7. Different values ofṽ 1 ,ṽ 2 and ξ can fit the observed masses using different dimensionless couplings still of order Oð1Þ. We show an example in Appendix E. The form of the Dirac neutrino mass matrix depends on the weights α and β. All the possible alignments are given in Tables III and IV. The Z 2 symmetry restricts ourselves to the case α ¼ 6 and β ¼ 0 or β ¼ 6. In the case of β ¼ 0, we only have two free parameters fy; y 1 g and we cannot find a good fit with the solar and the reactor angle being too small. Therefore, the only phenomenologically viable case is for α ¼ β ¼ 6 and we restrict ourselves to this case in the following.
As shown in Appendix B, we have to take into account the Clebsch-Gordan coefficients when contracting the modular form ðy ν s FÞ 1 and ðy ν a hϕ 2 iFÞ 1 into singlets, i.e., 3 × 3 → 1, given by after which the effective alignments for α ¼ 6 and β ¼ 6 look like respectively. The Dirac neutrino mass matrix is then given by The RH neutrino Majorana mass matrix is diagonal: with hierarchical RH neutrino masses given by the different powers of the field ξ. Furthermore, we have very heavy RH neutrino Majorana masses such that the left-handed neutrinos get a very small Majorana mass through type I seesaw [25]: The neutrino mass matrix looks like where α 6 and β 6 are the alignments defined in Eq. (39). The effective parameters at low energy are fy; y 1 ; y 2 ; y 3 g, previously defined in Tables III and IV. The Z 2 symmetry fixes the parameters fy; y 1 ; y 2 g to be real while y 3 is purely imaginary.
Finally we remark that this structure, with the expected hierarchy between the RH neutrinos, can give the correct baryon asymmetry of the Universe (BAU) through leptogenesis naturally. Leptogenesis is achieved through the CP violation in the neutrino Dirac mass matrix. The correct order of the BAU happens when the RH neutrino masses are M 1 ∼ 10 10 GeV and M 2 ∼ 10 13 GeV [26]. In this model, these are the natural expected masses as we can see from Eq. (41) and the sample fit in Appendix E. The contributions from the entries of the neutrino Dirac mass matrix and the expected BAU will fix the precise value of M 1 . We conclude that the CP violation in the neutrino sector and the RH neutrino mass hierarchy of the model assures us that the BAU can be generated naturally [11].

F. μ − τ reflection symmetry
The neutrino mass matrix in Eq. (43) is μ − τ reflection symmetric (μτ-R symmetric). This corresponds to the interchange symmetry between the muon neutrino ν μ and the tau neutrino ν τ combined with CP symmetry, namely where the star superscript denotes the charge conjugation of the neutrino field. This can easily be seen from the alignments in Eq. (39) which construct the neutrino mass matrix in Eq. (43). Since the parameters fy; y 1 ; y 2 g are real while y 3 is purely imaginary, the transformation in Eq. (44) leaves the alignments invariant and accordingly the neutrino mass matrix. For a review of μτ symmetry see e.g., [27] and references therein; also see the recent discussion [28]. It is known that having a neutrino mass matrix μτ-R symmetric in the flavor basis (which is our case) is equivalent to μ − τ universal (μτ-U) mixing in the Pontecorvo-Maki-Nakagawa-Sakata (PMNS) matrix; see Ref. [29]. The consequence of having μ − τ symmetry is that it leads to having a nonzero reactor angle, θ 13 , together with a maximal atmospheric mixing angle and maximal Dirac CP phase: We remark that this is a prediction of the model, due to having A 4 ⋉ Z 2 symmetry on the branes. The parameters fy; y 1 ; y 2 ; y 3 g in the neutrino mass matrix (43) will fit the rest of the PMNS observables, namely fθ l 12 ; θ l 13 ; Δm 2 21 ; Δm 2 31 g, together with the prediction of the The contribution to a χ 2 test function comes only from these predictions and we use the recent global fit values of neutrino data from NuFit4.0 [30]. The best fit points together with the 1σ ranges are θ 23 = ∘ ¼ 49.6 þ1.0 −1.2 and δ l = ∘ ¼ 215 þ40 −29 for normal mass ordering and without the Super-Kamiokande atmospheric neutrino data analysis. However, the distributions of these two observables are far from Gaussian and the predictions of having maximal atmospheric mixing angle θ 23 ¼ 45°and maximal CP violation δ l ¼ −90°still lie inside the 3σð4σÞ region with a χ 2 ¼ 5.48 (6.81) without (with) Super-Kamiokande. Appendix E explains how a numerical fit can be performed and Table V shows two numerical fits, although this is only an example as we can find a good fit for a large range of parameters y; y 1 ; y 2 and y 3 . 4 This is because the predictions of the model θ 23 ¼ 45°and δ l ¼ −90°are due to the μτ-R symmetry and the four free parameters are used to fit the rest of the observables in the PMNS matrix.
The best fit from NuFit4.0 is for normal mass ordering with inverted ordering being disfavored with a Δχ 2 ¼ 4.7 ð9.3Þ without (with) the Super-Kamiokande atmospheric neutrino data analysis. We tried a fit to inverted mass ordering and the χ 2 test function goes up to χ 2 ∼ 6800. Therefore, the model predicts normal mass ordering together with maximal atmospheric mixing and CP violation and a massless neutrino m 1 ¼ 0 since we are only adding two RH neutrinos.

IV. CONCLUSIONS
In this paper we have presented the first example in the literature of a GUTwith a modular symmetry interpreted as a family symmetry. The theory is based on supersymmetric SUð5Þ in 6d, where the two extra dimensions are compactified on a T 2 =Z 2 orbifold. We have shown that, if there is a finite modular symmetry, then it can only be A 4 . Furthermore, if we restrict ourselves to the case in which jω 1 j ¼ jω 2 j, the only possible value of the modulus parameter is τ ¼ ω ¼ e i2π=3 . We emphasize that this is one of the essential distinctions of the present model in contrast to recent works with modular symmetries, which regard the modulus τ as a free phenomenological parameter [15,16]. In the present paper, we assume a specific orbifold structure which fixes the modulus to a discrete choice of moduli, where we focus on the case τ ¼ ω ¼ e i2π=3 , although we do not address the problem of moduli stabilization.
We have shown that it is possible to construct a consistent model along these lines, which successfully combines an SUð5Þ GUT group with the A 4 modular symmetry and a Uð1Þ shaping symmetry. In this model the F fields on the branes are assumed to respect an enhanced symmetry A 4 ⋉ Z 2 which leads to an effective μ − τ reflection symmetry at low energies, which predicts the maximal atmospheric angle and maximal CP phase. In addition there are two right-handed neutrinos on the branes whose Yukawa couplings are determined by modular weights, leading to specific alignments which fix the Dirac mass matrix. The model also introduces two triplet flavons in the bulk, whose vacuum alignments are determined by orbifold boundary conditions, analogous to those responsible for Higgs doublet-triplet splitting. The charged lepton and down-type quarks have diagonal and hierarchical Yukawa matrices, with quark mixing due to a hierarchical up-quark Yukawa matrix.
The resulting model, summarized in Tables I and II, provides an economical and successful description of quark and lepton (including neutrino) masses and mixing angles and CP phases. Indeed the quarks can be fit perfectly, consistently with SUð5Þ, using only Oð1Þ parameters. In addition we obtain a very good fit for the lepton observables with χ 2 ≈ 5 ð7Þ without (with) Super-Kamiokande data, using four Oð1Þ parameters which determine the entire lepton mixing matrix U PMNS and the light neutrino masses (eight observables), which implies that the theory is quite predictive. The main predictions of the model are a normal neutrino mass hierarchy with a massless neutrino, and the μ − τ reflection symmetry predictions θ l 23 ¼ 45°a nd CP phase δ l ¼ −90°, which will be tested soon.

APPENDIX A: UNIQUENESS OF A 4 AS A MODULAR SYMMETRY FOR THE BRANES
In this Appendix, we show that the set of branes in Eq. (11) is invariant under the modular transformationsΓ 3 for an infinite set of discrete values of the modulus parameter τ.
First, we apply the finite modular transformations in Eq. (9) on the lattice vectors Therefore, the S-transformed branes becomē Using the orbifold transformations from Eq. (10), we can add ω 1 to the second and fourth branes, and obtain the original set in Eq. (11). Therefore the brane set is always invariant under the S transformation, for any value of ω 1 and ω 2 .
On the other hand, the T-transformed branes arē If the set is to be invariant, up to permutations of the branes, the second term in Eq. (A3) must correspond to one of the original branes.
Let us make the Ansatz that the second brane from Eq. (A3) corresponds to the fourth original brane ðω 1 þ ω 2 Þ=2 up to orbifold transformations, so that it must satisfy where p, q are general integer numbers and represent the general orbifold transformations. This relates the basis vectors ω 1 and ω 2 as and we may rewrite the transformed branes from Eq. (A3) in terms of ω 1 : We now make the Ansatz that the third brane corresponds to the third original brane ω 2 =2 (and automatically the fourth corresponds to the original second one), so that it must satisfy where r, s are general integer numbers which represent the general orbifold transformations. After some simple manipulations this can be written as The left-hand part is complex while the right-hand side is real. To cancel the imaginary part we must have so that 2ð2p þ 1Þ cosð2π=MÞ where, since the right-hand side is an integer, the left-hand side must also be which forces and the equation becomes which is a Diophantine equation to be solved for r. We have made two Ansätze to obtain this equation. This is the only solution since making any other Ansatz would obtain an equation without solutions (the equation would be an odd number equal to zero). These are straightforward calculations, which are done in the same way as this one and seem repetitive to show. The invariance condition on the branes fixes them to be: where the choice M ¼ 3, 6 only changes with m. Since only ω 2 is physical (and not the specific integers p, q), we can reabsorb the m dependence into p. As we stated before we will be studying discrete modular symmetries with M ≤ 5 so that the only solution is M ¼ 3 which fixes the relation of the branes to be where the p, q are integers that satisfy that is an integer, which has infinitely many discrete solutions.

APPENDIX B: GROUP THEORY
A 4 is the even permutation group of four objects, which is isomorphic to the symmetry group of a regular tetrahedron. It has 12 elements that can be generated by two generators, S and T, with the presentation A 4 has four inequivalent irreducible representations: three singlet 1; 1 0 ; 1 00 and one triplet 3 representations. We choose to work with the same complex basis as [15] and the representation matrices of the generators are shown in Table VI. The product of two triplets, φ ¼ ðφ 1 ; φ 2 ; φ 3 Þ and ψ ¼ ðψ 1 ; ψ 2 ; ψ 3 Þ, decomposes as 3×3 ¼ 1þ1 0 þ1 00 þ3 s þ3 a , where 3 s;a denote the symmetric or antisymmetric product.
The component decompositions of the products are shown in Table VII.
The 12 elements of A 4 are obtained as 1; S; T; ST; TS; T 2 ; ST 2 ; STS; TST; T 2 S; TST 2 and T 2 ST. The A 4 elements belong to four conjugacy classes: where mC k n refers to the Schoenflies notation where m is the number of elements of rotations by an angle 2πk=n.

APPENDIX C: GENERALIZED CP CONSISTENCY CONDITIONS FOR A 4
Here, we check the compatibility of the Z 2 symmetry on the branes with the A 4 flavor symmetry. The remnant Z 2 symmetry behaves as an effective generalized CP transformation and the fields on the branes will transform under Z 2 as where x 0 ¼ ðt; x 1 ; x 2 ; x 3 ; x 5 ; −x 6 Þ and X r is the representation matrix in the irreducible representation r. To combine the flavor symmetry A 4 with the Z 2 symmetry, the transformations have to satisfy certain consistency conditions [31], which were specifically applied to A 4 flavor symmetry in [20]. These conditions assure that if we perform a Z 2 transformation, then apply a family symmetry transformation, and finally an inverse Z 2 transformation is followed, the resulting net transformation should be equivalent to a family symmetry transformation. It is sufficient to only impose the consistency conditions on the group generators:   D 101, 015028 (2020) where ρ r denotes the representation matrix for the generators S and T; see Table VI. As shown in [20], S 0 and T 0 can only belong to certain conjugacy classes of A 4 , [see Eq. (B2) to find out the elements in each conjugacy class]. The transformations under the generalized CP symmetry Z 2 are then which are consistent with Eqs. (C2) and (C3) for S 0 ¼ S and T 0 ¼ T. However in the model under consideration, we do not have any field on the branes transforming under the 1 0 and 1 00 representation. Thus the Z 2 transformation only affects the 3 representations. We conclude that the 3 representations on the brane transform under A 4 ⋉ Z 2 as shown in Table VI and Eq. (C4).

APPENDIX D: MODULAR FORMS
In this section, we show the construction of modular forms forΓ 3 ≃ A 4 following [15].
In model building, the difference between the usual cited discrete symmetries, which arise as remnant symmetries of the branes, and the modular symmetries of the spacetime lattice is that, in the latter case, the fields transform, under a transformation of τ, as where ρ is the usual matrix representation of the transformation and k is called the weight and is an arbitrary number. The invariance of the action forces the usual dimensionless coupling in the superpotential y to behave as [32] y → ðcτ þ dÞ k y ρ y y; where k y is the weight and must be an even integer [24] and ρ y the usual matrix representation of the transformation. To build the invariant in global supersymmetry, we need to satisfy two conditions: first the weight k y has to cancel the overall weights of the fields and second the product of ρ y times the representation matrices of the fields has to contain an invariant singlet. When k ¼ 0 for every constant, we have the usual discrete symmetry. The weight 0 form is just a constant, singlet under A 4 . The first nontrivial modular form is of weight 2 that following Eq. (4) transforms as The weight 6 modular forms are written as Due to relations of the Dedekind functions, the modular forms satisfy which reduce the number of possible modular forms. In our case which reduces even further the possible modular forms. The only triplet that is different from zero in Eq.
All modular forms are built from products of the weight 2 triplet. We can build the modular forms for weight 8. Following [15], this is a 15-dimensional space that must be decomposed as 2 × 1 þ 2 × 1 0 þ 2 × 1 00 þ 3 × 3. For simplicity we can work out only the specific case where τ ¼ ω. This case is greatly restricted and can be checked by doing all possible multiplications of 3 × 3 × 3 × 3 that the only nonzero modular forms are where we can see that the triplet has the same structure as the weight 2 one. From this we conclude that any higher weight triplet would only repeat the previous structures without having any new one. For weight 10 we would have the same triplet as in weight 4 but two singlets since we can have the nontrivial products so that this is the first space that has two singlets. The next space that has the three singlets is built from powers of these singlets, so the modular form must have weight 20.

APPENDIX E: NUMERICAL FIT
We perform a χ 2 test function when fitting the effective neutrino mass matrix in Eq. (43) with input parameters x ¼ y; y 1 ; y 2 ; y 3 , from which we obtain a set of observables P n ðxÞ. We minimize the function defined as where the observables are given by P obs n ∈ fθ l 12 ; θ l 13 ; θ l 23 ; δ l ; Δm 2 21 ; Δm 2 31 g with statistical errors σ n . We use the recent global fit values of neutrino data from NuFit4.0 [30] and we ignore any renormalization group running corrections as well as threshold corrections associated with the two extra dimensions. Most of the observables follow an almost Gaussian distribution and we take a conservative approach using the smaller of the given uncertainties in our computations except for θ l 23 and δ l . The best fit from NuFit4.0 is for normal mass ordering with inverted ordering being disfavored with a Δχ 2 ¼ 4.7 ð9.3Þ without (with) the Super-Kamiokande atmospheric neutrino data analysis. We tried a fit to inverted mass ordering and we found a χ 2 ∼ 6800; therefore in the following results we only focus on the case of normal mass ordering.
The model predictions are shown in Table VIII. The neutrino mass matrix in Eq. (43) predicts the maximal atmospheric mixing angle, θ l 23 ¼ 45°, and maximal CP violation, δ l ¼ −90°, within the 3σ region from the latest neutrino oscillation data. This is a consequence of the μτ-R symmetric form of the neutrino mass matrix when y; y 1 ; y 2 TABLE VIII. Model predictions in the neutrino sector for weights α ¼ β ¼ 6. The neutrino masses m i as well as the Majorana phases are pure predictions of our model. We also predict the maximal atmospheric mixing angle θ l 23 ¼ 45°and maximal CP phase δ l ¼ 270°. The bound on P m i is taken from [34]. The bound on m ee is taken from [35]. There is only one physical Majorana phase since m 1 ¼ 0.

Data
Model Central value 1σ range α ¼ β ¼ 6 are real while y 3 is imaginary. Furthermore, since we only have 2RH neutrinos, m 1 ¼ 0 and there is only one physical Majorana phase α 23 [33]. The predicted effective Majorana mass m ee [33] is also given in Table VIII. The fit has been performed using the Mixing Parameter Tools (MPT) package [36]. The values of y; y 1 ; y 2 and y 3 are shown in Table V. Fit 1 shows a good fit where all of the dimensionless real parameters y are of Oð1Þ. However a large range of parameters can give an equally good fit; see for example fit 2. The VEV ratios jξ;ṽ i j are parameters that do not enter the fit directly and they are chosen to reproduce the hierarchy between the fermion Yukawa couplings, making the dimensionless couplings more natural numbers, i.e., Oð1Þ. In the case of the neutrino mass matrix, even for fixed jξj and jṽ 2 j, there is a large range of parameters y; y 1 ; y 2 and y 3 that can give a good fit to the observables, meaning that the modular forms for weight α ¼ 6 and β ¼ 6 give a constrained form of the neutrino mass matrix which is phenomenologically suitable. For comparison, we also give the value of the χ 2 test function in the case of β ¼ 0, in which we only have two free parameters y and y 1 , and it goes up to χ 2 ∼ 1500, while for β ¼ 6 with four free parameters we have found a perfect fit for a variety of values of y; y 1 ; y 2 and y 3 .
These VEV ratios jξ;ṽ i j also appear in the quark and charged-lepton mass matrices in Eqs. (36) and (37). For different values of jξj, as in fits 1 and 2 in Table V, different dimensionless Oð1Þ parameters y d i ; y e i and y u ij can be used to give the correct mass of the down-and up-type quarks and charged leptons and we show an example in Table IX for fit 1. In this case we take into account the running of the MSSM Yukawa couplings to the GUT scale and we follow the parametrization done by [37]. The matching conditions at the SUSY scale are parametrized in terms of four parametersη q;b;l which we set to zero and the usual tan β for which we choose tan β ¼ 5. IX. Input parameters that enter in the charged fermion mass matrices in Eqs. (36) and (37), giving the correct charged fermion masses and CKM parameters for a choice ofξ corresponding to fit 1. The Yukawa parameters y u ij are in general complex; however most of the phases can be reabsorbed and we are left with only four physical phases ϕ 21 , ϕ 23 , ϕ 31 and ϕ 32 where the subscript refers to the entry in the mass matrix where we are adding the phase.