New Conditions on Maximal Invariant Subgroups That Imply Solubility

Let G be a finite group and assume that a finite group of automorphisms A acts on G, such that the orders of A and G are relatively prime. We prove that the fact of imposing certain conditions on the set of maximal A-invariant subgroups of G, relating to nilpotency, p-nilpotency, normality or having p’-order, determines properties on the structure of G such as solubility, p-solubility or p-nilpotency.


Introduction
Suppose that a finite group G is acted on by a group of automorphisms A, where the orders of G and A are coprime. In recent years many results on the structure of G have been obtained when only the invariant structure of G is taken into account. For instance, the authors proved that the relevant and classical result of Schmidt about the structure of minimal non-nilpotent groups admits a version in the coprime action context [1, Theorem A]. The same occurs, for every prime p, with the characterization of minimal non-pnilpotent groups due to N. Itô [4,Satz IV.5.4], which was extended by H. Meng and A. Ballester-Bolinches in [8,Theorem A], as well as the renowned Frobenius criterion of p-nilpotency [6, 7.2.4], generalized by M.Y. Kizmaz in [5,Theorem B]. We note that the proofs of all these extensions are based on the Classification of the Finite Simple Groups.
The research of other conditions on maximal invariant subgroups has continued up to now. The authors demonstrated in [1,Theorem B] that if a group G has a nilpotent maximal A-invariant subgroup of odd order, then G is soluble, thus providing an extension of a well-known theorem of Thompson, and likewise, attained a coprime action variant of Deskins' solubility criterion [2]. The fact of assuming specific conditions on the indices of maximal invariant subgroups or on the set of second maximal invariant subgroups has also lead to obtaining properties on G (see [10,12]).
Our aim in this paper is to explore new conditions that when being imposed to a special subset or to the whole set of maximal invariant subgroups of a group G determine certain properties on the structure of G, such as solubility, p-solubility or p-nilpotency. We remark that most of the aforementioned results will be utilized to prove our main results.
Theorem A. Suppose that a group A acts coprimely on a group G. If every maximal A-invariant subgroup of G is nilpotent or normal in G, then G is soluble. Furthermore, G is p-nilpotent for some prime p dividing the order of G.
Theorem B. Suppose that a group A acts coprimely on a group G and let p be a prime divisor of the order of G. If every maximal A-invariant subgroup of G whose order is divisible by p is nilpotent, then G is soluble.
We point out that Theorem A generalizes, under the coprime action hypothesis and using a completely different approach, the main result of [7]. Further, we employ the above two theorems to derive a widespread result.
Theorem C. Suppose that a group A acts coprimely on a group G and let p be a prime divisor of the order of G. If every maximal A-invariant subgroup of G is either nilpotent or normal or has p -order, then G is soluble.
We note that if the action of A on G is not coprime, then our theorems are simply not true. For instance, take a subgroup H ∼ = Alt(4) of the alternating group Alt(5) acting by conjugation on Alt (5). The only maximal H-invariant subgroup of Alt (5) is H itself, however, it is not nilpotent nor normal and has order divisible by 2 (and by 3).
We have also examined what happens if we weaken the nilpotency assumption to p-nilpotency in Theorems A, B and C, and we have got local variants of Theorems A and C, but only for odd primes. This is done by employing Glauberman and Thompson's p-nilpotency criterion [4, Satz IV.6.2].

Theorem D. Suppose that a group A acts coprimely on a group G and let p be an odd prime. If every maximal
Theorem B, on the contrary, does admit a local version for every prime and this can be obtained as an straightforward consequence of [8, Theorem A]. We will see, however, that Theorem D is false for p = 2 by means of a counterexample (Examples 4.4 and 4.5).
All groups are supposed to be finite and the notation is standard as in [6].

Preliminaries
Regarding coprime action, we refer the reader to [6,Chapter 8] for a detailed presentation and basic properties. Nevertheless we present here those properties that we more frequently use. Let A be a group that acts on a group G by automorphisms such that (| A |, | G |) = 1. For every prime p, it is well known that there exists an A-invariant Sylow p-subgroup in G and that any two A-invariant Sylow psubgroups of G are conjugate by an element of the fixed point subgroup C G (A) [6, 8.2 If π is any set of primes, the above properties also hold for A-invariant π-subgroups and Ainvariant Hall π-subgroups when G is π-separable, so in particular, when is soluble [6, 8.2.6]. Of course, every A-invariant subgroup of G always lies in some maximal A-invariant subgroup of G.
Among the less known results on maximal invariant subgroups, we start with an elementary property.
Lemma 2.1 [11,Lemma 2.3]. Suppose that a finite group A acts coprimely on a finite group G. If every maximal A-invariant subgroup of G is normal, then G is nilpotent.
For proving our main results, we also need the following theorems quoted in the Introduction. The first extends Schmidt characterization of minimal nonnilpotent groups, and the second transfers Frobenius' normal p-complement theorem, both into the coprime action context. We want to mention again that their proofs are based on the Classification of the Finite Simple Groups.

Theorem 2.2 [1, Theorem A].
Let G and A be finite groups of coprime orders and assume that A acts on G by automorphisms. If every maximal A-invariant subgroup of G is nilpotent but G is not, then G is soluble and | G |= p a q b for two distinct primes p and q, and G has an A-invariant normal Sylow subgroup. Theorem 2.3 [5,Theorem B]. Let G be a group and A be a group acting on G by automorphisms such that (| A |,| G |) = 1. Assume that for each nontrivial The authors proved in [1,Lemma 5.3] that if a group G is assumed to be perfect, that is, G = G , and has trivial Fitting subgroup, then every nilpotent maximal invariant subgroup of G must be a Sylow 2-subgroup of G. This was proved by an application of the well-known p-nilpotency criterion of Glauberman and Thompson, quoted in the Introduction. For our purposes, however, this is not enough but we need to extend [1, Lemma 5.3] to a larger class of non-soluble groups. We do it by following a different method, concretely in the vein of [9, Theorem 1].

Theorem 2.4. Suppose that G and A are finite groups such that A acts coprimely on G and that
Proof. Suppose first that the soluble radical of G, S(G), is trivial. The maximality of M implies that the normalizer in G of every nontrivial A-invariant Sylow subgroup of M , which is A-invariant too, is exactly M . We claim further that M is a Hall subgroup of G. Indeed, if p is a prime dividing both | M | and | G : M |, then the normalizer in G of any A-invariant Sylow p-subgroup of G cannot be contained in M . But this forces M to be normal in G, contradicting the fact that S(G) = 1. Now, by [1, Theorem B], we know that M cannot have odd order, so M possesses a nontrivial (and unique) A-invariant Sylow 2subgroup, say P , and we take U to be the A-invariant complement of P in M .
, the Schur-Zassenhaus theorem implies that N K (Q) has complements in N G (Q) and that all of them are N G (Q)-conjugate. We can apply Glauberman's Lemma (for instance [6, Theorem 6.6.2]) to ensure that there exists at least an A-invariant complement of N K (Q) in N G (Q), say M * , which moreover is isomorphic with M . In particular, M * is a Hall-subgroup of G with the same order as M . Now, by a Theorem of Wielandt [4, Satz III.5.8], all these Hall subgroups are conjugate in G, and again by an application of Glauberman's Lemma we obtain that M * and M are conjugate by some element in the fixed point subgroup C G (A). In particular, this allows to conclude that M * is maximal A-invariant of G as well. Thus, M * , Q is an A-invariant subgroup normalizing Q, and the maximality forces M * , Q = G, in other words, Q G. This contradicts the fact that S(G) = 1. Therefore, U = 1, or equivalently, M is a Sylow 2-subgroup of G, as wanted.
Suppose now that S := S(G) = 1, and as above take P and U to be the A-invariant Sylow 2-subgroup and the A-invariant 2-complement of M , respectively. As G is non-soluble, SM < G, and then S ≤ M . Also, by the above paragraph, M/S is a Sylow 2-subgroup of G/S and then U ≤ S ≤ M . It follows that U is the unique 2-complement of S and this implies that U G.
Remark 2.5. We would like to comment that Meng and Ballester-Bolinches ([8, Corollary 2]) provided a Classification-free proof of Theorem 2.2. We highlight that now it is feasible and easy to give a new Classification-free proof of it. It suffices to argue by induction on the order of G and employ Theorem 2.4, which does not depend on the Classification either, to get the solubility of G.

Proofs of Theorems A, B and C
We are ready to prove our main results. We start by stating again the first one.
Theorem A. Suppose that a group A acts coprimely on a group G. If every Proof. For proving the solubility of G, we argue by minimal counterexample and take G to be a counterexample of minimal order. Suppose first that every maximal A-invariant subgroup of G is not nilpotent. Then, by hypothesis, such subgroups are normal. This leads to the solubility of G by Lemma 2.1, which is a contradiction. Henceforth, we can assume that G has nilpotent maximal invariant subgroups. On the other hand, G must possess some non-nilpotent maximal A-invariant subgroup, which must be normal in G by hypothesis, otherwise Lemma 2.2 would imply the solubility of G. As a result, we conclude that G also possesses at least a nontrivial proper A-invariant normal subgroup. Thus, we can take N < G a minimal A-invariant normal subgroup of G. As the hypotheses are inherited by quotients of A-invariant normal subgroups, by minimality we get that G/N is soluble, and consequently, N cannot be soluble. By using this reasoning we can prove that G has a unique minimal A-invariant normal subgroup, because the existence of such two subgroups leads to the solubility of G. This argument shows further that G does not have soluble A-invariant normal subgroups distinct from 1, so in particular Z(G) = 1. Thus, by Theorem 2.4, we obtain that every nilpotent maximal A-invariant subgroup of G is a Sylow 2-subgroup of G.
Take now p = 2 a prime divisor of | N | and let P be an A-invariant Sylow p-subgroup of N . By the Frattini argument, G = N G (P )N , and observe that N G (P ) < G, since G does not have A-invariant normal p-subgroups. If we take M to be a maximal A-invariant subgroup of G containing N G (P ), it is clear that M cannot be nilpotent since p divides | M |. However, M cannot be normal in G either, because if M were normal in G, then N ≤ M and hence G = N N G (P ) ≤ M . This contradiction proves the solubility part.
Next we prove the p-nilpotency of G for some prime p dividing its order by induction on | G |. We take N to be a minimal A-invariant normal subgroup of G, which by solubility of G, is a p-subgroup for some prime p. Since the hypotheses are inherited by quotients of G by A-invariant normal subgroups, we have that G/N is q-nilpotent for some prime q. Of course, we can assume q = p, otherwise G would be q-nilpotent too, so the theorem would be proved. Thus, let K/N be the (A-invariant) normal p-complement of G/N . By Schur-Zassenhaus Theorem and Glauberman's Lemma, we can write K = K 0 N , where K 0 is an A-invariant p-complement of K (and of G). Notice that K 0 cannot be normal in G, otherwise we are finished, and moreover, as K G, the Frattini argument yields G = N G (K 0 )K = N G (K 0 )N . Now, note that 149 Page 6 of 11 A. Beltrán and C. Shao Results Math N G (K 0 ) cannot be contained in any proper A-invariant normal subgroup of G, so by hypothesis N G (K 0 ) must lie in some nilpotent maximal A-invariant subgroup of G. In particular, N G (K 0 ) is nilpotent too. Then we choose q = p a prime divisor of the order of N G (K 0 ) and take H the A-invariant qcomplement of N G (K 0 ), which is normal in N G (K 0 ) by nilpotency. Then HN N G (K 0 )N = G, and we conclude that G possesses normal q-complement, HN , as wanted.
We prove now Theorem B, which we state again in a distinct manner for our convenience. Proof. We argue by counterexample of minimal order, so we fix a group G of minimal order and a prime p dividing | G | such that every maximal A-invariant subgroup of G is nilpotent or has p -order. It is clear that G has maximal Ainvariant subgroups of order divisible by p (it suffices to consider a maximal A-invariant subgroup that contains an A-invariant Sylow p-subgroup), and these subgroups must be nilpotent by our assumptions. Also, since G is nonsoluble, by Lemma 2.2, G must have a non-nilpotent maximal A-invariant subgroup, which must have p -order by our assumptions.
Suppose first that N = O p (G) = 1 and consider the (non-soluble) factor group G/N . If N is not a Sylow p-subgroup of G, then G/N has order divisible by p and satisfies the hypothesis, so by minimality G/N has a non-nilpotent maximal A-invariant subgroup M/N of order divisible by p. Then the existence of M , a non-nilpotent maximal A-invariant subgroup of G, gives a contradiction. Therefore, we can assume that N is a Sylow p-subgroup of G and take M to be a non-nilpotent maximal A-invariant subgroup of p -order of G, as indicated in the first paragraph. It follows that G = MN and M ∩ N = 1. Notice that every maximal A-invariant subgroup of M , say M 0 , produces a maximal A-invariant subgroup M 0 N of G, which moreover has order divisible by p. Hence it is nilpotent. In particular, we deduce that M 0 is nilpotent, and then we apply Lemma 2.2 to get that M is soluble. But this fact contradicts the non-solubility of G.
Henceforth, we assume that O p (G) = 1. This is equivalent to claim that every nontrivial A-invariant p-subgroup U of G is not normal in G, or equivalently N G (U ) < G. It is evident that every maximal A-invariant subgroup of G containing N G (U ) has order divisible by p, and thus, it is nilpotent. This implies, in particular, that N G (U ) is nilpotent too. Then we apply Theorem 2.3 to get that G is p-nilpotent. Therefore, there exists K G, which is also Ainvariant, such that G = KP and K ∩P = 1, where P is any A-invariant Sylow p-subgroup of G. Observe that, by Feit-Thompson Theorem, we can assume that p = 2. We distinguish two cases depending on whether the center of G is trivial or not. Assume first that Z(G) = 1. Then by Theorem 2.4, every nilpotent maximal A-invariant subgroup of G is a Sylow 2-subgroup, so P , which has odd order, cannot be contained in any nilpotent maximal A-invariant subgroup of G. This contradicts our assumption on G. Suppose now that Z(G) = 1. As we are assuming O p (G) = 1, it is obvious that p does divide the order of G/Z(G), which is non-soluble. By minimality, G/Z(G) must have a non-nilpotent maximal A-invariant subgroup of order divisible by p, and this easily leads to a contradiction. This final contradiction completes the proof.
From Theorem 3.1, it immediately follows our solubility criterion, Theorem B, as stated in the Introduction. We will see in Example 4.5 that the hypothesis on divisibility by p cannot be replaced by the non-divisibility by p. By employing Theorems A and B, we can prove now our third main result.

Theorem C. Suppose that a group A acts coprimely on a group G and let p be a prime divisor of the order of G. If every maximal A-invariant subgroup of G is either nilpotent or normal or has p -order, then G is soluble.
Proof. We argue by minimal counterexample and suppose that G is a nonsoluble group of minimal order satisfying the hypotheses. We know by Theorem B that G must possess non-nilpotent maximal A-invariant subgroups of order divisible by p, and then such subgroups must be normal in G by hypothesis. In particular, G has nontrivial and proper A-invariant normal subgroups.
We claim that the soluble radical of G, S(G), is trivial. We prove it by showing that every minimal A-invariant normal subgroup of G is non-soluble. Let N be a minimal A-invariant normal subgroup of G and suppose first that G/N has p -order, so in particular, | N | is divisible by p. If every maximal A-invariant subgroup of G/N were nilpotent, then G/N would be soluble by Theorem 2.2, and thus, N would not be soluble and we are finished. We can assume then that G/N has non-nilpotent maximal A-invariant subgroups. If M/N is such a subgroup, then certainly M is a non-nilpotent maximal Ainvariant subgroup of G with p dividing its order, so by hypothesis M G. Thus, we can apply Theorem A and conclude that G/N is soluble, and consequently, N is not, as wanted. If, on the contrary, p divides | G/N |, then G/N clearly satisfies the hypotheses of the theorem, so by minimality G/N must be soluble, and N cannot be soluble either. This proves the claim. In particular, we have Z(G) = 1. Therefore, we can apply Theorem 2.4 to get that every nilpotent maximal A-invariant subgroup of G is a Sylow 2-subgroup of G. Now assume that p = 2 and let P be an A-invariant Sylow p-subgroup of G. Since P is not normal in G because S(G) = 1, we can take M a maximal A-invariant subgroup of G containing N G (P ). Note that M is not normal in G, has order divisible by p, and moreover, is not nilpotent because it is not a Sylow 2-subgroup of G. This contradicts our assumptions on G, and henceforth, we assume that p = 2.
Let R be a maximal A-invariant subgroup of G containing the normalizer of some A-invariant Sylow 2-subgroup of G. Notice that R is not normal in G, since S(G) = 1 implies that this normalizer is proper in G, so the hypothesis forces that R is nilpotent, and then R necessarily is a Sylow 2-subgroup of G. Take N a (non-soluble) minimal A-invariant normal subgroup of G. By maximality of R, it is clear that G = NR. On the other hand, we take q = 2 a prime divisor of | N | and Q an A-invariant Sylow q-subgroup of N (and also of G). As Q is not normal in G, we consider a maximal A-invariant subgroup of G, say H, with N G (Q) ≤ H. Observe that H is not nilpotent and neither normal in G, so by hypothesis, H has odd order. Hence N G (Q) has odd order too. Now, the Frattini argument gives G = N G (Q)N , and thus 1 = G/N ∼ = N G (Q)/N N (Q) has odd order as well. But we have proved above that G/N = RN/N ∼ = R/R ∩ N is a 2-group, so this contradiction completes the proof.

Local Versions of Theorems A, B and C
In this section we give local versions of our main results as far as possible. As commented in the Introduction, in [8] it is proved that, for a fixed prime p and under the coprime action hypothesis, if every maximal A-invariant subgroup of a group G is p-nilpotent, then G is p-soluble. The Classification is only required for proving the case p = 2. Thus, from this result, we easily get a p-nilpotent version of our Theorem B.

Corollary 4.1. Suppose that a group
A acts coprimely on a group G and let p be a prime divisor of the order of G. If every maximal A-invariant subgroup of G whose order is divisible by p is p-nilpotent, then G is p-soluble.
Proof. Since every maximal A-invariant subgroup of G whose order is not divisible by p is p-nilpotent as well, the corollary follows immediately by [8, Theorem A].
We have been able to extend simultaneously [8, Theorem A] and Theorem A for odd primes.

Theorem D. Suppose that a group A acts coprimely on a group G and let p be an odd prime. If every maximal
Proof. We argue by minimal counterexample. The hypotheses are clearly inherited by invariant factors of G, so it is easy to see that we can assume that G has exactly one minimal A-invariant normal subgroup, say N , which is nonsoluble (possibly N = G), and that G/N is p-soluble by minimality. Let P be an A-invariant Sylow p-subgroup of G, so P 0 = P ∩ N is an A-invariant Sylow p-subgroup of N . Note that P 0 = 1, otherwise G would be p-soluble. Now, let J(P 0 ) = 1 be the Thompson subgroup of P 0 , which is clearly Ainvariant too. Then Z(J(P 0 )) is an A-invariant subgroup of G which is not normal in G. Accordingly, N G (Z(J(P 0 ))) is a proper A-invariant subgroup of G, which must lie in some maximal A-invariant subgroup, M , of G. We claim that M cannot be normal in G. Indeed if M G, then the uniqueness of N implies that N ≤ M . On the other hand, since Z(J(P 0 )) is characteristic in P 0 , then N G (P 0 ) ≤ N G (Z(J(P 0 ))), and this leads to N N G (P 0 ) ≤ M . But the Frattini argument yields N N G (P 0 ) = G, so we get a contradiction. Therefore, as M is not normal in G, then M is p-nilpotent, and in particular, so is N N (Z(J(P 0 ))). By Glauberman-Thompson's p-nilpotency criterion [4, Satz IV.6.2], we conclude that N is p-nilpotent, and hence, G would be p-soluble. This contradiction finishes the proof.
In case when the action is trivial, we deduce the following consequence extending Itô's theorem on minimal non-p-nilpotent groups, of which we are not aware that has already been published.

Corollary 4.2.
Let G be a group and p an odd prime. If every maximal subgroup of G is p-nilpotent or normal in G, then G is p-soluble.
Since groups of p -order are trivially p-nilpotent, a trivial consequence of Theorem D is the following p-nilpotent version of Theorem C for odd primes.   (6), which is normal of index 2, and the normalizer of every Sylow qsubgroup for q = 2, 3, or 5. These normalizers are respectively isomorphic to: the dihedral group of order 20 when q = 5; (C 3 × C 3 ) Q 8 when q = 3; and the semidihedral group of order 16 when q = 2 (see [3]). All of them are 2-nilpotent, whereas M 10 is certainly not 2-soluble.
Example 4.5. The above example serves to show that Theorem D, and accordingly Corollary 4.3, are untrue for p = 2, even when the action of A on G is nontrivial. Take again K = M 10 and let H be a non-abelian split metacyclic extension of order pq, where p and q are two distinct primes such that p, q ∈ {2, 3, 5}, so we have (| K |, | H |) = 1. Write H = P Q, where P H, | P |= p and | Q |= q. Set G = K × P and let us consider the coprime action of Q on G by acting trivially on K and by the corresponding action on P . Now we take into account the following elementary property: the subgroups of any direct product of two groups of coprime order, like G = K × P , are exactly the direct products of the form K 1 × P 1 , where K 1 and P 1 are subgroups of K and P , respectively. Observe then that the set of Q-invariant subgroups of G consists of the following subgroups: K 1 × 1 and K 1 × P , where K 1 is any subgroup of K (which is trivially Q-invariant). It follows that the maximal Q-invariant subgroups of G are exactly those of the form K 0 × P , where K 0 149 Page 10 of 11 A. Beltrán and C. Shao Results Math is any maximal subgroup of K, and K × 1. Having in mind the set of maximal subgroups of K, given in Example 4.4, we obtain that every maximal Q-invariant subgroup of G is either normal (these are exactly Alt(6) × P and K × 1) or is equal to K 0 × P , where K 0 is a non-normal maximal subgroup of K. Since K 0 is 2-nilpotent, then K 0 × P is 2-nilpotent as well. This example also shows that it is not possible to exchange in Theorem B the hypothesis concerning maximal invariant subgroups of "being divisible by p", by "not being divisible by p", because the non-soluble group G satisfies that every maximal Q-invariant subgroup of odd order of G (this is the empty set) is nilpotent. Data Avaliability This manuscript has no associated data.

Declarations
Conflict of interest A. Beltrán and C.G. Shao declare they have no financial interests.
Open Access. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons. org/licenses/by/4.0/.