On finite self-complementary metacirculants

Abstract

We prove that the automorphism group of a self-complementary metacirculant is either soluble or has $\mathrm{A}_5$ as the only insoluble composition factor, extending a result of Li and Praeger which says the automorphism group of a self-complementary circulant is soluble. The proof involves a construction of self-complementary metacirculants which are Cayley graphs and have insoluble automorphism groups. To the best of our knowledge, these are the first examples of self-complementary graphs with this property.

1 Introduction

A graph $\varGamma =(V,E)$ consists of a vertex set $V$ and an edge set $E$ which is a subset of the set $\{\{u,v\}\mid u,v\in V, u\ne v\}$. The complement $\overline{\varGamma }$ of a graph $\varGamma$ is the graph with the same vertex set $V$ such that $\{u,v\}$ is an edge of $\overline{\varGamma }$ if and only if $u\ne v$, and $\{u,v\}$ is not an edge of $\varGamma$. A graph is said to be self-complementary if its complement is isomorphic to itself.

A graph $\varGamma =(V,E)$ is called a metacirculant if $\mathsf{Aut}\varGamma$ contains a metacyclic subgroup $R$ which is transitive on $V$; refer to [16] for the definition which is slightly more general than the original definition of Alspach and Parsons in [2]. (Recall that a group $R$ is called metacyclic if $R$ has a normal subgroup $N$ such that both $N$ and $R/N$ are cyclic.) To emphasise the metacyclic group, we sometimes call $\varGamma$ a metacirculant of R.

In this paper, we study metacirculants that are self-complementary, which will be called self-complementary metacirculants.

A special subclass of metacirculants consists of circulants, whose automorphism group contains a transitive cyclic subgroup. Self-complementary circulants have been studied for a long time and have been taken as good models for studying other combinatorial objects, such as Ramsey numbers and communication networks. The first family of self-complementary circulants was constructed by Sachs in 1962, and since then self-complementary circulants have been widely studied, see [6, 19, 22, 24, 25, 28] for the work till 1980s. In 1990s, the orders of self-complementary circulants were determined, see [1] for Alspach-Morris-Vilfred’s proof and [7] for an alternative approach. Some special families of self-complementary circulants are constructed in [11, 18, 21]. More recently, it was proved that the automorphism groups of self-complementary circulants are soluble by Praeger and the first author in [14], and the structure of self-complementary circulants of prime-power order is characterised in [17].

On the other hand, self-complementary metacirculants form a subclass of self-complementary vertex-transitive graphs. One of the motivations of studying self-complementary vertex-transitive graphs is to investigate Ramsey numbers. For a positive integer $m$, we wish to construct a graph $\varGamma$ with vertex set $V$ such that neither $\varGamma$ nor $\overline{\varGamma }$ contains a complete subgraph $\mathsf{K}_m$ and $|V|$ is as big as possible. By intuition, the distribution of pairs of elements of $V$ between $\varGamma$ and $\overline{\varGamma }$ should be “balanced” so that $\varGamma$ and $\overline{\varGamma }$ should not be very different. In the extreme case, $\varGamma \cong \overline{\varGamma }$. Furthermore, since $\mathsf{K}_m\not <\varGamma$ is a global property of $\varGamma$, the edges of $\varGamma$ should be distributed “homogeneously”. In the ideal case, $\varGamma$ is vertex-transitive. Because of this, self-complementary vertex-transitive graphs have been effectively used as models to find good lower bounds of Ramsey numbers (see [4, 5, 9, 23] for references).

The study of self-complementary vertex-transitive graphs has a long history, refer to the excellent survey of Beezer [3]. In particular, the orders of general self-complementary vertex-transitive graphs have been completely determined by Muzychuck [20]. Since 2000, the study of self-complementary vertex-transitive graphs has been advanced by the work in [10, 13]. The self-complementary vertex-transitive graphs whose orders are products of two primes are classified by the authors in [15].

Recall that for a group $G$ and subgroups $K\lhd H\leqslant G$, the quotient group $H/K$ is called a section of $G$. The main result of this paper gives a characterisation of the automorphism groups of self-complementary metacirculants.

Theorem 1.1

Let $\varGamma$ be a self-complementary metacirculant of $R$. Then either $\mathsf{Aut}\varGamma$ is soluble or the following hold:

• (i) the only insoluble composition factor of $\mathsf{Aut}\varGamma$ is the alternating group $\mathrm{A}_5$ and

• (ii) $\mathsf{Aut}\varGamma$ has a section that is of the form $\mathbb {Z}_p^2{:}(\mathbb {Z}_\ell \circ \mathrm{SL}(2,5))$ such that $\mathbb {Z}_p^2$ is a section of $R$, where $p$ is a prime number and $p\equiv 1~\text{ or }~9~(\mathsf{mod~}40)$, and $\ell$ divides $p-1$ and is divisible by $4$.

Remark

Although $\mathrm{A}_5$ is the only insoluble composition factor of $\mathsf{Aut}\varGamma$, this factor can appear multiple times, see Lemma 3.5.

In Sect. 3, we construct examples of self-complementary metacirculants that are Cayley graphs and have insoluble automorphism groups. Moreover, as a result of Lemma 3.3, we have the following.

Theorem 1.2

There exist self-complementary metacirculants of which the automorphism groups are insoluble.

Praeger and the first author in [14] proved that the automorphism group of a self-complementary circulant is soluble. A metacirculant $\varGamma$ is called a Sylow-circulant if $\mathsf{Aut}\varGamma$ has a transitive metacyclic subgroup $R$ of which all Sylow subgroups are cyclic. In this case, $R$ has a normal Hall subgroup $N$; in particular, the orders of $N$ and $R/N$ are coprime. By Theorem 1.1 (ii), if $\mathsf{Aut}\varGamma$ is insoluble, then there exists a prime $p$ such that a Sylow $p$-subgroup of $R$ is not cyclic. Therefore, a consequence of Theorem 1.1 is the following corollary, which extends the result for circulants.

Corollary 1.3

If $\varGamma$ is a self-complementary Sylow-circulant, then $\mathsf{Aut}\varGamma$ is soluble. In particular, if $\varGamma$ is a self-complementary metacirculant of square-free order, then $\mathsf{Aut}\varGamma$ is soluble.

We end this section with a conjecture.

Conjecture

Self-complementary metacirculants are Cayley graphs.

We remark that a metacirculant is generally not necessarily a Cayley graph; for example, the Petersen graph is a metacirculant but not a Cayley graph.

2 Preliminaries

In this section, we collect some preliminary results that will be used.

Let $\varGamma =(V,E)$ be a self-complementary graph. Let $\sigma \in \mathrm{Sym}(V)$ be an isomorphism from $\varGamma$ to $\overline{\varGamma }$, called a complementing isomorphism of $\varGamma$. Then $\sigma$ interchanges $\varGamma$ and $\overline{\varGamma }$, and $\sigma$ has order $o(\sigma )=2^eb$ for $e\geqslant 1$ , where $b$ is an odd number. Observe that $\sigma =\sigma ^b\cdot \sigma ^{1-b}$, where $\sigma ^b$ is a complementing isomorphism of $2$-power order and $\sigma ^{1-b}\in \mathsf{Aut}\varGamma$. We thus sometimes assume a complementing isomorphism $\sigma$ to be of $2$-power order. If $o(\sigma )=2$, then $\sigma$ interchanges some vertices $u,v$ and fixes $\{u,v\}$, which is not possible. So $4$ divides $o(\sigma )$.

Lemma 2.1

A complementing isomorphism of a self-complementary graph has order divisible by $4$.

For a regular self-complementary graph of order $n$, a vertex has an equal number of neighbours and non-neighbours, and hence $n$ must be an odd number. Furthermore, the graph has precisely half number of edges of the complete graph, and so $n(n-1)/4$ is an integer, and $n\equiv 1~(\mathsf{mod~}4)$.

Lemma 2.2

The order of a regular self-complementary graph is congruent to $1$ (modulo 4).

We next introduce a classical method for constructing self-complementary graphs.

For a finite group $R$, let $R^\#:=R{\setminus }\{1\}$, the set of all non-identity elements of $R$. For a subset $S\subseteq R^\#$ with $S=S^{-1}:=\{s^{-1}\mid s\in S\}$, the associated Cayley graph $\mathsf{Cay}(R,S)$ is the graph with vertex set $V=R$ and edge set $E=\{\{a,b\}\mid a,b\in R, ba^{-1}\in S\}$.

By the definition, the Cayley graph $\mathsf{Cay}(R,R^\#{\setminus } S)$ is the complement of $\mathsf{Cay}(R,S)$. Observe that any automorphism $\sigma \in \mathsf{Aut}(R)$ induces an isomorphism from $\mathsf{Cay}(R,S)$ to $\mathsf{Cay}(R,S^\sigma )$. Thus, if there exist $S\subseteq R^\#$ and $\sigma \in \mathsf{Aut}(R)$ such that

$$\begin{aligned} S^\sigma =R^\#{\setminus } S, \end{aligned}$$

(*)

then $\mathsf{Cay}(R,S)\cong \mathsf{Cay}(R,S^\sigma )=\mathsf{Cay}(R,R^\#{\setminus } S)$ is self-complementary. Since $S$ and $R^\#{\setminus } S$ are disjoint, $\sigma$ fixes no non-identity element of $R$, that is, $\sigma$ is a fixed-point-free automorphism of $R$. Moreover, since $S=S^{-1}$, and $\sigma$ cannot interchange any pair of vertices, we further have that $\sigma ^2$ is fixed-point-free, and $g^{-1}\in g^{\langle \sigma ^2\rangle }$ for any $g\in R$.

Conversely, if a group $R$ has an automorphism $\sigma$ of order a power of 2 such that $\sigma ^2$ is fixed-point-free, then the following construction indeed produces self-complementary graphs, see [12].

Construction 2.3

Let $R$ be a group and let $\sigma \in \mathsf{Aut}(R)$ of order a power of $2$ such that $\sigma ^2$ is fixed-point-free. Then every orbit of $\langle \sigma \rangle$ on $R^\#$ has length divisible by 4 and is divided into two parts by $\sigma ^2$. We define a subset $S$ and a Cayley graph $\mathsf{Cay}(R,S)$ as follows:

• (1) Let $\Delta _1, \Delta _2,\dots ,\Delta _r$ be the $\langle \sigma \rangle$-orbits on $R^\#$, and label the two orbits of $\langle \sigma ^2\rangle$ on $\Delta _i$ as $\Delta _i^+$ and $\Delta _i^-$, where $1\leqslant i\leqslant r$.

• (2) Set $S=\bigcup _{i=1}^r\Delta _i^{\varepsilon _i}$, where $\varepsilon _i=+$ or $-$. (We remark that there are $2^r$ different choices for such a subset $S$.)

Another construction method of self-complementary vertex-transitive graphs is based on the lexicographic product (sometimes called the wreath product). Let $\varGamma _1=(V_1,E_1)$ and $\varGamma _2=(V_2,E_2)$ be graphs. The lexicographic product of $\varGamma _2$ by $\varGamma _1$ is defined as the graph with vertex set $V=V_1\times V_2$ such that $(u_1,v_1)$ is adjacent to $(u_2,v_2)$ if and only if either $u_1$ is adjacent to $u_2$ in $\varGamma _1$ or $u_1=u_2$ and $v_1$ is adjacent to $v_2$. This graph is denoted by $\varGamma _1[\varGamma _2]$.

The lexicographic product has the following nice properties (see [3]):

Lemma 2.4

If both $\varGamma _1,\varGamma _2$ are vertex-transitive or self-complementary, then so is $\varGamma _1[\varGamma _2]$.

We will also need to know about the order of the maximal $p$-elements in $\mathrm{GL}(d,p)$.

Lemma 2.5

If $d\geqslant 2$, then the largest order $p^e$ of $p$-elements of $\mathrm{GL}(d,p)$ satisfies $p^e\geqslant d>p^{e-1}$.

Proof

Let $U$ be the set of the elements of $\mathrm{GL}(d,p)$ of the following form:

$$\begin{aligned} \left( \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 1 &{} a_{12} &{} a_{13} &{} \cdots &{} a_{1d}\\ 0 &{} 1 &{} a_{23} &{} &{} a_{2d}\\ \vdots &{} 0 &{} \ddots &{} \ddots &{} \vdots \\ \vdots &{} \vdots &{} \ddots &{} \ddots &{} a_{d-1,d}\\ 0 &{} 0 &{} \cdots &{} 0 &{} 1 \end{array}\right) . \end{aligned}$$

Then $U$ forms a Sylow $p$-subgroup of $\mathrm{GL}(d,p)$. Let $u\in U$ such that $o(u)=p^e$ is the largest order of elements of $U$, where $e$ is an integer. Let $u=A+I$. Notice that for any $u\in U$, we have $u^{p^e}=I$ if and only if $(A+I)^{p^e}=I$, if and only if (using the binomial theorem) $A^{p^e}=0$. By careful choice of $u$, we can ensure that $A^b\not =0$ for $b<d$, so that $A^{p^e}=0$ implies $p^e\geqslant d$ and $p^{e-1}<d$. $\square$

3 Examples with insoluble automorphism groups

We construct examples of self-complementary metacirculants which have insoluble automorphism groups.

Let $R=\mathbb {Z}_p^2$ with $p$ prime. Then $\mathsf{Aut}(R)=\mathrm{GL}(2,p)$. Assume that $p\equiv \pm 1$ $(\mathsf{mod~}10)$. Then $\mathrm{GL}(2,p)$ contains an irreducible subgroup $\mathrm{SL}(2,5)$, see [26, page 411]. Assume further that $p\equiv 1$ $(\mathsf{mod~}8)$. Then the centre $\mathbf{Z}(\mathrm{GL}(2,p))\cong \mathbb {Z}_{p-1}$ has order divisible by 8. Let $\sigma$ be a 2-element of $\mathbf{Z}(\mathrm{GL}(2,p))$ which has order at least 8. Since $\sigma$ is a scalar matrix, its action on $R$ is fixed-point-free. Then $\langle \sigma ,\mathrm{SL}(2,5)\rangle =\langle \sigma \rangle \circ \mathrm{SL}(2,5)$ is a subgroup of $\mathrm{GL}(2,p)$. We note that the condition that $p\equiv \pm 1\pmod {10}$ and $p\equiv 1\pmod 8$ is equivalent to $p\equiv 1$ or $9\pmod {40}$.

Lemma 3.1

Let $M=\langle \sigma ^2,\mathrm{SL}(2,5)\rangle$ and $H=\langle \sigma ,\mathrm{SL}(2,5)\rangle$. Then $M$ divides each $H$-orbit on $R^\#$ into two parts of equal size.

Proof

Let $\Delta$ be an $H$-orbit on $R^\#$, and let $x\in \Delta$. Then $x^\sigma \in \Delta$. Suppose that $M$ is transitive on $\Delta$. Then there exists an element $g\in M$ such that $x^g=x^\sigma$. Thus, $x^{\sigma g^{-1}}=x$. Since $\sigma$ is in the centre of $H$, $\sigma g^{-1}=g^{-1}\sigma$. Hence $(\sigma g^{-1})^{o(g)}=\sigma ^{o(g)} (g^{-1})^{o(g)}=\sigma ^{o(g)}$. As no element of $\mathrm{SL}(2,5)$ has order divisible by 8, the power $\sigma ^{o(g)}$ is not the identity. Since $\sigma g^{-1}$ fixes $x$, so does the power $(\sigma g^{-1})^{o(g)}=\sigma ^{o(g)}$. This is not possible since the scalar $\sigma ^{o(g)}$ is a fixed-point-free automorphism of $R$. Therefore, $M$ is not transitive on $\Delta$. Since $M$ is a normal subgroup of $H$ of index 2, $M$ acts on $\Delta$ half-transitively and has exactly two equal size orbits. $\square$

This lemma enables us to have the following construction.

Construction 3.2

Using the notation above, we define a graph as below.

• (1) Let $\Delta _1,\ldots ,\Delta _r$ be the orbits of $H$ acting on $R^\#$, and let $\Delta _i^+$ and $\Delta _i^-$ be the two orbits of $M$ on $\Delta _i$, where $1\leqslant i\leqslant r$.

• (2) Let $S=\bigcup _{i=1}^r\Delta _i^{\varepsilon _i}$, where $\varepsilon _i=+$ or $-$, and let $\varGamma =\mathsf{Cay}(R,S)$.

Lemma 3.3

Let $\varGamma$ be a Cayley graph constructed in Construction 3.2. Then $\varGamma$ is a self-complementary metacirculant with insoluble automorphism group $\mathsf{Aut}\varGamma \geqslant \mathbb {Z}_p^2{:} (\langle \sigma ^2\rangle \circ \mathrm{SL}(2,5))$, and $\sigma$ is a complementing isomorphism.

Proof

By definition, $\sigma$ interchanges $\Delta _i^+$ and $\Delta _i^-$, where $1\leqslant i\leqslant r$. Thus,

$$\begin{aligned} S^\sigma =\bigcup _{i=1}^r({\varDelta }_i^{\varepsilon _i})^\sigma =\bigcup _{i=1}^r {\varDelta }_i^{-\varepsilon _i}=R^\#{\setminus } S. \end{aligned}$$

Since $\sigma$ is a scalar of order divisible by 8, the power $\sigma ^{o(\sigma )/2}$ maps every element $x\in R$ into the inverse $x^{-1}$. Hence each $\Delta _i$ is self-inverse and so is $S$. So $\varGamma$ is undirected and is self-complementary with $\sigma$ being a complementing isomorphism.

Since $R$ is metacyclic, the Cayley graph $\varGamma$ of $R$ is a metacirculant. Moreover, $\mathsf{Aut}\varGamma \geqslant R{:}M=\mathbb {Z}_p^2{:}\langle \sigma ^2,\mathrm{SL}(2,5)\rangle =\mathbb {Z}_p^2{:}(\langle \sigma ^2\rangle \circ \mathrm{SL}(2,5))$ is insoluble. $\square$

By iteratively taking the lexicographic products of graphs constructed via Construction 3.2, we can in fact construct self-complementary metacirculants with insoluble automorphism groups, which are Cayley graphs on $\mathbb {Z}_{n_1n_2}^2$ for any coprime pair $n_1,n_2$ of square-free integers. Construction 6.1 and Lemma 6.2 of [8] show that the resulting graphs are in fact self-complementary metacirculants with insoluble automorphism groups in which the insoluble composition factor $\mathrm{A}_5$ appears multiple times.

We next give a different construction of self-complementary metacirculants of which the automorphism groups contain a section $\mathrm{A}_5\times \dots \times \mathrm{A}_5$.

Let $p_1,\ldots ,p_t$ be distinct primes such that $p_i\equiv 1~\text{ or }~9~(\mathsf{mod~}40)$ for each $i\in \{1,\ldots ,t\}$. Let $R_i=\mathbb {Z}_{p_i}^2$, and let $L_i<\mathsf{Aut}(R_i)=\mathrm{GL}(2,p_i)$ be isomorphic to $\mathrm{SL}(2,5)$. Let $\sigma _i\in \mathbf{Z}(\mathsf{Aut}(R_i))$ have order divisible by 8. Let $R=R_1\times \dots \times R_t$, $L=L_1\times \dots \times L_t$, and $\sigma =(\sigma _1,\dots ,\sigma _t) \in \mathsf{Aut}(R)$. Let $H=R{:}(\langle \sigma \rangle \circ L)$ and $M=R{:}(\langle \sigma ^2\rangle \circ L)$.

Lemma 3.4

The subgroup $M$ divides every $H$-orbit on $R^\#$ into two parts of equal size.

Proof

(The proof is similar to the proof of Lemma 3.1.) Let $\Delta$ be an $H$-orbit on $R^\#$, and let $(x_1,\dots ,x_t)\in \Delta$. Suppose that there is an element $g\in M$ such that $(x_1,\dots ,x_t)^g=(x_1,\dots ,x_t)^\sigma$. Then $(x_1,\dots ,x_t)^{\sigma g^{-1}}=(x_1,\dots ,x_t)$. As $\sigma g^{-1}=g^{-1}\sigma$, we have $(\sigma g^{-1})^{o(g)}=\sigma ^{o(g)} (g^{-1})^{o(g)}=\sigma ^{o(g)}$. Noticing that no element of $L$ has order divisible by 8, we know $\sigma ^{o(g)}\not =1$. Since $\sigma g^{-1}$ fixes $(x_1,\dots ,x_t)$, so does the power $\sigma ^{o(g)}$. Now some $x_i\not =1$, and $x_i^{\sigma _i^{o(g)}}=x_i$, which contradicts that the scalar $\sigma _i^{o(g)}$ is a fixed-point-free automorphism of $R_i$. So $M$ is intransitive on $\Delta$ and has exactly two orbits of equal size. $\square$

Thus, applying Construction 3.2 with $H,M$ defined before Lemma 3.4 produces examples of self-complementary metacirculants of which the automorphism groups have a section $\mathrm{A}_5^t$ for any positive integer $t$.

Lemma 3.5

Let $p_1,\ldots ,p_t$ be distinct primes such that $p_i\equiv 1~\text{ or }~9~(\mathsf{mod~}40)$ for every $i\in \{1,\ldots ,t\}$. Then there exist self-complementary metacirculants $\varGamma$ of order $p_1^2\ldots p_t^2$ such that $\mathsf{Aut}\varGamma \geqslant \mathbb {Z}_{p_1\ldots p_t}^2{:}(\mathbb {Z}_\ell \circ \mathrm{SL}(2,5)^t)$, where $\ell \,\big |\,(p_i-1)$ for each $i$ and $8\,\big |\,\ell$.

4 Proof of Theorem 1.1

Let $\varGamma =(V,E)$ be a self-complementary metacirculant of a metacyclic group $R$. By [14], we may assume that $R$ is not cyclic. Let $G=\mathsf{Aut}\varGamma$, and let $\sigma$ be a complementing isomorphism of $\varGamma$. Let $X=\langle G,\sigma \rangle$. Then $\sigma ^2\in G$, and $X=G.\mathbb {Z}_2$. Let $n$ be a positive integer, and let $p$ be a prime factor of $n$. By $n_p$ we denote the highest power of $p$ that divides $n$, so that $p$ and $n/n_p$ are coprime.

4.1 The primitive case

We consider first the vertex-primitive case. Suppose that $X$ is primitive on $V$. Then a result of [10] states that either $X$ is affine or the socle $\mathsf{soc}(X)=\mathrm{PSL}(2,q^2)^\ell$ and $|V|=(\frac{1}{2}q^2(q^2+1))^\ell$, where $q$ is odd and $\ell \geqslant 2$. We first exclude the latter case.

Lemma 4.1

The group $X$ is affine.

Proof

Suppose that $X$ has socle $N=\mathrm{PSL}(2,q^2)^\ell$, and $|V|=(\frac{1}{2}q^2(q^2+1))^\ell$. Since $R$ is transitive on $V$, it follows that $q^{2\ell }$ divides $|R|$. Let $q=p^f$ with $p$ prime and $f\geqslant 1$. Since $R$ is metacyclic, $R$ contains an element $x$ of order $q^\ell =p^{f\ell }$. Let $K$ be the largest subgroup of $X$ which normalises each direct factor of $N$. Then $N\lhd K\leqslant {\mathrm{P}\Gamma \mathrm{L}}(2,q^2)^\ell$. Note that ${\mathrm{P}\Gamma \mathrm{L}}(2,q^2)=\mathrm{PGL}(2,q^2){:}\mathbb {Z}_{2f}$, and $\mathrm{PGL}(2,q^2)$ has a Sylow $p$-subgroup $\mathbb {Z}_p^{2f}$. Hence a $p$-element of $K$ has order at most $pf_p$. Thus,

$$\begin{aligned} X/K\geqslant \langle x\rangle K/K\cong \langle x\rangle /(\langle x\rangle \cap K), \end{aligned}$$

and so $X/K\leqslant S_\ell$ contains an element of order $p^{f\ell -1}/f_p$. However, $p^{f\ell -1}/f_p$ does not divide $|\mathrm{S}_\ell |=\ell !$, which is a contradiction. $\square$

We now show that only affine groups of dimension $2$ can appear.

Lemma 4.2

The primitive group $X$ is affine of dimension $2$.

Proof

By Lemma 4.1 the socle $N=\mathsf{soc}(X)=\mathbb {Z}_p^d$, where $p$ is a prime and $d\geqslant 1$. Then $X\leqslant \mathrm{AGL}(d,p)$, and $p^d$ divides $|R|$. Since $R$ is metacyclic, there exists an element $g\in R$ of order divisible by $p^{\lceil \frac{d}{2}\rceil }$. Since $\mathrm{AGL}(d,p)<\mathrm{GL}(d+1,p)$ and they have isomorphic Sylow $p$-subgroups, by Lemma 2.5, the largest order $p^e$ of the $p$-elements of $\mathrm{AGL}(d, p)$ is such that $p^e\geqslant d+1>p^{e-1}$. Thus, $p^{\lceil \frac{d}{2}\rceil -1}\leqslant p^{e-1}<d+1$. Noticing that $p\geqslant 3$, this implies that $d\leqslant 4$.

Suppose that $d=3$. Then $p^{\lceil \frac{d}{2}\rceil }=p^2\leqslant p^e$, and $p\leqslant p^{e-1}<d+1=4$. Thus $p=3$, and $|V|=3^3\not \equiv 1$ $(\mathsf{mod~}4)$, which is not possible by Lemma 2.2.

Suppose that $d=4$. Then $\lceil \frac{d}{2}\rceil =2$, and it follows that $p\leqslant p^{e-1}<d+1=5$. Thus $p=3$, and $|V|=3^4$. Without loss of generality, we may assume that $R$ is a $3$-group. Note that $\mathrm{AGL}(4,3)<\mathrm{GL}(5,3)$, by Lemma 2.5, the largest order of the $3$-elements of $\mathrm{GL}(5,3)$ is $9$. Thus, the metacyclic group $R=\mathbb {Z}_9.\mathbb {Z}_9$. However, checking the subgroups of $\mathrm{GL}(5,3)$ by GAP [27] leads to the conclusion that $\mathrm{GL}(5,3)$ does not have a metacyclic group $\mathbb {Z}_9.\mathbb {Z}_9$.

Finally, since we have assumed that $R$ is not cyclic, we conclude that $d=2$. $\square$

Next, we determine the insoluble case.

Lemma 4.3

Assume that $X$ is primitive and insoluble. Then we have

$$\begin{aligned} X=\mathbb {Z}_p^2{:}(\mathbb {Z}_\ell \circ \mathrm{SL}(2,5)), and G=\mathbb {Z}_p^2{:}(\mathbb {Z}_{\ell /2}\circ \mathrm{SL}(2,5)), \end{aligned}$$

where $p\equiv 1~\text{ or }~9~(\mathsf{mod~}40)$, and $\ell$ divides $p-1$ and is divisible by $8$.

Proof

Since $X$ is primitive and insoluble, by Lemma 4.2, we have $|V|=p^2$, and $X\leqslant \mathrm{AGL}(2,p)$, and the stabiliser $X_v$ is an irreducible subgroup of $\mathrm{GL}(2, p)$.

Notice that $G$ has index 2 in $X$ and is not 2-transitive on $V$, so $G_v$ does not contain $\mathrm{SL}(2,p)$. Inspecting the subgroups of $\mathrm{SL}(2,p)$ listed in [26, Theorem 6.17], we conclude that $X_v$ contains a subgroup isomorphic to $\mathrm{SL}(2,5)$ with $p=5$ or $p\equiv \pm 1~(\mathsf{mod~}5)$. Noting that $\mathbb {Z}_{p-1}\circ \mathrm{SL}(2,5)$ is maximal in $\mathbb {Z}_{p-1}\circ \mathrm{SL}(2,p)$ and there is no element of $\mathrm{GL}(2,p){\setminus }(\mathbb {Z}_{p-1}\circ \mathrm{SL}(2,5))$ which normalises $\mathrm{SL}(2,5)$, we have

$$\begin{aligned} X_v=\langle z\rangle \circ \mathrm{SL}(2,5)=\mathbb {Z}_\ell \circ \mathrm{SL}(2,5), \end{aligned}$$

where $z$ is a scalar of order $\ell$ dividing $p-1$. Since $G$ is transitive on $V$, the index $|X_v:G_v|=|X:G|=2$. We may assume $\sigma \in X_v$. As $\mathrm{SL}(2,5)$ has no subgroup of index 2, $\sigma \notin \mathrm{SL}(2,5)$. Since $\sigma$ normalises $G_v$, $\sigma$ normalises $\mathrm{SL}(2,5)$, and we may choose $\sigma \in \langle z\rangle$. Then $\sigma \notin \langle z^2\rangle$, and $G_v=\langle z^2\rangle \circ \mathrm{SL}(2,5)$. By Lemma 2.1, the order $o(\sigma )$ is divisible by 4.

In particular, the minimal normal subgroup $\mathbb {Z}_p^2$ of $X$ is regular on $V$ and is a Sylow $p$-subgroup. So $\varGamma$ is a metacirculant and a Cayley graph of $R$, say

$$\begin{aligned} \varGamma =\mathsf{Cay}(R,S). \end{aligned}$$

Suppose that $\ell$ is not divisible by 8. Then since $\sigma \in \mathbb {Z}_\ell$, $\sigma$ is of order $4m$ with $m$ odd, and

$$\begin{aligned} X_v=\langle \sigma ^4\rangle \times (\mathbb {Z}_4\circ \mathrm{SL}(2,5))=\mathbb {Z}_m\times (\mathbb {Z}_4\circ \mathrm{SL}(2,5)), \end{aligned}$$

and $G_v=\mathbb {Z}_m\times \mathrm{SL}(2,5)$. Let $\tau =\sigma ^m\in \langle z\rangle$. Then $\tau$ is of order 4, and $X_v=\langle \tau \rangle \circ G_v$. For every element $x\in R$, we have $x^\tau =x^j$ for some integer $j$ which has order 4 in the multiplicative group $\mathrm{GF}(p)^*$ of $\mathrm{GF}(p)$.

Let $y\in G_v$ be of order $4$. Since 4 divides $p-1$, the group $\langle y\rangle$ is reducible on $R=\mathsf{soc}(X)$. Thus, $y$ normalises a cyclic subgroup $\langle w\rangle \cong \mathbb {Z}_p$, where $w\in R$, and $y$ induces an automorphism of $\langle w\rangle$ of order 4. So $w^y=w^k$, for some integer $k$ which has order 4 in $\mathrm{GF}(p)^*$. Since $\mathrm{GF}(p)^*$ has a unique subgroup of order 4, $k=j$ or $j^{-1}$ in $\mathrm{GF}(p)^*$. Thus, $w^\tau =w^y$ or $w^{y^{-1}}$, respectively. Without loss of generality, assume that $w\in S$. Since $y\in G$, both $w^y$ and $w^{y^{-1}}$ belong to $S$. However, $w^\tau \in S^\tau =R^\#{\setminus } S$, which is a contradiction. Hence $8$ divides $o(\sigma )$, and $p\equiv 1~(\mathsf{mod~}8)$.

Finally, since $p\equiv \pm 1~(\mathsf{mod~}5)$, it follows that $p\equiv 1~\text{ or }~9~(\mathsf{mod~}40)$. The proof is completed. $\square$

4.2 Proof of Theorem 1.1

In order to prove Theorem 1.1, we assume that $X$ is insoluble. We complete our proof by induction on the order $|V|$.

If $X$ is primitive on $V$, then by Lemmas 4.2 and 4.3, the statement of Theorem 1.1 is true. We thus assume that $X$ is imprimitive on $V$, and further assume that Theorem 1.1 holds for any self-complementary metacirculant of order properly dividing $|V|$.

Let ${\mathcal {B}}$ be an $X$-invariant partition of $V$, called a block system of $X$ on $V$. Each element $B\in {\mathcal {B}}$ is called a block. By $[B]_\varGamma$ we denote the induced subgraph of $\varGamma$ on $B$. We need the following result (see [13]).

Theorem 4.4

(Li-Praeger 2003) Suppose that $V$ has a block system ${\mathcal {B}}$ under the action of $X$. Then the following hold:

• (i) for each $B\in {\mathcal {B}}$, the induced subgraph $[B]_\varGamma$ is self-complementary, $G_B^B\leqslant \mathsf{Aut}[B]_\varGamma$, and $\sigma$ induces a complementing isomorphism of $[B]_\varGamma$;

• (ii) there is a self-complementary graph ${\varSigma }$ with vertex set ${\mathcal {B}}$ such that $G^{\mathcal {B}}\leqslant \mathsf{Aut}{\varSigma }$ and any element of $X^{\mathcal {B}}{\setminus } G^{\mathcal {B}}$ is a complementing isomorphism of ${\varSigma }$.

Let $K=X_{({\mathcal {B}})}$ be the kernel of $X$ acting on ${\mathcal {B}}$. Then, as $|{\mathcal {B}}|$ properly divides $|V|$, by our assumption, Theorem 1.1 holds for $X^{\mathcal {B}}$, that is, if $X^{\mathcal {B}}$ is insoluble, then $\mathrm{A}_5$ is the only insoluble composition factor of $X^{\mathcal {B}}$, and $X^{\mathcal {B}}$ has a section which has the form

$$\begin{aligned} \mathbb {Z}_q^2{:}(\mathbb {Z}_m\circ \mathrm{SL}(2,5)), \end{aligned}$$

where $q$ is a prime, $8\,\big |\,m$ and $m\,\big |\,(q-1)$ and $\mathbb {Z}_q^2$ is a section of $R^{\mathcal {B}}$.

If $K$ is soluble, then $X=K.X^{\mathcal {B}}$ satisfies Theorem 1.1. We thus assume that $K$ is insoluble. Then $K^B$ is insoluble and so is $X_B^B$.

To complete the proof, we may further assume that ${\mathcal {B}}$ is a minimal block system of $X$, that is, for a block $B\in {\mathcal {B}}$, the induced action $X_B^B$ is primitive. By Lemmas 4.2 and 4.3, we have that $X_B^B$ is affine of degree $p$ or $p^2$ with $p$ prime, and as $X_B^B$ is insoluble,

$$\begin{aligned} X_B^B=\mathbb {Z}_p^2{:}(\mathbb {Z}_\ell \circ \mathrm{SL}(2,5)), \end{aligned}$$

and $R_B^B=\mathbb {Z}_p^2$, where $p\equiv 1$ or 9 $(\mathsf{mod~}40)$ is a prime, as in part (ii) of Theorem 1.1.

Since $K\lhd X_B$ is insoluble, $K^B=\mathbb {Z}_p^2{:}(\mathbb {Z}_{\ell '} \circ \mathrm{SL}(2,5))$, where $\ell '\,\big |\,\ell$. Furthermore, since $K\leqslant K^{B_1}\times \dots \times K^{B_t}$, where ${\mathcal {B}}=\{B_1,\dots ,B_t\}$, we conclude that $\mathrm{A}_5$ is the only insoluble composition factor of $K$. Finally, since $X=K.X^{\mathcal {B}}$ is an extension of $K$ by $X^{\mathcal {B}}$, every composition factor of $X$ is a composition factor of $K$ or $X^{\mathcal {B}}$. Therefore, $\mathrm{A}_5$ is the only insoluble composition factor of $X$. This completes the proof of our theorem. $\square$