Butterfly wing patterns are attractive, not just to artists, writers and naturalists, but also to other butterflies. In fact, they play an important role in signalling between potential butterfly mates, which means that a change in pattern can lead to the evolution of a new species. A new study has shown that closely related butterfly species are more likely to differ in pattern if they live together (sympatric) than if they live in different areas (allopatric). This provides new evidence to support the idea that selection can directly lead to increased isolation, a process known as reinforcement. In this case that isolation is manifested as a change in colour pattern.

Speciation mostly happens as a by-product of the same processes that lead to evolutionary change within populations: natural selection or genetic drift. The reason that reproductive isolation is usually not directly favoured by selection is obvious: traits such as sterility that reduce the fitness of hybrid individuals, will always be costly and must therefore be a by-product of evolution in the parental populations. Under certain scenarios, however, it is possible for natural selection to directly favour an increase in isolation between nascent species (Dobzhansky, 1937), a process known as reinforcement (Butlin, 1987). If divergent populations hybridise while in contact and hybrid matings are less productive than within-species matings, then selection may favour traits that reduce the probability of hybridisation. Reinforcement is appealing because it provides a direct role for natural selection in speciation, but its importance remains unclear. It is generally accepted that reinforcement is theoretically plausible over a range of scenarios, and a handful of convincing empirical examples support this (Marshall et al, 2002; Servedio and Noor, 2003). The outstanding question is therefore not whether reinforcement can occur, but rather, how often it does.

One data set that played a key role in reviving enthusiasm for reinforcement was Coyne and Orr's review of reproductive isolation in Drosophila (Coyne and Orr, 1989; Coyne and Orr, 1997). This showed a clear pattern of increased premating isolation in sympatric species pairs compared to allopatric species of a similar age, a pattern known as Reproductive Character Displacement. This is exactly the pattern expected if reinforcement has occurred, but unfortunately there are also alternative explanations. First, it is possible that species always diverge in allopatry while only those species pairs that already have a significant degree of premating isolation can coexist if the opportunity arises. This explanation imples that sympatric species should resemble a subset of the allopatric species, which does not seem to be the case in the Coyne and Orr data, since the youngest sympatric species show greater isolation than any allopatric species of a corresponding age. Secondly, there are other processes that occur in sympatry that might generate similar results. Ecological character displacement is the process whereby competition for resources leads to ecological divergence between sympatric species. Such divergence could lead to increased reproductive isolation as a by-product. Alternatively, reinforcement can initiate divergence that would continue primarily due to sexual selection (Liou and Price, 1994). Finally, reproductive character displacement can evolve after speciation is complete (eg if all hybrids are sterile), to reduce costly interspecific courtship and mating. These processes are all rather difficult to distinguish from ‘true reinforcement’ without knowledge of the ecology, mating systems and degree of hybridisation occurring in particular cases.

In a new development, Lukhtanov's group (2005) used a molecular phylogeny of Agrodiaetus, a group of small blue butterflies in the family Lycaenidae, to show that closely related sympatric species are more likely to differ in male colour pattern than allopatric species of a similar age. Many butterflies use their wing patterns to find and select a mate, so reinforcement could have driven this divergence in colour pattern. This study improves on data sets such as that of Coyne and Orr because it includes phylogenetic information rather than simply genetic distances between taxa, ensuring that data points are phylogenetically independent. It also presents a novel line of evidence not observed in previous data sets, namely that the more divergent sympatric taxa are actually less likely to differ in colour pattern (Lukhtanov et al, 2005). This provides strong evidence for divergence being directly due to interactions between closely related species in sympatry. Nonetheless, contrary to what is argued by the authors, it does not distinguish between several alternative processes. In addition to reinforcement, ecological character displacement is also more likely between closely related species that are likely to have similar ecological niches. Given the importance of colour patterns in crypsis, thermoregulation and other ecological processes, it seems quite plausible that the changes in pattern might be largely driven by ecological selection, rather than reinforcement (Nijhout, 1991). In other butterflies, particular wing patterns are known to be associated with ecological variables such as host plant use (Willmott and Mallet, 2004). The authors argue that ‘ecological divergence in allopatry does not lead to a change in colouration’ (Lukhtanov et al, 2005; supplementary material), but this argument is unconvincing without detailed studies of how allopatric and sympatric species differ ecologically. Ecological character displacement can only occur in sympatry, and might drive sympatric species to diverge along different ecological dimensions as compared to allopatric species. In addition, as acknowledged by the authors, their data also do not distinguish between ‘true reinforcement’ and interspecific reproductive character displacement. To do this it would be necessary to show that sympatric species are still hybridising.

Another methodological problem with the study was a reliance on mitochondrial DNA (mtDNA) to derive the phylogenetic hypothesis for Agrodiaetus, which is known to introgress between species. In the Lepidoptera hybrid breakdown is common in females (Haldane's Rule), reducing the likelihood of mtDNA intogression, but it is still known to occur (Sperling, 1993). This would bias any comparison between allopatric and sympatric species, with sympatric taxa appearing more closely related than they really are. Introgression probably does not affect the comparison of pattern differences between younger and older sympatric lineages. Nonetheless, given the ease of obtaining sequence data, it would be straightforward to add nuclear gene sequences, which would have removed any doubts in this regard.

The authors also argue for a traditional view of reinforcement as a process that occurs subsequent to divergence in allopatry. Their evidence to support this comes from an ‘Age-Range Correlation’ plot, or plot of the degree of range overlap against time since divergence. The interpretation of such plots is controversial, and in particular is confounded by range movement after speciation (Losos and Glor, 2003). Another problem is that the definition of species in allopatry is necessarily subjective under the Biological Species Concept. Thus, a tendency for taxonomists to erroneously split allopatric populations of the same species into separate species, based on minor morphological or karyotypic differences, would inflate the number of closely related allopatric taxa, giving a false signal of allopatric speciation. It may therefore be premature to entirely rule out sympatric speciation in this group. In any case, the ubiquitous evidence for reproductive character displacement in this and other studies shows that the processes responsible for divergence in sympatry can occur readily. This provides further support for the plausibility of sympatric speciation and suggests that it may be more common than is often supposed.