The Neanderthals were a species of human that lived next to modern humans in Europe and parts of Asia up to perhaps 24000 years ago.
Much of the discussion about the supposed ugliness or not of the Neanderthals has to do with the prejudices that we as modern humans might have about our relationship to an "inferior" species of human.
Given that Neanderthal fossils consist entirely of bones and artifacts, and we have no fossils of skin or hair, reconstructions of Neanderthals can vary somewhat as to how attractive or unattractive they appear.
But the question of Neanderthal ugliness isn't just an idle aesthetic judgement, because ugliness plays a direct role in our sex lives, and a central question about Neanderthals and modern humans is how much did they have sex with each other?
In both historical and modern times it has been observed that people prefer to find partners from their own race and culture. But at the same time we know that it doesn't take much to break the barriers down, and there's enough inter-racial boy-meet-girl stuff going on that most genes flow freely across the racial boundaries, and almost the only genes which don't flow freely are precisely those genes which define visible racial characteristics.
The sequencing of the Neanderthal genome from DNA found in Neanderthal bone fossils is currently a work in progress, and when that work is completed we will have a very clear idea of how separated the Neanderthals were from modern humans, and how much genetic transfer, if any, there was in one direction or another.
But even now, with only a limited amount of the Neanderthal genome sequenced, the evidence points to a clear separation between the two species.
At the same time, there is compelling evidence that some genetic transfer did occur. Analysis of the gene microcephalin (also called MCPH1) within modern human populations indicates that there are two versions of the gene, one which has been in the human gene pool for about a million years, and the other which was introduced into the gene pool somewhere between 60,000 and 20,000 years ago. Of course to be introduced, it had to be introduced from somewhere, and that somewhere must have been a distinct human species which did not normally interbreed with modern humans. The only possible candidate for such a non-human human species known from the fossil record over the required time period is the Neanderthal species, although one can never rule out the existence of unknown species for which fossils have not yet been found. (For more details, see the paper and also this summary. Also, very frustratingly, sequenced Neanderthal genome does not yet include this gene.)
Assuming that most of the time Neanderthals and modern humans did not interbreed, we can ask what prevented them from doing so. There are only four major options that I can think of:
I don't think that Neanderthals and modern humans were different enough in shape for anatomy to be a problem.
Although Neanderthals and modern humans may never have occupied exactly the same area at exactly the same time, this seems likely to have been due to competitive exclusion, in which case the two species would have constantly been in contact with each other over the length of a boundary which no doubt moved back and forth as a function of changing circumstances and changing competitive advantage and disadvantage.
Which leaves us with mutual sterility and mutual repulsion. Actually, it can be argued that if mutual sterility and mutual repulsion are the only factors preventing inter-breeding, then mutual repulsion will always play a role. If two species are not mutually sterile, then repulsion is the only thing that will prevent them inter-breeding. If two species are mutually sterile, then it becomes very unproductive for members of either species to make the effort to reproduce with each other, and this should create a selective pressure in favour of mutual unattractiveness, if not downright mutual ugliness.
The main difference between these two possibilities is that if the species are not mutually sterile, then it is the female who is the big loser when inter-breeding occurs, because she has to carry the baby that will be disadvantaged by having a non-adaptive mixture of genes, whereas if they are mutually sterile, the male will lose out more, because all the effort that a male normally makes in order to achieve sexual success will be wasted if nothing comes of it.
It might seem that the historical ugliness or otherwise of the Neanderthals could be measured by a sufficiently accurate reconstruction of their appearance. But this ignores the likelihood that our "modern" human ancestors of 40,000 years ago may have had different sexual preferences to modern day "modern" humans with respect to Neanderthals, for the simple reason that Neanderthals existed then and now they don't. As soon as the Neanderthals disappeared, the selective pressure to avoid inter-breeding with them disappeared as well. It no longer mattered to be disinclined to mate with a Neanderthal if there were no Neanderthals to mate with. Thus the repulsion that we feel when looking at a reconstructed Neanderthal may be only a ghost of the true repulsion that our ancestors felt towards members of their sibling species.
When the barriers between species are formed by selective pressures, it is because any potential progeny of an inter-species mating would be disadvantaged by the mixing of genes that are not adapted for each other's presence. Considering two species A and B that could mate, but which don't, the implication is that the genes of species A are maladapted to co-exist with those of species B, and vice versa.
However, the longer two species remain separated, the more likely it becomes that one of the species, for instance species A, will evolve an adaptation that has not evolved in the other species, i.e. species B, but which would be advantageous to species B if it could somehow get hold of a copy of the relevant genes.
But the only way that species B is going to get hold those genes is by inter-breeding with species A, which is precisely the thing that species B has evolved not to do, as part of its becoming a separate species.
In this respect, the logic of inter-species mating is similar to the logic of point mutations. In both cases there is an upside and a downside. For inter-species mating the upside is assimilation of new evolved adaptations from the other species, and the downside is the cost of mixing genes which are not co-adapted. In the case of mutation, the upside is that a mutation might be beneficial, and the downside is that the mutation probably won't be beneficial, and indeed is more likely to be harmful.
It seems in both cases that the downside exceeds the upside. But the comparison depends on how many generations after the event we consider. The maximum downside of a harmful mutation is that the mutated organism dies. This is a loss of one member of the population. The maximum upside of a beneficial mutation is that eventually all the members of the population are descendants of the mutated organism. If the amount of benefit is relatively low, this doesn't affect the size of the upside, it only affects how long it takes to reach full effect.
The same applies to inter-species sex. The maximum downside is the loss of viability of the inter-species progeny. The maximum upside is that eventually all members of the receiving species will contain the new better-adapted genes received from the other species.
The implication in both cases is that there is always an optimal rate of the relevant genetic event which may be arbitrarily small, but which is always greater than zero.
Thus, in the case of Neanderthals and modern humans, as long as there was some possibility of inter-breeding occurring, there was an optimal rate of inter-breeding which was always greater than zero. This rate might have been as low as once a century in a population of a hundred thousand in each species, and descendants of the progeny of these mating events might have survived in only 1% of cases. But that still equates to new genes being injected from one species to the other every 10,000 years.
Considering the specific case of the microcephalin gene, and assuming that the recently introduced haplotype did come from the Neanderthals, we can speculate about the specific details of the relevant encounter.
Firstly, in order to inject genes into the modern human species, the child of the Neanderthal/modern union would have to have been raised in modern human society. Almost certainly the major cultural contrasts between Neanderthals and our own ancestors would have made it impossible for an adult of one species to incorporate themselves successfully into the society of the other species. Both these considerations point towards a casual encounter between a Neanderthal male and a modern human female.
Assuming also the existence of a mutual repulsion between the sexes, we would have to wonder why two particular individuals would ignore such a repulsion. We can consider the probability p that one individual lacked the normal repulsion, and this would have to be a very small probability to maintain the normal species barrier. The probability that two individuals who happened to meet both lacked the normal repulsion is the even smaller number p2. It's more likely that one individual would have been willing, and the other individual would not have been willing. Add into this the consideration that Neanderthals were stronger and heavier than modern humans (and brute force counts more in an accidental one-on-one encounter), and that we are already assuming that the father had to be Neanderthal and the mother a modern human, and we end up with a scenario with a Neanderthal male whose sexual tastes are sufficiently deviant to lust after a modern human femaile, he has a chance encounter with a modern human female, and she is then forced to mate with him.
It would have been nice to write about a Neanderthal Romeo and a modern human Juliet, or perhaps a modern human Romeo and a Neanderthal Juliet, but unfortunately the numbers seem to argue against this romantic scenario, and the more probable scenario is the least pleasant one. (And I'm open to any suggestions as to how the plausibility of the more romantic scenario can be resuscitated.)
Just in case the reader feels I am painting too negative a picture of the relationship between Neanderthals and modern humans, I will point out that sex isn't everything. Many of the interactions between Neanderthals and modern humans may have been no different to interactions between different Neanderthal groups or between different modern human groups.
The Neanderthals and modern humans may have traded with each other. They may have taught each other skills and techniques. Their children may have played together. And even if modern humans committed the occasional genocide, they probably targeted other tribes from their own species as often as they targeted Neanderthals.
