Actually there are some big difference in humans , like dwarfism for example.
And if humans were sufficiently dwarfish (dwarvish?) they might actually not be able to breed human wildtype male x dwarf female. There doesn't appear to be any reproductive isolation in humans, but give it enough time and there would be. Compare that to the largest and smallest of dogs, or to flies with major mutations. Prezygotic barriers to mating will promote postzygotic barriers to breeding.