Simple.
Why should I protect animals I only care about in a relationship in which I kill and eat them, when I am no longer allowed to kill in eat them?
Animals only have as much value as people assign to them; if that value is putting it on your wall, or in your freezer, that's still a lot more value assigned than none. Sport hunters want to keep wild animals because they like to hunt them. If you don't let them hunt, they have no reason to support legislation that prevents them from digging, drilling or paving habitat.
In the modern era, habitat destruction is far more dangerous to a species than hunting, especially the regulated sort. The market for buffalo tongue or walrus musketproof vests is pretty slim, and few Westerners are going out and shooting stuff for food.