I agree. The legend of the "wild man" goes way back, even to the epic of Gilgamesh. One might even suspect it to be an archetype of our collective unconscious . . . .
I'm not sure we even need to dig so deeply. When the Agricultural Revolution occurred, it did not occur in all places at the same time. The Natufians were the first to settle in permanent farming and herding villages in Mesopotamia around 12KYA (some evidence suggests an earlier date), but the twin technologies of plant cultivation and animal husbandry were invented later in other parts of the world.
Agriculture spread out from its places of origin, but slowly. Giving up the nomadic lifestyle, which anthropologists tell us required "working" only about 25 hours per week, and moving into a village in which the formerly carefree nomad would spend about 120 hours per week just on what we now call "subsistence farming," plus more time building and maintaining newfangled contraptions like "houses," "plows," "furniture," and "pottery"... well you can see how that might not have appealed to every caveman!
So for thousands of years, there were quite a few Paleolithic tribes on this planet, and their members obviously would have crossed paths with the city dwellers rather often. The "wild man" was not merely a legend.
A mere half-millennium ago, when the Europeans began in earnest to explore the entire planet, they discovered hundreds of pre-agricultural tribes in Africa, North and South America, Australia, Malaysia, Polynesia, the Caribbean, and other nooks and crannies.
There are still a few tribes of hunter-gatherers in some of those places.
So sightings of the "Wild Man" were not all imaginary as recently as the late 19th century, and probably even later.