... It is the close proximity of the encounters which is fascinating for me; just a few hundred kilometers with the object then being flung into outer space! It's a new insight into the dynamics of the solar system. For the length of time that life has existed on the Earth, an ultra-close near miss is likely to have happened, in my opinion. It is just the size of object and the date which is in debate.
The accelerations that can be given to an object passing near the Earth DO NOT DEPEND UPON THE MASS OF THE OBJECT.* (Strictly speaking, I am assuming it does not significantly change the Earth's position because it is much less massive and/or very rapidly passing - only an "impulse" exchanged.)
Quite possibly the closest "near miss" occurred a few years ago when Earth was used as a gravity assist boost to the velocity of a satellite. I gave the facts about it in the thread about 180 degree earth and argued that 180 earth could not exist as the calculation which gave this satellite 3 or 4 gravity assists boosts would not have worked if 180 Earth existed and its gravity (not included in the calculation) acting for more than a year on the satellite being boosted by near misses had been ignored.
I am too lazy to find my post in that thread for you, but as I recall the satellite missed Earth by only a couple hundred miles on its third gravity assist. (first two being off Venus) and still had one more to make off Jupiter to get going fast enough to reach (and fly past) Pluto in less than a decade.
I don't think it will be "flung out into space" even with four near misses. Recall that the accelerations of the passing object do not depend on the mass of the object, so it the near miss object is to leave the solar system it was very likely going to do so without any near misses.
------------
*Gravitation force acting on the object is proportional to it mass and the acceleration a force produces is inversely proportional to the object's mass. (Thus, mass "cancels out" when the force is gravitational.) If the object is much more massive than the Earth, then of course it is Earth that will receive the greater acceleration. The force on Earth will be proportional to the mass of the passing object and the inertia of the earth is proportional to the Earth's mass, so in this case the Earth can be thrown into a very different orbit, but with a mass much larger than Jupiter passing thru solar system the entire solar system would be very disturbed and some of the near miss planets could be separated from the sun. It is however highly unlikely that any object of that mass exists near our solar system and certainly none is part of it. Despite that, I postulated a small black hole, (2.2 solar masses) rapidly passes thru the solar system (missing Earth by 12 AU) as the basis of my book,
Dark Visitor. The gravitational impulse slightly changes earth's eccentricity (but it is still less than Mars has). Even this slight change kills most of Earth inhabitants as it causes a permanent ice age in the Northern Hemisphere. 11% greater apogee in N. Hemisphere's summer - colder summers and milder winters with large "spring snow falls" every day of the milder winter do not entirely melt the following summer. S. Hemisphere gets little snow but the hotter summers at perigee evaporate more ocean water and all the coastal cities, like Rio, are washed into the sea by the torrential rains that fall every summer eve. We S.H. people do not all die and can still grow rice etc.
PS "common sense" when applied to conditions far different from any you have ever experience is usually a source of mistaken beliefs. Thus, rather than be a "common-sense seeker," you should strive to be an "understanding seeker." If this post forcefully demonstrates that to you (and others) it will be one of my most useful ones. Common sense tells almost everything wrong about the quantum world but mathematical understanding of it can agree with experiments to more than 13 significant figures in some cases!