This is probably not going to convince you but here goes. I hope my poor writing dos not confuse my meaning too much.
Kepler determined that the orbits of the planets were elliptical. When Newton came up with his theory of gravity, he knew that his theory had agree with Kepler's discovery. Newton's formulae do recreate Kepler's theories and more. Kepler's theory was that the orbits of planets are elliptical. Newton in his wisdom knew that his theory would be used to calculate ballistic trajectories. These trajectories would have to be elliptical but that also meant that it would be very difficult to calculate the impact point of a ballistic object given its trajectory. So he did something very interesting. He simplified the problem by changing the geometry. He proposed an infinite flat earth with gravity always perpendicular to the surface which with a flat infinite earth meant that all down pointing gravity vectors were parallel. Using this method, which was approximate for all practical purposes, he was able to use parabolas to replace the elliptical orbits that ballistic objects should travel on. Parabolas are much easier to calculate trajectories with. As long as your the ballistic trajectory is less that a couple miles, it is a close approximation.
Einstein did something that is similar in a backwards way. He looked at the problem of the invariant speed of light and analyzed it in a geometric domain. Obviously the result had to also inhabit the same geometric domain. He realized that the formulae he derived were familiar and were indicative of a non-Euclidean geometry. Non-Euclidean geometry was much more popular in the 1800s than it is now. Or maybe it is just folded into other mathematical structures today. But the answer was quite familiar to Einstein. He realized that the answer to the problem at hand was to change the geometry. He realized that the problem which was posed in geometry would be solved with geometry.
Geometry is very closely tied to physics and is very physical in nature.You claimed that there has to be a physical connection between objects in order for there to be an effect between them. There is. It is geometry. Change the geometry and you change the physical laws, just as Newton did. Physics is not just objects and forces. It includes geometry, and also information. Maybe other stuff too. As many here have said, space in not just "nothing". It has properties that relate to geometry and other attributes. The geometry of spacetime is very well studied and understood, and it is really the only answer.
If you really look at the aether problem, I think you will find that the aether really looks a lot like "empty" space.
Great post.