We infer time dilation through the effect on tangible things. If we got rid of the clocks, you can't infer anything. Space-time is a reference variable that allows us to gauge how tangible things change, but it is not real without tangible things.
Here is the experiment; We will have a large empty crate full of space-time that we will pump in from the ground. Next, we will fly around the world and when we return we will open the crate and let the two space-times merge and notice the waves. What do you think will happen? You will see nothing since the effect is all in the imagination. You need substance since space-time is only a mental grid on which substance is placed and it should not lead theory unless imaginary comes first.
In the case of the clocks, above, the impact of the airplane motion lingers within the matter (time changed due to material changes). How is that possible if space-time returns to normal when the plane lands?