You don't move relative to another object, you move relative to points in space, ie, the start point where you start the elapsed time, and the stop point where you stop the elapsed time. The points in space at where you start and stop the stop watch have a distance between them that can't be changed.
So how do you propose we locate these points in space?