A religion is characterized by faith or belief in a some sort of spirit or spiritual energy, which are utterly non-tangible but influences everything tangible in the material world.
Physics is a science, or rather the foundation of most science. It examines the tangible influences of the material world, and hypothesizes tangible influences.
The difference between physics and science is human common sense.
That is, a hypothesis, no matter how intangible it may resemble to the souls, spirits, et cetera of religion, it would definitely be tested for validity. On the other hand, once such an intangible hypothesis is tested, religion insists that something else is behind it, and that 'something' is truly intangible and unable ever to be tested.
It strikes me as ironic that many self-proclaimed religious people would try to lump science as a religion, because if it was, then the existence of God, for example, would have to be tested eventually. And wouldn't it scare them to death at the slightest chance that their mightiest force in the universe were to be demonstrated wrong through the power of reason?
Physics and religion are two very different aspects of the human experience. The American Physical Society says so too.