Well, Joseph knew he didn't sleep with her. That is why he secretly wondered if he shouldn't leave her (Matt.1:19).
If you believed in prophesy at all, you would still have to expect this sign:
Isaiah 7 (734 BCE)
14 Therefore the Lord himself will give you a sign: The virgin (/young woman) will be with child and will give birth to a son, and will call him Immanuel ("God is with us").
If the Judaic objection is valid, that the word used (almah) means 'young woman' (who has not given birth before) - it is true for both the prophesy and its fulfilment, and Jesus' messiah-ship remains intact. The rest of the prophesy remains true:
Is.9:6 For to us a child is born,
to us a son is given,
and the government will be on his shoulders.
And he will be called
Wonderful Counselor, Mighty God,
Everlasting Father, Prince of Peace.