I wouldn't feel safe in a unisex dormitory. Sure most males aren't rapists, but most doesn't equal all.
And I would feel very endangered in a unisex toilet in most places. Pissed guys seem to think it perfectly fine to put their hands all over you, assume you are joking when you push them away and tell them to back off, and only come to their senses when you attempt to blind them with your fingers (yeah I did this)...and this is in a crowded social venue. Now translate that to a toilet where there may or may not be people around to regulate things.
I once was grabbed by a girl.
I was on 6th street in Austin. Never saw her before, yet she walked right up and grabbed my butt.
I was furious.
I've heard what you just said from many women. But I've also heard many women admit that since so many women act demure and like aggressive men- that may lead to confusion since women like to act like they don't want what they do want.
However you look at it- I am glad I'm male.