Sometimes as a heard I think we've developed the understanding that we will instinctively side with those most similar to our form. As such if you've got someone to take the piss out of for been differen't no one will take the piss out of you because everyones already picked one target.
I think its mostly conditioning how that happens, but we do know that people are naturally racist (
cba to google them now, but so many studies gauge peoples reactions to the same situations with differing actors), so why not naturally xenophobic.