They're too chummy with the humans. Plus, their magic and golems...and those ears. It's unnatural. Not something you want in an ally.