Factions of hate might be loud, but they're few.