They haven't bothered me. Perhaps bandits don't like watermelons. But they sure harass the other farmers an awful lot. You should talk to Jeb across the bridge. They won't leave him alone.