They refuse to leave the mansion alone. Those pirates see a building with a full set of walls and no rum stains, and they're drawn to it like winged insects to a source of luminescence.