Stop riding your broom. This will also remove the Automatic Broom effect.