Remove the follow buff from yourself.