They become minions for Jormag and do not leave. If only they would listen to the warnings.