As long as you have a mesmer portal open, you will not be allowed to use the orb.