You cannot leave your selected attunement. Your familiar's basic skill charges more quickly, and activating it recharges your weapon skills by a small amount. Activating your familiar's empowered skill recharges your weapon skills and triggers traits as if you entered the attunement.