I just got that tower. Don't use ice blocks, they'll see you. I glided in from the north and landed on a circle platform by the tower, waited for stamina, swam to tower. Once there I saved because wizzrobes will zap you off a few times. I did it with no stamina upgrades. About 10 tries.