The Gerudo Town is within the Southwest portion of the map. If you can get to the Gerudo Tower, you could just glide from there. Or, you can follow the road. Along the road, there's a Bazaar where you can find a memory. To get into Gerudo town, you'll want to go on top of the Bazaar's inn in the morning. There's a strange 'woman' who gives you vai clothing for some Rupees.