Actually, Ganon destroying Saria's ocarina isnt a bad theory at all. In game, he could've destroyed it when he blasted Link for challenging him. i always assumed though that he saw Zelda chuck the ocarina in the moat, and thats how he knew you had the Ocarina of Time.