While ultimatley true, Ganon would rather see Link's life ended then for him to be stopped by an impassable obstacle because he could have just blocked all the entrances to the dungeons with stone or some other sturdy object