if you don't see a name, outside of towns or stables, and they're trying to get your attention by waving, crying, or some other obvious attention grab, just ignore them, the chances of them being Yiga Clan are just too high. (sidequest flags are different, as are monster attacks, but any other travelers are suspect, unless you see a name.)