Actually it's because no in-game time has passed. To advance the time, just play some races. Getting a horse thru its growth mode should be enough (that's roughly 2 years in-game). But yes, it's because you are staring at them (instead of doing some races).