I think the idea was, he has a bad memory, so he places the stamp on the hands of his customers. Even though you travel back in time, and hence never met him to deposit the rupees (so really, Link steals from him), he trusts the stamp more than his potentially poor memory