The concern is that every time you open the container you introduce an opportunity for contamination. If, however, it is within the same day and the milk is well cooled(both the fresh and the stored), it should be OK.
But the experts are more concerned about adding fresh milk to frozen milk, because the partially thaws the frozen milk and freezing and thawing does affect the quality of the milk.
Hope it helps