-
Notifications
You must be signed in to change notification settings - Fork 27.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change progress logging to once across all nodes #28373
Change progress logging to once across all nodes #28373
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's just include the changes to trainer_callback.py
please, and #28364 we'll leave for handling the checkpoint issue. Thanks!
e939d70
to
b824a63
Compare
Done. Thank you for reviewing. |
22e4c9a
to
5b67725
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, @muellerzr feel free to merge if this is alright for you as well
@muellerzr What do you think? Will close if you think this does not seem worth changing.. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for doing this!
What does this PR do?
Change progress logging to be once across all nodes rather than once per node.
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Library: