🧑‍🤝‍🧑 Proper metrics gathering across ranks before logging

[zhc7]

according to #2468

What does this PR do?

Fixes #2468

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

I realized that the problem was in fact not only confined to DPO but also to other trainers. So I thought twice about the approach, and I think the right solution is actually to gather at the time these metrics are calculated. It's the most logical and it's what's done in the other trainers.
Thanks for spotting on the issue, I'll wait until the CI is green and merge.

[kashif]

@zhc7 accelerate has a dedicated helper for metrics: https://huggingface.co/docs/accelerate/en/package_reference/accelerator#accelerate.Accelerator.gather_for_metrics which is what is recommended to be used

[qgallouedec]

Valid point! thanks @kashif. I generalised gather_for_metrics in 17383f9