Bring old seed technique back

[muellerzr]

What does this PR do?

This PR disables the seedable random sampler by default. While yes it can make user results fully reproducible, it has a questionable degradation in quality which outweighs its benefits. It can be enabled explicitly in the Accelerator.

Fixes #2268
Fixes #2208

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@BenjaminBossan

[muellerzr]

re-done version of #2264, rebase issues 😓

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

In principle, this LGTM. I added some comments, but no blockers for merging.

Regarding the original issue(s), do we have confirmation that the usage of SeedableRandomSampler causes problems for the validation/test scores when the seed is fixed correctly?

Even if not, I can see the argument for going back to the previous default for conserving previous behavior, but arguably it should be recommended to use this new setting then.

[vanbasten23]

Hi @muellerzr , we observed that the test accelerate/test_utils/scripts/test_script.py started failing on cloud TPU v3 (but not v4) after this commit (failure). Per huggingface/transformers#28421, it sounds to me the plan is to make Trainer take an accelerate configuration ( Need to set seedable_sampler=True), right?
cc: @will-cromar @JackCaoG

[muellerzr]

Correct. I’ll be doing that this week/next

[vanbasten23]

hi @muellerzr , just want to follow up and check if we can make Trainer take an accelerate configuration so to make our test pass on cloud TPU v3.

[muellerzr]

@vanbasten23 almost. See the PR for it here: huggingface/transformers#28664

[vanbasten23]

Hi @muellerzr , could you check if huggingface/transformers#28664 is ready to merge? We would like to build on top your PR and bring the TPU tests into a successful state. Thanks!