Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] MPS #28016

Merged
merged 3 commits into from
Dec 15, 2023
Merged

[docs] MPS #28016

merged 3 commits into from
Dec 15, 2023

Conversation

stevhliu
Copy link
Member

As a part of a larger effort to clean up the Trainer API docs in #27986, this PR moves the Trainer for accelerated PyTorch training on Mac section to the currently empty Training on Specialized Hardware page.

Other updates include rewriting it a bit so it doesn't sound like it's copied directly from the blog post and removing the link to the paywalled article for setup 🙂

@stevhliu stevhliu requested a review from amyeroberts December 13, 2023 20:17
Copy link
Collaborator

@amyeroberts amyeroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice - thanks for reworking and tidying up!

Note: Most of the strategies introduced in the [single GPU section](perf_train_gpu_one) (such as mixed precision training or gradient accumulation) and [multi-GPU section](perf_train_gpu_many) are generic and apply to training models in general so make sure to have a look at it before diving into this section.
<Tip warning={true}>

Some PyTorch operations are not implemented in MPS yet and will throw an error. To avoid this, you should set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU kernels instead (you'll still see a `UserWarning`).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to have trainer just use the CPU entirely and ignore the MPS backend?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can set use_cpu=True here, but cc'ing @pacman100 who'll know more about it 🙂

use_cpu: bool = field(

Comment on lines -575 to -427
2. Distributed setups `gloo` and `nccl` are not working with `mps` device.
This means that currently only single GPU of `mps` device type can be used.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this no longer the case?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe its still true, I didn't see mps among the supported backends for torch.distributed (included in the second to last paragraph of the new doc)

@stevhliu stevhliu merged commit ebfdb9c into huggingface:main Dec 15, 2023
8 checks passed
@stevhliu stevhliu deleted the mps-docs branch December 15, 2023 21:17
iantbutler01 pushed a commit to BismuthCloud/transformers that referenced this pull request Dec 16, 2023
* mps docs

* toctree
staghado pushed a commit to staghado/transformers that referenced this pull request Jan 15, 2024
* mps docs

* toctree
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants