Distributed Data Parallelism #402
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…mpute_outputs_and_loss which is incompatible with DDP
* delete useless files * solve typeError
* change output_dir for get-labels * solve issue 421 * review changes
* add caps_directory option in get-labels * add block announce for clinicadl 1.3.0 * fix conflicts
* Fix missing mods parsing * Fix output path * Fix missing mods parsing
ravih18
reviewed
Sep 7, 2023
There was a problem hiding this comment.
Nice work ! looks fine to me. only a few small comments.
However it would be great to add more docstrings to class and methods implemented in clinicadl/utils/maps_manager/cluster/ files as it will be easier in the future to understand what they do, what they are used for to ease the maintenance.
…version with a packaging Version object
… subpackage works
ravih18
reviewed
Sep 20, 2023
ravih18
approved these changes
Sep 20, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DDP allows the usage of multiple GPU to compute a larger batch of data. It allows us to either increase the size of the model or increase the batch size or increase our data memory footprint by for instance removing downsampling, etc.
DDP is more efficient and more flexible than DP. The usage of two GPU on the same node, or two different nodes of GPU are both covered by this PR.
To use DDP, each process of clinicadl needs to know the world size, its rank, as well as the master address and master port. For now, the cluster resolver I suggest only supports the SLURM scheduler, but if this PR is successful, we will add other cluster resolver in the future.
I also suggest the introduction of ZeRO (Zero Redundancy Optimizer). This technology from DeepSpeed (Microsoft) shards optimizer states along the data parallelism dimension. ZeRO is most optimal with the deepspeed library but that would add another dependency so that's a discussion for another day. Pytorch does have a small implementation. It only has the first stage of ZeRO (only optimizer states sharding, stage 2 introduces gradients sharding and stage 3 parameters sharding). Also, unlike DeepSpeed version, Pytorch implementation of ZeRO does increase the volume of communication required to synchronize devices. This is due to the fact that Pytorch probably wanted to limit the amount of code one would need to change to add this feature. As a result, they did not discard the second part of the gradient all-reduce (this all-gather, the first part being a ReduceScatter, or MPI_All_to_all) which is unnecessary with ZeRO. On the other hand, it indeed makes ZeRO usage painless, it's only a few additional lines of code. This feature reduces the memory footprint of the optimizer. The more GPU you have, the less memory per GPU you need. With a 300M parameters network and Automatic Mixed Precision, it reduced the amount of memory needed from 17.1GB to 15.5GB with 4 GPU. So that's neat!