[FSDP] [EPIC] Determine how well FSDP works on Intel Gaudi and AMD MI

[RobotSail]

We want to verify that FSDP works in the following scenarios:

• AMD MI Single-GPU training #203
• AMD MI Multi-GPU, single-node training #204
• AMD MI Multi-GPU, multi-node training #205
• Intel Gaudi Single-GPU training #206
• Intel Gaudi Multi-GPU, single-node training #207
• Intel Gaudi Multi-GPU, multi-node training #208

[JamesKunstle]

We have validated that FSDP works fine on AMD cards. Haven't gotten anything to run correctly on Gaudi cards.