AWS Questions regarding, Multi-GPU, Trainium & Inferentia

[asutermo]

Hi there,

I have a process that finetunes SDXL w/ ControlNet on SageMaker using the AWS + HuggingFace estimator. Most of the materials were written for Transformers, but I've largely gotten this to work.

1. With the exception of Multi-gpu. I'm able to run the estimator, by referencing the diffusers git url. Unfortunately this also seems to bypass invoking accelerate. This is all python (not jupyter, not a sagemaker notebook).

        est = HuggingFace(
            entry_point="examples/controlnet/train_controlnet_sdxl.py",
            git_config=git_config,
            image_uri=image_uri,
            ...

How can I get multi-gpu to work. All of the transformers notebooks suggest this:

distribution = {"smdistributed": {"dataparallel": {"enabled": True}}}

but it appears to have no effect. I've also dabbled with mpi per other notebooks, but that causes other issues

2. There's little documentation on the subject of Trainium and Inferentia. Is it possible to finetune ControlNet (SDXL) on Trainium?

3. I was going to ask about Inferentia, but I forgot I had already posted in optimum-neuron, and it appears there's a PR there for those interested: Add Stable Diffusion ControlNet support optimum-neuron#622