ORT Models Failing because of the latest fsdp changes on transformers Trainer.

[AdamLouly]

System Info

optimum from source
transformers from source

Who can help?

@JingyaHuang

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction (minimal, reproducible, runnable)

when trying to run training using ortmodule all models will fail due to latest changes on transformers trainer.
fsdp was removed as an attribute and it included other changes.

I can work on the fix if you guys don't have the bandwith.

@JingyaHuang
We also been getting a lot of this types errors, can we work on some CI pipeline to spot these failures so we can fix them fast?

Thanks.

Expected behavior

AttributeError: 'ORTTrainer' object has no attribute 'fsdp'

[JingyaHuang]

Hi @AdamLouly , thanks a lot for reporting it. The contribution will be super helpful!

[JingyaHuang]

The CI for ORTTrainer will be restored again with the PR #1575 .

The current issue is caught here: https://github.com/huggingface/optimum/actions/runs/7142761592/job/19452757722?pr=1575

[prathikr]

@JingyaHuang any updates on this issue? I am also running into it.

[JingyaHuang]

Hi @prathikr , Adam has opened a PR for fixing it #1586 .

[nvijayrania]

Hi @JingyaHuang ,
Could you please confirm what is the last stable version which I can use with latest transformers(4.36.2) which doesn't have the fsdp issue? Tried 1.16.0, it throws the same error, or is there any plan to release the patch soon?

[JingyaHuang]

Hi @nvijayrania, I just merged the fix contributed by @AdamLouly, ORTTrainer shall work as expected if you build optimum from source. We will plan a patch release soon (but unfortunately not this week as team members are OOO this week.