Flash Attention for fine-tuning

[prince14322]

How can we use flash attention v2 for fine-tuning with huggingface models?

Does the path only works for pre-training(or extended pre-training)?

• Link

All the discussions mentioned below are for pre-training(or extended pre-training).

• Unable to train "bigcode/starcoder" model on 80 A100-80GB GPUs using FSDP huggingface/accelerate#1864

• Incorrectness in Flash Attention #1

I would like to fine-tune bigcode/starcoder 15.5 billion parameter model with 2k context length using A100-80GB.

[cmosguy]

@prince14322 and @pacman100

I am getting the error:

ValueError: The current architecture does not support Flash Attention 2.0. Please open an issue on GitHub to request support for this architecture: https://github.com/huggingface/transformers/issues/new
[2023-10-29 08:38:36,863] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0

What is going with why @pacman100 was able to run the code back in August, but now it is broken when we install the latest transformers from the git main line? What is the solution to fixing this?

[susnato]

Hi @prince14322 , Flash Attention is now added to StarCoder. PR
Please install from transformers main and and you will be able to use it for starcoder.

[younesbelkada]

Great work adding it in transformers @susnato ! That PR is part of the newest release (4.35.0) you can just do pip install -U transformers

[pacman100]

Hello, we have updated the code to use the FA2 support from the 🤗 transformers instead of the monkey patching. The Transformers support should handle packing as well as non packing scenarios resolving this issue.