Add support for OLMo's November release #34497
Conversation
|
Tested that an intermediate checkpoint can be converted and matches the original model (max logit diff <5e-5). Tested that a random OLMo 1B checkpoint can still be converted and still matches the original model (max logit diff <5e-5). |
There was a problem hiding this comment.
Thanks for the pr!
We should create a new model with https://huggingface.co/docs/transformers/en/modular_transformers 🤗
| self.q_norm = ( | ||
| get_layer_norm(config.layer_norm_type, self.num_heads * self.head_dim, config.rms_norm_eps) | ||
| if config.use_q_norm | ||
| else None | ||
| ) | ||
| self.k_norm = ( | ||
| get_layer_norm(config.layer_norm_type, self.num_key_value_heads * self.head_dim, config.rms_norm_eps) | ||
| if config.use_k_norm | ||
| else None | ||
| ) |
There was a problem hiding this comment.
hey! This goes a bit against the transformers philosophy: we never change an old model to support a new architecture! We need a Olmo2 model 🤗
with modular this should be fairly simple to implement!
There was a problem hiding this comment.
Ok, I've re-implemented this new model as a new model using modular (original model unchanged). I'll put out the PR once we have decided on a suitable model name internally (we don't intend to call this release OLMo 2).
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Making a fresh PR with modular transformers. |
What does this PR do?
An updated OLMo model will be released in November. The new model has a few small architecture changes compared to the existing model in transformers:
This PR updates the OLMo implementation in transformers to support the November release.
@ArthurZucker
Fixes #34496
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.