-
Notifications
You must be signed in to change notification settings - Fork 27.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Modernbert Release Fixes #35344
Modernbert Release Fixes #35344
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The rotary embedding solution is a temporary fix - I'll introduce a modular-approved one soon, but we have a release to hit.
I didn't realise the ForTokenClassification model also requires the dense -> act -> norm -> drop, I thought it was only drop like with BERT. My apologies, thanks for the quick fix.
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks 🤗
Hi, thanks, is there a plan to support multiplechoice ? |
Have you already road-tested the token classification implementation? E.g. using the ModernBERT Large model only yields to ~91% F-Score on CoNLL-2003 (Test set, I tried some batch size and learning rate configurations), in two different frameworks (Transformers and Flair). I am not sure if ModernBERT needs really special hyper-parameters or if there are fixes needed in the architecture 🤔 I tested it with b5a557e. |
This PR resolves two issues with ModernBERT so it will be working for release in just under an hour.
First, ModernBERT has a final head layer
ModernBertPredictionHead
with a pretrained Linear layer and LayerNorm . In downstream heads, these two layers are loaded toModernBertPoolingHead
. I readdedModernBertPoolingHead
toModernBertForTokenClassification
so TokenClassification models will load the pretrained layer weights.Second, #35235 appears to have broken the SDPA/Eager paths by changing how the RoPE layer ModernBERT inherited from works. In the interest of expediency, I unmoduarized it so we can have a working ModernBERT implementation on all paths.
cc @ArthurZucker @tomaarsen