Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ONNX export support for DinoV2, Hiera, Maskformer, PVT, SigLIP, SwinV2, VitMAE, and VitMSN models #2001

Merged
merged 28 commits into from
Dec 19, 2024

Conversation

xenova
Copy link
Contributor

@xenova xenova commented Aug 29, 2024

What does this PR do?

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Who can review?

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@xenova xenova changed the title Add ONNX export support for Hiera models Add ONNX export support for DinoV2, Hiera, Maskformer, PVT, SigLIP, SwinV2, VitMAE, and VitMSN models Aug 30, 2024
@SangbumChoi
Copy link

@xenova Hi xenova! I think you missed to add official support for maskformer

ONNX Model for semantic-segmentation, with an all-MLP decode head on top e.g. for ADE20k, CityScapes. This class officially supports segformer.

Copy link
Collaborator

@echarlaix echarlaix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the delay @xenova !! Looks so good, thanks for adding so many new models ❤️

Comment on lines 1207 to 1208
"feature-extraction",
"masked-im",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think "image-feature-extraction" (or "feature-extraction") should be enough https://huggingface.co/datasets/huggingface/transformers-metadata/blob/main/pipeline_tags.json#L1128

Suggested change
"feature-extraction",
"masked-im",
"feature-extraction",

if not the correct class needed to load the model can be added to _CUSTOM_CLASSES : for vit-mae I only see AutoModel or AutoModelForPreTraining

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed in 7a2e94a

"vit-msn": supported_tasks_mapping(
"feature-extraction",
"image-classification",
"masked-im",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same I think can be removed :

Suggested change
"masked-im",

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed in 01929b2

@echarlaix
Copy link
Collaborator

echarlaix commented Dec 18, 2024

Also seems there is an issue for Maskformer for the feature-extraction task

FAILED exporters/onnx/test_exporters_onnx_cli.py::OnnxCLIExportTestCase::test_exporters_cli_pytorch_cpu_357_maskformer_feature_extraction_default_hf_internal_testing_tiny_random_MaskFormerForInstanceSegmentation - RuntimeError: number of output names provided (1) exceeded number of outputs (0)

the expected output name seems to be "transformer_decoder_last_hidden_state" https://github.com/huggingface/transformers/blob/69e31eb1bf123d5bd0183b753da83ab7dd9c2eec/src/transformers/models/maskformer/modeling_maskformer.py#L1615 while feature-extraction gives "last_hidden_state", can be fixed by updating MaskFormerOnnxConfig

edit : should be fixed with 3fa346c

@echarlaix echarlaix merged commit 0c42291 into main Dec 19, 2024
39 of 41 checks passed
@echarlaix echarlaix deleted the add-hiera-onnx branch December 19, 2024 08:47
@xenova
Copy link
Contributor Author

xenova commented Dec 19, 2024

Thanks so much @echarlaix! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants