Feature Request: ONNX Export

[jplu]

Hello!

I was wondering if you planned at some point to integrate the possibility to export the trained models in ONNX format in order to use them into an ONNX Runtime environment.

Thanks!

[asahi417]

Hi, I'm actually not familiar with the ONNX format. Is there an easy way to export a pytorch model checkpoint in ONNX format?

[jplu]

Hi,

Yes there are some easy ways, here the two most straightforward approaches:

• https://pytorch.org/docs/stable/onnx.html
• https://huggingface.co/docs/transformers/serialization

[vsuman-ontic]

Hi,
@jplu and @asahi417 Do have the .onnx format model, Does anyone try to do it?
If so can we have it, I am working on how we increase the model inference time.
Currently each message take atleast ~ 3 sec.

Thank you,
Vicky Suman

I was trying to do it, but facing an Error

ValueError: Outputs values doesn't match between reference model and ONNX exported model: Got max absolute difference of: 0.00021386146545410156 for [-0.898747 0.32406276 -0.1184229 ... -0.19128662 -0.04069862 0.42074022] vs [-0.8987234 0.32408154 -0.11840978 ... -0.19131573 -0.04068365 0.42075542]

(onxx_env) (base) vsuman-mbp@MacBook-Pro NER % python -m transformers.onnx --model=app/models/tner/roberta-large-ontonotes5 app/models/onnx --opset=11 --framework=pt  
Some weights of RobertaModel were not initialized from the model checkpoint at app/models/tner/roberta-large-ontonotes5 and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using framework PyTorch: 2.0.1
Overriding 1 configuration item(s)
	- use_cache -> False
================ Diagnostic Run torch.onnx.export version 2.0.1 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Validating ONNX model...
	-[✓] ONNX model output names match reference model ({'last_hidden_state'})
	- Validating ONNX Model output "last_hidden_state":
		-[✓] (3, 9, 1024) matches (3, 9, 1024)
		-[x] values not close enough (atol: 1e-05)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/vsuman-mbp/Documents/code/analytics/NER/onxx_env/lib/python3.11/site-packages/transformers/onnx/__main__.py", line 242, in <module>
    main()
  File "/Users/vsuman-mbp/Documents/code/analytics/NER/onxx_env/lib/python3.11/site-packages/transformers/onnx/__main__.py", line 234, in main
    export_with_transformers(args)
  File "/Users/vsuman-mbp/Documents/code/analytics/NER/onxx_env/lib/python3.11/site-packages/transformers/onnx/__main__.py", line 178, in export_with_transformers
    validate_model_outputs(onnx_config, preprocessor, model, args.output, onnx_outputs, args.atol)
  File "/Users/vsuman-mbp/Documents/code/analytics/NER/onxx_env/lib/python3.11/site-packages/transformers/onnx/convert.py", line 465, in validate_model_outputs
    raise ValueError(
ValueError: Outputs values doesn't match between reference model and ONNX exported model: Got max absolute difference of: 0.00021386146545410156 for [-0.898747    0.32406276 -0.1184229  ... -0.19128662 -0.04069862
  0.42074022] vs [-0.8987234   0.32408154 -0.11840978 ... -0.19131573 -0.04068365
  0.42075542]