[Question]: How to export in correct or working ONNX format?

[fullymiddleaged]

Question

Hi,

After training a text classifier with Flair in Colab, I have followed this guide to successfully create onnx embeddings and to quantize my model. The final step saves this model, however it isn't an actual ONNX model, it remains a pytorch model.

Is there a way to correctly export or save the model in the ONNX format? It seems if I attempt to, then the file is unable to be loaded by ONNX.

The new model works in Flair but the aim here is to run this in production via ONNX runtime. As that's why we use ONNX right? ;)

Following the guide I ran:

import onnx
from flair.models import TextClassifier
from flair.embeddings import TransformerWordEmbeddings, TransformerDocumentEmbeddings

model = TextClassifier.load("final-model.pt")
assert isinstance(model.embeddings, (TransformerWordEmbeddings, TransformerDocumentEmbeddings))


testsentences = list(corpus.test)
sentences = random.sample(testsentences, 5)

model.embeddings = model.embeddings.export_onnx("best-embeddings.onnx", sentences, providers=["CPUExecutionProvider"], session_options={})

model.predict(sentences)

model.embeddings.optimize_model(
            "optimised-embeddings.onnx", opt_level=2, use_gpu=False, only_onnxruntime=True, use_external_data_format=False,
        )

model.embeddings.quantize_model(
            "quantized-embeddings.onnx", extra_options={"DisableShapeInference": True}, use_external_data_format=False
        )

model.save("best_onnx_model.onnx")

E.g. I attempt to load with onnx by:

onnx_model = onnx.load("best_onnx_model.onnx")

It errors with:

DecodeError                               Traceback (most recent call last)
[<ipython-input-62-9f688e935849>](https://localhost:8080/#) in <cell line: 2>()
      1 import onnxruntime
----> 2 onnx_model = onnx.load("best_onnx_model.onnx")

1 frames
[/usr/local/lib/python3.10/dist-packages/onnx/__init__.py](https://localhost:8080/#) in load_model(f, format, load_external_data)
    208         Loaded in-memory ModelProto.
    209     """
--> 210     model = _get_serializer(format, f).deserialize_proto(_load_bytes(f), ModelProto())
    211 
    212     if load_external_data:

[/usr/local/lib/python3.10/dist-packages/onnx/serialization.py](https://localhost:8080/#) in deserialize_proto(self, serialized, proto)
    116                 f"Parameter 'serialized' must be bytes, but got type: {type(serialized)}"
    117             )
--> 118         decoded = typing.cast(Optional[int], proto.ParseFromString(serialized))
    119         if decoded is not None and decoded != len(serialized):
    120             raise google.protobuf.message.DecodeError(

DecodeError: Error parsing message with type 'onnx.ModelProto'

Google Colab, Python 3.10.12
Versions:

onnx==1.16.2
onnxruntime==1.19.2
flair==0.14.0

[fullymiddleaged]

Oh dear, I was working too late...

The other .onxx files specified in my example work just fine!