How do you convert a model trained on GPU to be used for inferencing on CPU?

[TheArowanaDude]

Hi, I am trying to use to a model I trained on a server that had cuda GPUs on CPU only; are there any library functions I can use to pull this off? Thanks in advance

[krzynio]

You don't need to convert it - it works out of the box.

[TheArowanaDude]

You don't need to convert it - it works out of the box.

That's what I thought, however when I specified cpu using this command:
flair.device = torch.device('cpu')

I got this error: "RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'"

[alanakbik]

Normally it should work out of the box, but there are some embedding types that we import for which this functionality does not work. Notably, that's the embeddings we get from the allennlp library, i.e. the ELMoEmbeddings. Is your model trained with these embeddings? Which embeddings are you using?

[TheArowanaDude]

Normally it should work out of the box, but there are some embedding types that we import for which this functionality does not work. Notably, that's the embeddings we get from the allennlp library, i.e. the ELMoEmbeddings. Is your model trained with these embeddings? Which embeddings are you using?

Ahh, that makes sense, I am using elmo embeddings

[alanakbik]

Yeah unfortunately this is a known problem, see #635. Perhaps the fix in that thread will work for you? I hope we can fix this at some point.

[gaurav8707]

If the model is torch based you can convert the model to ONXX model , then you can further optimize it