Model.encode() error

[mertozlutiras]

Problem:

When I run the sample code provided for evaluation, I run for the following error in the file supert.py:

Checking the SBERT documentation, I see that model.encode() only returns embeddings, not the tokens itself.

I tried to use the AutoTokenizer from huggingface in order to return tokens, however I ran into assertion error as length of tokens were not equal to length of embeddings.

Before doing more reverse engineering, I wanted to ask you what you wanted to achieve here and if it is a deprecated usage how can we solve it?

Thank you for your help!

[mertozlutiras]

I couldn't really understand what is meant by all_tokens.

Why does the token tensors' size should be equal to embedding tensors' size?