Sentence object with pre-tokenized text

[KDercksen]

Dear Flair team,

I was wondering what the best approach is to use Flair with pre-tokenized text. My data is, broadly speaking, already in the form (["Hi", ",", "how", "are", "you", "Mr.", "President", "?"], ["O", "O", "O", "O", "O", "B-PER", "L-PER", "O"]) for entity tagging. As far as I can see, Sentence does not have functionality for this out of the box. Do I just populate the tokens list manually starting from an empty sentence, or how should I go about this?

Thanks for your awesome work!

[KDercksen]

I think the most elegant way is the following:

text, labels = ["Hi", ...], ["O", ...]

# join tokens with whitespace, then split on whitespace
sentence = Sentence(" ".join(text), use_tokenizer=False)

# add labels
for token, label in zip(sentence, labels):
    token.add_tag("ner", label, confidence=1.0)

Assuming there are no whitespace tokens in the original list, this should work fine.

[AAnirudh07]

It still works three years later! Thank you

[helpmefindaname]

I just want to point out, that the Sentence also directly accepts tokens, hence Sentence(tokens) should be more elegant than Sentence(" ".join(tokens), use_tokenizer=False)