Add CleanCoNLL object

[susannaruecker]

Here's the PR for adding a CleanCoNLL object. Simple usage:

from flair.datasets import CLEANCONLL

cleanconll = CLEANCONLL()
print(cleanconll)

When called for the first time, this will download the necessary files, so

• the original CoNLL-03 from https://data.deepai.org/conll2003.zip
• our patch files concerning new line breaks, and our annotation files (where the tokens are masked) from the CleanCoNLL repo

It then applies the patch files to the original CoNLL-03 tokens (for our new line break etc.) and then merges those new tokens with our new annotations.

Note: As requested, I replaced all previous usage of subcalling bash scripts with pure python. Especially the patching process which until now was

subprocess.run(['patch', str(file_path), str(patch_path), '-o', output_path])

is now done with own methods, but unfortunately rather lengthy now...

Please check if it works for you! 🙂 (If you already have the CleanCoNLL files in your .flair/datasets folder you should delete those before, otherwise those simply will be read and the new reconstruction is not tested)

[elenamer]

Thanks a lot for adding this and implementing the patching in python! :)

I'm getting an error TypeError: CLEANCONLL.download_and_prepare_data() takes 1 positional argument but 2 were given, and I saw that the same one is appearing when the tests are run.

Also, so that all checks pass, you need to run mypy, ruff and black for checks and formatting

[elenamer]

it's better to also specify an encoding when reading/writing files, as different os have different default encodings

[susannaruecker]

Thanks! I fixed the deprecated argument and added the encoding everywhere.

[susannaruecker]

Now the checks pass, after some formatting fixes.

[stefan-it]

Hi @susannaruecker ,

many thanks for adding this! I have trained FLERT models on the CleanCoNLL dataset with very great results.

I have one question about further experiments with that dataset: is it possible that the SpanTagger training also gets officially integrated into Flair - I opened this issue some time ago #3457 and it would be awesome to have support in Flair for this approach as well :)