chore: Remove CLI - this was originally intended for local development

[bryantbiggs]

Removing the CLI that was originally intended for local development

The CLI functionality has been split out into its own crate. This means the main tokenizers project doesn't have to contend with compiling the clap library nor messing with enabling/disabling the CLI feature flag. This also allows the CLI to be published to be made available for use through crates, Homebrew, etc. if that is of interest.

With splitting out the CLI, workspaces are added - this also gives the benefit of being able to run the CLI from anywhere within the project but not bring in the clap dependency into tokenizers. A lot of the crate boilerplate attributes are standardized in the root Cargo.toml and inherited by the individual crates. Currently both crates share the same version but this can easily be changed if thats desired.

I don't think this is a breaking change since my understanding is that the CLI is more of an internal developer feature, but please correct me if I am wrong on that assumption

[Narsil]

Hi, thanks for this very nice PR and detailed work.

Do you have any reason why deactivating cli in your deployments was not enough ?

You are correct that the CLI is not highly maintained and not really a core dependency, although such a change would be breaking a require 0.16 to be releaed imho.

For the name, my instinct would be tokenizers-cli since it would be more in line with other HF cli we have out there (huggingface-cli, transformers-cli etc..).
Although the tendency has been more to deprecate them (to not have a bazillion CLIs and focus on a handful)
My take would actually be to just remove cli from the default features in 0.16, as it would be simpler overall implementation wise.

@ArthurZucker what's your opinion ?

[bryantbiggs]

My take would actually be to just remove cli from the default features in 0.16, as it would be simpler overall implementation wise.

Even better!

Do you have any reason why deactivating cli in your deployments was not enough ?

No, nothing in particular - it was more of a bit of confusion when I was looking at the source and saw cli tucked in there behind a feature flag

[ArthurZucker]

Down to remove it as well 😉

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[bryantbiggs]

Done!

[ArthurZucker]

LGTM thanks for cleaning