Code

Encode special tokens#1437

Code

Merged

ArthurZucker merged 17 commits intomainhuggingface/tokenizers:mainfrom encode-special-tokenshuggingface/tokenizers:encode-special-tokensCopy head branch name to clipboardJan 19, 2024

+173

Commits

Commits on Dec 16, 2023

• add doc in the code

  ArthurZucker

  committed
  e535bf5

Commits on Dec 18, 2023

• add option to skip special tokens

  ArthurZucker

  committed
  ed302a8

• nits

  ArthurZucker

  committed
  a581a04

• add api dummy for now

  ArthurZucker

  committed
  524acfe

Commits on Jan 18, 2024

• Fmt.

  Narsil

  committed
  262e9d2

• Fix fmt.

  Narsil

  committed
  820e88b

• Fix the stub.

  Narsil

  committed
  11e4ffc

• add a test

  ArthurZucker

  committed
  33415e0

• Merge branch 'encode-special-tokens' of github.com:huggingface/tokenizers into encode-special-tokens

  ArthurZucker

  committed
  842eced

Commits on Jan 19, 2024

• add a test in python

  ArthurZucker

  committed
  7fb3c18

• style it

  ArthurZucker

  committed
  7ac2fab

• nits

  ArthurZucker

  committed
  456e515

• add getter and setters

  ArthurZucker

  committed
  5460db0

• stub

  ArthurZucker

  committed
  bf1dbe6

• update python test

  ArthurZucker

  committed
  af40ae2

• fmt

  ArthurZucker

  committed
  ebbcc8e

• last nit

  ArthurZucker

  committed
  f7beeba