Code

add deserialize for pre tokenizers#1603

Code

Merged

Narsil merged 7 commits intomainhuggingface/tokenizers:mainfrom pre_tokenizer-serdehuggingface/tokenizers:pre_tokenizer-serdeCopy head branch name to clipboardAug 8, 2024

+143-10

Commits

Commits on Aug 7, 2024

• add deserialize

  ArthurZucker

  committed
  da394ee

• copy from the decoder

  ArthurZucker

  committed
  04e8032

• fmt

  ArthurZucker

  committed
  794209a

• clippy

  ArthurZucker

  committed
  e99daff

• fix rust tests

  ArthurZucker

  committed
  dd6652b

• fmt

  ArthurZucker

  committed
  f275e03

• don't change the test

  ArthurZucker

  committed
  0fe7b80