Code

Refactor spm_encode#952

Code

Closed

vmarkovtsev wants to merge 49 commits intogoogle:mastergoogle/sentencepiece:masterfrom poolsideai:refactor-encodepoolsideai/sentencepiece:refactor-encodeCopy head branch name to clipboard

+369,915-102,537

Commits

Commits on May 18, 2023

• Add "nfkc_code" normalization rules

  vmarkovtsev

  committed
  35c2eba

• Add multiple memory and performance optimizations

  Show description for 9e15573
  vmarkovtsev

  committed
  9e15573

• Add installation instructions and use external abseil

  vmarkovtsev

  committed
  8fc56b5

• Allow switching between internal and external abseil

  vmarkovtsev

  committed
  180a6fb

• Fix the overflow errors

  vmarkovtsev

  committed
  527af0c

Commits on May 19, 2023

• Update the built-in protobuf sources

  vmarkovtsev

  committed
  23c0c09

• Repair the python package build

  vmarkovtsev

  committed
  f64cbf4

• Allow missing tcmalloc

  vmarkovtsev

  committed
  20be455

Commits on May 22, 2023

• Support --verbatim_control_char to mark source code sentences

  vmarkovtsev

  committed
  8d932e5

• Ensure that Encode/Decode restores whitespace in the code

  vmarkovtsev

  committed
  8f26fad

Commits on May 23, 2023

• Fix not adding all-whitespace pairs on --verbatim_control_char

  vmarkovtsev

  committed
  20f12dc

• Get rid of TBB to sort in parallel

  Show description for 6faa2d5
  vmarkovtsev

  committed
  6faa2d5

• Parallel spm_encode

  vmarkovtsev

  committed
  27f78bd

• Indicate progress in spm_encode

  vmarkovtsev

  committed
  d020431

Commits on May 24, 2023

• Link external protobuf statically

  vmarkovtsev

  committed
  581efe6

• Cache sentence frequencies to a file

  vmarkovtsev

  committed
  4d156ed

Commits on May 25, 2023

• Implement merging cached bpe frequency dicts

  vmarkovtsev

  committed
  e8f8d3b

Commits on May 26, 2023

• Reduce memory pressure in bpe trainer symbols construction

  vmarkovtsev

  committed
  28f7bf1

• Fix memory pressure in sorting final merged bpe cache

  vmarkovtsev

  committed
  4f984b0

• Initialize bpe symbols in parallel

  vmarkovtsev

  committed
  8e28f03

• Successfully load 80% of the Pile

  vmarkovtsev

  committed
  429c830

• Switch to 32-bit bpe pointers

  vmarkovtsev

  committed
  cc6a20f

• Parallelize bpe pairs construction

  vmarkovtsev

  committed
  f637bf0

• Polish UpdateActiveSymbols

  vmarkovtsev

  committed
  aeb2bd7

• Add cache only mode in GetPairSymbol

  vmarkovtsev

  committed
  ebd85f3

Commits on May 29, 2023

• Make bpe position updates parallel

  vmarkovtsev

  committed
  cc8c56d

• Avoid the second overhead of GetCachedPairSymbol()

  vmarkovtsev

  committed
  b6c7666

• Optimize the parallel BPE iterations

  vmarkovtsev

  committed
  17acb6d

• Balance sorting and freq computation

  vmarkovtsev

  committed
  ba48031

Commits on May 31, 2023

• Fix encoding non-verbatim text

  vmarkovtsev

  committed
  150ce9d

Commits on Oct 5, 2023

• Add some install instructions for finding the build path.

  

  joerowell

  authored and

  vmarkovtsev

  committed
  d1a4bb4

Commits on Oct 28, 2023

• Fixing tests (#6)

  Show description for 5cec47d
  

  kuba--

  and

  rbehjati

  authored
  5cec47d

Commits on Oct 31, 2023

• Add development documentation

  

  Razieh Behjati

  committed
  f0821a6

Commits on Nov 1, 2023

• Merge pull request #10 from poolsideai/devdocs

  Show description for dc661b8
  rbehjati

  authored
  dc661b8

Commits on Nov 2, 2023

• handle 0x01~0x04 delimiters

  

  yiyangh-ps

  authored and

  Razieh Behjati

  committed
  8985958

• Refactor code

  

  yiyangh-ps

  authored and

  Razieh Behjati

  committed
  f5f0392

• Process lines fewer than 1000

  

  yiyangh-ps

  authored and

  Razieh Behjati

  committed
  dde84eb

• Support mixed code-text format

  

  yiyangh-ps

  authored and

  Razieh Behjati

  committed
  515e65b

• Fix build command

  

  yiyangh-ps

  authored and

  Razieh Behjati

  committed
  d092158

• Remove unsed eos/bos/verbatim_control_char

  

  yiyangh-ps

  authored and

  Razieh Behjati

  committed
  a237ae4

• Remove 0x04 from encoding sequence

  

  yiyangh-ps

  authored and

  Razieh Behjati

  committed
  752a0c0

• Refactor encoding using MixedTextCodeIterator

  

  yiyangh-ps

  authored and

  Razieh Behjati

  committed
  7f16648

Commits on Nov 3, 2023

• Read from stdin

  

  Razieh Behjati

  committed
  d16e5da

• Fix a race condition issue

  yiyangh-ps

  authored
  1bce046

Commits on Nov 13, 2023

• Add ReadLineStdin to allow reading from stdin

  

  Razieh Behjati

  committed
  aa566d3

Commits on Nov 14, 2023

• Use std::shared_ptr and a slow-down mechanism

  

  Razieh Behjati

  committed
  c4bd5ea

Commits on Nov 15, 2023

• Fixes after review

  

  Razieh Behjati

  committed
  61265db

Commits on Nov 18, 2023

• Add log lines instead of assert

  

  Razieh Behjati

  committed
  ce412d7

Commits on Dec 19, 2023

• Refactor spm_encode_main

  vmarkovtsev

  committed
  2055686