Adapted megatronlm server implementation for interacting with lm eval…#8

Open

KlaudiaTH wants to merge 30 commits intomainfrom megatron_lmeval_server

+1,607-360

Commits on Aug 4, 2023

integrated first methods of hf tokenizer
Michael Fromm
committed

Commits on Aug 6, 2023

added tokenizer
Michael Fromm
committed

Commits on Aug 7, 2023

bugfix
Michael Fromm
committed
retrieve eod id from tokenizer
Michael Fromm
committed
bugfix 2
Michael Fromm
committed
bugfix 3
Michael Fromm
committed
bugfix 4
Michael Fromm
committed
bugfix 4
Michael Fromm
committed
_HFTokenizer typo
Michael Fromm
committed

Commits on Aug 8, 2023

added functions
Michael Fromm
committed

Commits on Aug 11, 2023

integrated pretrained hf tokenizer
Michael Fromm
committed
Add metadata query
janEbert
committed
bugfix PretrainedHFTokenizer
Michael Fromm
committed
bugfix
Michael Fromm
committed

Commits on Aug 12, 2023

Commits on Aug 16, 2023

Adaptations for greedy until generation; minor fixes
KlaudiaTH
committed

Commits on Aug 18, 2023

API and SP tokenizer adaptions for handling continuations
KlaudiaTH
committed

Commits on Aug 24, 2023

Server: Don't return padding tokens
KlaudiaTH
committed

Commits on Aug 29, 2023

Commits on Sep 5, 2023

Corrected monolingual bpe sp 32k example
KlaudiaTH
committed

Commits on Oct 16, 2023

Server: Add argument for specifying HTTP port
KlaudiaTH
committed

Commits on Nov 3, 2023