Skip to content

Commit

Permalink
[docs] add the missing tokenizer when pushing models to huggingface h…
Browse files Browse the repository at this point in the history
…ub (#33428)

* add tokenizer

* typo
  • Loading branch information
faaany authored Sep 11, 2024
1 parent c403441 commit cea9ec0
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 0 deletions.
1 change: 1 addition & 0 deletions docs/source/en/tasks/language_modeling.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,7 @@ At this point, only three steps remain:
... train_dataset=lm_dataset["train"],
... eval_dataset=lm_dataset["test"],
... data_collator=data_collator,
... tokenizer=tokenizer,
... )

>>> trainer.train()
Expand Down
1 change: 1 addition & 0 deletions docs/source/en/tasks/masked_language_modeling.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,7 @@ At this point, only three steps remain:
... train_dataset=lm_dataset["train"],
... eval_dataset=lm_dataset["test"],
... data_collator=data_collator,
... tokenizer=tokenizer,
... )

>>> trainer.train()
Expand Down

0 comments on commit cea9ec0

Please sign in to comment.