New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add script for MDS conversion of bucket of text files #570

Merged

irenedea merged 38 commits into mosaicml:main from irenedea:convert_mds_script

Sep 15, 2023

Contributor

irenedea commented Aug 31, 2023 •

edited

Loading

Similar to the other data conversion scripts in foundry, this one is aimed at a continued pretraining API, which will accept a remote bucket containing text files and pretokenconcat and convert to mds for continued pretraining.

https://databricks.atlassian.net/browse/GRT-2273

Manual Test

Trained mpt-125m on the first 100 text files in a remote s3 bucket for 1 epoch (i.e. 6 batches)
Text files were processed with 4 processes, EleutherAI/gpt-neox-20b tokenizer
Tested with both mds shards uploaded to s3 and local mds shards.

https://gist.github.com/irenedea/7256479e61519b73bf7f231c7597610c

total tokens found when writing to mds: 3028992 (This is equivalent to 5 full batches with a 6th batch that has remaining samples)

Without data duplication (StreamingDatatset num_canonical_nodes=1):
time/token is 3028992 at the end of training

With data duplication:
time/token is 3145728 at the end of training

num of tokens for 6 full batches: 256 *6 * 2048 = 3145728

irenedea commented

View reviewed changes

scripts/data_prep/convert_text_to_mds.py Show resolved Hide resolved

irenedea marked this pull request as ready for review

September 1, 2023 16:44

irenedea requested a review from dakinggg

September 1, 2023 17:19

irenedea force-pushed the convert_mds_script branch 2 times, most recently from c22102f to 432bf45 Compare

September 1, 2023 20:03

irenedea commented

View reviewed changes

setup.py Outdated Show resolved Hide resolved

irenedea commented

View reviewed changes

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

dakinggg reviewed

View reviewed changes

Collaborator

dakinggg left a comment

I think you're already in process of doing this, but would like a manual test that this script produces data we can train on, and expected dataset size looks right before merging.

setup.py Outdated Show resolved Hide resolved

scripts/data_prep/utils.py Outdated Show resolved Hide resolved

scripts/data_prep/utils.py Outdated Show resolved Hide resolved

scripts/data_prep/utils.py Outdated Show resolved Hide resolved

scripts/data_prep/utils.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Show resolved Hide resolved

tests/test_convert_text_to_mds.py Outdated Show resolved Hide resolved

tests/test_convert_text_to_mds.py Outdated Show resolved Hide resolved

irenedea force-pushed the convert_mds_script branch from 535ce37 to fa4759b Compare

September 7, 2023 17:19

irenedea and others added 19 commits

September 7, 2023 10:22


          Hardcoded script works with multi processing

db545e3


          working with hashing and multiprocessing by files

9eae83c

wip

f7c9052


          working with parsed args

1940aa2


          use names to check for reprocessing

06c6b84


          wip testing

d83efef


          single process tests are working

8aed950

wip

fed82b4

wip

f8aa03a


          fix doc strings

2ff219c

wip

7997e19


          Fix tests for linux

78e1e90


          Finish support for local paths

0e9b1f3


          Test is_already_processed and refactor

0e602f1


          Fix tests

63ebcfb


          Update already processed message in scripts/data_prep/convert_text_to…

562873c

…_mds.py

Co-authored-by: Daniel King <[email protected]>


          Update docstring in scripts/data_prep/utils.py

cff07dd

Co-authored-by: Daniel King <[email protected]>


          Address PR feedback

fb6d69e


          Move data_prep/utils.py to llmfoundry

36c1ab5

irenedea force-pushed the convert_mds_script branch from fa4759b to 36c1ab5 Compare

September 7, 2023 17:22


          Remove use of glob

212750b

irenedea commented

View reviewed changes

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

irenedea added 3 commits

September 11, 2023 09:41


          Remove testing code

58aae43


          Merge branch 'main' into convert_mds_script

a2cd409


          format

afbbc47

irenedea requested a review from dakinggg

September 11, 2023 18:42


          Do not use dataloader

3d7f9c9

irenedea force-pushed the convert_mds_script branch from a9fbf91 to 3d7f9c9 Compare

September 12, 2023 21:00

irenedea requested a review from sakshigupta93

September 12, 2023 21:46


          Merge branch 'main' into convert_mds_script

5b8cee5

irenedea commented

View reviewed changes

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved


          Update scripts/data_prep/convert_text_to_mds.py

e2d772b

irenedea commented

View reviewed changes

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

irenedea and others added 3 commits

September 13, 2023 12:46


          Update scripts/data_prep/convert_text_to_mds.py

823ac45


          Merge branch 'main' into convert_mds_script

2c365ee


          Merge branch 'main' into convert_mds_script

caad5d9

dakinggg reviewed

View reviewed changes

Collaborator

dakinggg left a comment

LGTM pending a few small comments. Also I requested review from Karan to make sure the streaming stuff looks ok

llmfoundry/utils/data_prep_utils.py Outdated Show resolved Hide resolved

tests/test_convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

dakinggg requested a review from karan6181

September 15, 2023 00:25

irenedea and others added 3 commits

September 14, 2023 18:19


          Update scripts/data_prep/convert_text_to_mds.py

86215e9

Co-authored-by: Daniel King <[email protected]>


          Address comments

0eddc90


          Remove get_done_file_name function and replace with static var

b81c201

irenedea requested a review from dakinggg

September 15, 2023 04:11


          Remove extra print

01d97e9

karan6181 reviewed

View reviewed changes

llmfoundry/utils/data_prep_utils.py Show resolved Hide resolved

llmfoundry/utils/data_prep_utils.py Outdated Show resolved Hide resolved

llmfoundry/utils/data_prep_utils.py Outdated Show resolved Hide resolved

llmfoundry/utils/data_prep_utils.py Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved

scripts/data_prep/convert_text_to_mds.py Show resolved Hide resolved

irenedea added 3 commits

September 15, 2023 11:40


          Address PR style/docstring comments

94ae8d9


          Remove build_dataloader refactor

a8a2645


          Remove max_mds_writer_workers arg. It is not necessary for writing to…

… local folder.

karan6181 reviewed

View reviewed changes

Contributor

karan6181 left a comment

LGTM. Thank You!


          Merge branch 'main' into convert_mds_script

1fa9797

dakinggg approved these changes

View reviewed changes

Collaborator

dakinggg left a comment

LGTM! Lets start trying it out!

irenedea merged commit c308d10 into mosaicml:main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet