Documentation Improvements #745
Conversation
aman-17
added
the
type/documentation
scripts/download_checkpoints.py
Outdated
| except requests.exceptions.RequestException: | ||
| continue |
There was a problem hiding this comment.
Why would you swallow these exceptions?
There was a problem hiding this comment.
Oh, because you don't expect all files to be there? Then at least catch only 404 errors.
But better yet, list the contents of the directory in one call to check what's there, instead of making six calls every time we have to check a directory.
There was a problem hiding this comment.
The website does not support directory listing, so I had to manually look for ("config.yaml", "model.pt", "optim.pt", "train.pt", "model.safetensors", "optim.safetensors")
scripts/download_checkpoints.py
Outdated
| parser.add_argument('--save-dir', type=str, default='./checkpoints', | ||
| help='Base directory to save downloaded checkpoints') | ||
| parser.add_argument('--step', type=str, help='Specific step number to download (optional)') | ||
| parser.add_argument('--list-steps', action='store_true', help='List available step numbers and exit') |
There was a problem hiding this comment.
If you have a tool that can perform multiple different actions, use subcommands.
There was a problem hiding this comment.
I think Dirk is referring to this: https://docs.python.org/3/library/argparse.html#sub-commands .
| @@ -58,7 +55,7 @@ The core models in the OLMo family released so far are (all trained on the [Dolm | |||
| URLs to checkpoints at intermediate steps of the models' trainings can be found in the csv files under [`checkpoints/official/`](https://github.com/allenai/OLMo/blob/main/checkpoints/official). These 'directory' URLs cannot currently be directly accessed, but files within the directory are publicly accessible. These URLs can also be provided to the training script to resume training from the checkpoint (see [Training](#training)). Each checkpoint directory consists of: | |||
|
|
|||
| - `config.yaml`: the config at that training step. | |||
| - `model.pt`, `optim.pt`, `train.pt`: model, optimizer and training state at that training step. | |||
| - `model.safetensors`, `optim.safetensors`, `train.pt`: model, optimizer and training state at that training step. | |||
There was a problem hiding this comment.
train.safetensors? Also, for the original model we just have *.pt so we should have that format mentioned somewhere.
There was a problem hiding this comment.
We are going to save in .safetensors starting from OLMo-2
scripts/download_checkpoints.py
Outdated
| parser.add_argument('--save-dir', type=str, default='./checkpoints', | ||
| help='Base directory to save downloaded checkpoints') | ||
| parser.add_argument('--step', type=str, help='Specific step number to download (optional)') | ||
| parser.add_argument('--list-steps', action='store_true', help='List available step numbers and exit') |
There was a problem hiding this comment.
I think Dirk is referring to this: https://docs.python.org/3/library/argparse.html#sub-commands .
Documentation Improvements
Changes Made
scripts/download_checkpoints.pyto automate checkpoint downloads.scripts/train.py.New Features
The new
scripts/download_checkpoints.pyscript: