Add resume for adapter_v2, enable continued finetuning for adapter #1354
+40
−5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
altria-zewei-wang
requested review from
awaelchli,
carmocca and
lantiga
as code owners
rasbt
approved these changes
Apr 25, 2024
| @@ -16,12 +16,12 @@ | |||
| from torchmetrics import RunningMean | |||
|
|
|||
| from litgpt.adapter_v2 import GPT, Block, Config, adapter_filter, mark_only_adapter_v2_as_trainable | |||
| from args import EvalArgs, TrainArgs | |||
| from litgpt.args import EvalArgs, TrainArgs | |||
There was a problem hiding this comment.
This is also how we import elsewhere and looks good to me.
|
Thanks for looking into this. Sorry, I haven't spent much time on thinking through the ramifications here. But would the simple resuming from the https://github.com/Lightning-AI/litgpt/blob/main/litgpt/finetune/full.py#L43 |
|
I was specifically looking into testing finetuning with adapter and loras in my paper, and that my gpu cuts off after certain time limit. I figure adding this feature can help anyone who is in similar situation as me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi all!
I was checking #238 to add a function to finish resume finetuning for adapter. It would search the largest number of step point it saved from out_dir and update the state_dict.
Current Problem: I updated the step_count but find out to keep the iteration count from last time would have to read in the metrics in the log folder. The problem is that I don't know how to retrieve the corresponding version in the log file without adding an input of the version of the metrics.csv (currently not implemented).
Let me know what you think! Thanks for your repo!