Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

save a pickle file at save_every iteration? #27

Open
wingillis opened this issue Oct 18, 2018 · 6 comments
Open

save a pickle file at save_every iteration? #27

wingillis opened this issue Oct 18, 2018 · 6 comments

Comments

@wingillis
Copy link
Contributor

Instead of saving the data at the end, can we update a pickle at every save_every iteration? It would be nice to have a saved output in case the modeling fails for some reason
@wingillis
Copy link
Contributor Author

I'm testing a new branch v0.1.2-save-model-progress that will save a pickle file at every --save-model-progress iterations. If modeling progress stopped too early, and you restart the model, the cli script will look for the checkpoint to reload all the parameters

@wingillis
Copy link
Contributor Author

Some current caveats - loading in a checkpoint means that any parameters that are changed in the CLI script will be disregarded. I don't fully know how to handle this right now.

@jmarkow
Copy link
Collaborator

jmarkow commented Dec 9, 2018

I can have a look in a few days. Thanks for taking the time to hack this together...

I think it's fine to ignore the CLI parameters if we include a warning statement.

@jmarkow
Copy link
Collaborator

jmarkow commented Jul 28, 2019

Where are we w/r/t this?

@wingillis
Copy link
Contributor Author

wingillis commented Aug 14, 2019 •
edited
Loading

This is implemented in the v0.2.0 branch but will require some io functions to properly handle the pickle files. link to the saving function here

@aymanzay
Copy link
Contributor

Is it necessary to have an additional copy of the model along with the checkpoints we currently have implemented?

aymanzay added a commit that referenced this issue May 15, 2020
@aymanzay
Updated model checkpoints, removed save_every: #27
42f9f9e 
Updated tests to test for checkpoints
Fixed testing bug: prefix=tmp
Implemented safer trans_disnts retrieval, mentioned in Issue #47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jmarkow @wingillis @aymanzay