Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use real latest checkpoint in training #2564

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Use real latest checkpoint in training #2564

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions nnunetv2/run/run_training.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,15 +74,18 @@ def maybe_load_checkpoint(nnunet_trainer: nnUNetTrainer, continue_training: bool
raise RuntimeError('Cannot both continue a training AND load pretrained weights. Pretrained weights can only '
'be used at the beginning of the training.')
if continue_training:
expected_checkpoint_file = join(nnunet_trainer.output_folder, 'checkpoint_final.pth')
if not isfile(expected_checkpoint_file):
expected_checkpoint_file = join(nnunet_trainer.output_folder, 'checkpoint_latest.pth')
# special case where --c is used to run a previously aborted validation
if not isfile(expected_checkpoint_file):
expected_checkpoint_file = join(nnunet_trainer.output_folder, 'checkpoint_best.pth')
if not isfile(expected_checkpoint_file):
print(f"WARNING: Cannot continue training because there seems to be no checkpoint available to "
f"continue from. Starting a new training...")
checkpoint_files = [
join(nnunet_trainer.output_folder, 'checkpoint_final.pth'),
join(nnunet_trainer.output_folder, 'checkpoint_latest.pth'),
join(nnunet_trainer.output_folder, 'checkpoint_best.pth'),
]
# Filter out the files that actually exist
existing_checkpoints = [ckpt for ckpt in checkpoint_files if isfile(ckpt)]
if existing_checkpoints:
# Select the checkpoint with the most recent modification time
expected_checkpoint_file = max(existing_checkpoints, key=os.path.getmtime)
else:
print("WARNING: Cannot continue training because there seems to be no checkpoint available to continue from. Starting a new training...")
expected_checkpoint_file = None
elif validation_only:
expected_checkpoint_file = join(nnunet_trainer.output_folder, 'checkpoint_final.pth')
Expand Down