ViT-L-14-336 fine-tuning failed

[zhaozhipeng1997]

I failed when fine-tuning ViT-L-14-336, and the val loss kept rising. The fine-tuning script is as follows:
PYTHONPATH=/home/zzp/code/open_clip/src python -m open_clip_train.main
--save-frequency 10
--zeroshot-frequency 10
--report-to tensorboard
--train-data /home/zzp/4t_ssd/annotations_train.csv
--val-data /home/zzp/4t_ssd/annotations_test.csv
--csv-img-key filepath
--warmup 1000
--csv-caption-key caption
--batch-size=16
--lr=1e-3
--epochs=50
--wd 0.1
--aug-cfg scale='(0.5, 1.0)'
--workers=32
--model ViT-L-14-336
--precision amp
--pretrained=openai

[rwightman]

@zhaozhipeng1997 unfortunately I can't help debug training issues outside of familiar datasets. Can put this in the discussion forum but you'd probably need to share details of your dataset.

I can only realistically help if you can reproduce issues with datasets like

• https://huggingface.co/datasets/pixparse/cc12m-wds
• https://huggingface.co/datasets/pixparse/cc3m-wds

[zhaozhipeng1997]

Thanks for your reply. My data is as follows:
filepath,label
/path/img1.jpg, A photo of scissors.
/path/img2.jpg, A photo of hand.
The image size is 224*224.