Optimum parameters for eurosat.yaml

[robmarkcole]

Testing the new LightningCLI with the supplied eurosat.yaml example, running first fit then torchgeo test --config eurosat.yaml, the achieved metrics are not great and the model terminates at min_epochs

┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│   test_AverageAccuracy    │     0.113612599670887     │
│       test_F1Score        │    0.11796296387910843    │
│     test_JaccardIndex     │   0.024194467812776566    │
│   test_OverallAccuracy    │    0.11796296387910843    │
│         test_loss         │     2.301994562149048     │

I will do some experimentation, but keen to hear if anyone has better parameters to try?

[isaaccorley]

The default weights are random. I would try setting these to imagenet weights. Also increase patience to avoid the early stopping.

[robmarkcole]

Something is off. Ive set

model:
  class_path: ClassificationTask
  init_args:
    loss: "ce"
    model: "resnet18"
    lr: 1e-3
    patience: 10
    weights: true
    in_channels: 13
    num_classes: 10

Train and val metrics look good:

wandb: 
wandb: Run summary:
wandb:                 epoch 9
wandb: train_AverageAccuracy 0.98333
wandb:         train_F1Score 0.98611
wandb:    train_JaccardIndex 0.96333
wandb: train_OverallAccuracy 0.98611
wandb:            train_loss 0.03321
wandb:   trainer/global_step 1269
wandb:   val_AverageAccuracy 0.96023
wandb:           val_F1Score 0.9613
wandb:      val_JaccardIndex 0.92377
wandb:   val_OverallAccuracy 0.9613
wandb:              val_loss 0.15133

But then test are terrible:

wandb: Run summary:
wandb:                epoch 0
wandb: test_AverageAccuracy 0.15452
wandb:         test_F1Score 0.15259
wandb:    test_JaccardIndex 0.06478
wandb: test_OverallAccuracy 0.15259
wandb:            test_loss 2.32303
wandb:  trainer/global_step 0

[calebrob6]

I'm pretty sure the test command is using random weights instead of the trained weights.

…

On Thu, Oct 5, 2023, 4:17 PM Robin Cole ***@***.***> wrote: Something is off. Ive set model: class_path: ClassificationTask init_args: loss: "ce" model: "resnet18" lr: 1e-3 patience: 10 weights: true in_channels: 13 num_classes: 10 Train and val metrics look good: [image: image] <https://user-images.githubusercontent.com/11855322/272932442-53326cbf-77f9-427a-9768-697e7aeb46eb.png> [image: image] <https://user-images.githubusercontent.com/11855322/272932545-bd85ac9e-739b-4a72-951e-e0b1393ad236.png> wandb: wandb: Run summary: wandb: epoch 9 wandb: train_AverageAccuracy 0.98333 wandb: train_F1Score 0.98611 wandb: train_JaccardIndex 0.96333 wandb: train_OverallAccuracy 0.98611 wandb: train_loss 0.03321 wandb: trainer/global_step 1269 wandb: val_AverageAccuracy 0.96023 wandb: val_F1Score 0.9613 wandb: val_JaccardIndex 0.92377 wandb: val_OverallAccuracy 0.9613 wandb: val_loss 0.15133 But then test are terrible: wandb: Run summary: wandb: epoch 0 wandb: test_AverageAccuracy 0.15452 wandb: test_F1Score 0.15259 wandb: test_JaccardIndex 0.06478 wandb: test_OverallAccuracy 0.15259 wandb: test_loss 2.32303 wandb: trainer/global_step 0 — Reply to this email directly, view it on GitHub <#1620 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIJUTVR35NQYY7USUC3DDTX526PNAVCNFSM6AAAAAA5UDGZEWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TCOJZGA2DA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[adamjstewart]

Yes, you have to tell it which weights to use if you want it to use weights.

[robmarkcole]

Right like python main.py test --trainer.ckpt_path=...

[adamjstewart]

Yes, torchgeo test --trainer.ckpt_path=...

[robmarkcole]

Yes that must be the case. If I run in a notebook I get

[{'test_loss': 1.1078932285308838,
  'test_AverageAccuracy': 0.6500000357627869,
  'test_F1Score': 0.6499999761581421,
  'test_JaccardIndex': 0.5400000214576721,
  'test_OverallAccuracy': 0.6499999761581421}]

[robmarkcole]

OK for some reason EuroSATDataModule was returning the version with 100 images only. Will attempt to reproduce

[robmarkcole]

Updated, now running full dataset:

task = ClassificationTask(
    model="resnet18",
    weights=True,
    num_classes=10,
    in_channels=13,
    loss="ce", 
    patience=10
)

Achieves

[{'test_loss': 0.1498228907585144,
  'test_AverageAccuracy': 0.9574975967407227,
  'test_F1Score': 0.9581481218338013,
  'test_JaccardIndex': 0.9178630709648132,
  'test_OverallAccuracy': 0.9581481218338013}]

[robmarkcole]

I tried using the S2 weights with weights=ResNet18_Weights.SENTINEL2_ALL_MOCO, and it terminates at 15 epochs with slightly worse metrics

[{'test_loss': 0.1741539090871811,
  'test_AverageAccuracy': 0.9530659914016724,
  'test_F1Score': 0.9544444680213928,
  'test_JaccardIndex': 0.9114892482757568,
  'test_OverallAccuracy': 0.9544444680213928}]

[adamjstewart]

Could require slightly different hyperparameters to beat ImageNet. We've generally found better results when using models pre-trained on satellite imagery.

[robmarkcole]

And using the RGB bands with weights=ResNet18_Weights.SENTINEL2_RGB_MOCO yields:

[{'test_loss': 0.2393910437822342,
  'test_AverageAccuracy': 0.9335962533950806,
  'test_F1Score': 0.9353703856468201,
  'test_JaccardIndex': 0.8776867985725403,
  'test_OverallAccuracy': 0.9353703856468201}]

There is a lot of potential for optimisation, but at least initially it appears RGB do most of the work. Im interested in doing a more comprehensive eval like this and really understanding the significance of each channel