Guarin lig 3056 add mae imagenet benchmark

[guarin]

Closes #1255

Reference values from paper:

The paper uses ViT-L with 1600 epoch pretraining by default while we use only ViT-B with 100 epoch pretraining. So I would expect finetune performance to be <82.3% following Figure 7. solo-learn reports 81.6% finetune top1 for ViT-B after 100 epochs. The paper reports ViT-B finetune performance of 83.6% after 1600 epoch pretraining.

From Figure 7 it looks like linear eval should be <57.3 as we use ViT-B instead of ViT-L. ViT-B linear eval after 1600 epoch pretraining is reported as 68% in Table 12.

First Results

Imagenet
    max val_online_cls_top1: 0.17285999655723572                                                                                                                                                                                                                    
    max val_online_cls_top5: 0.36941999197006226 
    knn val_top1: 0.13728000223636627                                                                                                                                                                                                                               
    knn val_top5: 0.2959200143814087 
    max linear val_top1: 0.4907799959182739                                   
    max linear val_top5: 0.728380024433136
    max finetune val_top1: 0.7822399735450745                                 
    max finetune val_top5: 0.9412800073623657

Results are a bit worse than expected. We should get ~82% finetune top1 but got 78.2%. Similarly for linear top1 we should get something a bit below 57% but we only got 49%.

Interestingly, knn top1 is really bad. Could be a sign that pretraining doesn't work properly or that (linear) finetuning is very important.

[codecov]

Codecov Report

Attention: 12 lines in your changes are missing coverage. Please review.

Comparison is base (889825a) 85.25% compared to head (e27946e) 85.79%.

Files	Patch %	Lines
lightly/models/utils.py	83.78%	6 Missing ⚠️
lightly/models/modules/masked_autoencoder.py	84.84%	5 Missing ⚠️
lightly/utils/benchmarking/metric_callback.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1263      +/-   ##
==========================================
+ Coverage   85.25%   85.79%   +0.54%     
==========================================
  Files         130      130              
  Lines        5519     5547      +28     
==========================================
+ Hits         4705     4759      +54     
+ Misses        814      788      -26     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[guarin]

Closing in favor of: #1468