Knowledge distillation for vision guide

[merveenoyan]

This is a draft PR that I opened in the past on KD guide for CV, but I accidentally removed my fork. I prioritized TGI docs so this PR might stay stale for a while, I will ask for a review after I iterate over comments left by @sayakpaul in my previous PR. (Mainly training MobileNet with random initial weights and not with pre-trained weights from transformers)

[merveenoyan]

@sayakpaul I changed the setup and didn't observe a lot of difference, but I felt like it would be still cool to show how to distill a model. WDYT?

[amyeroberts]

cc @rafaelpadilla for reference

[rafaelpadilla]

Fantastic to see knowledge distillation being discussed—such an exciting topic! 🚀
Just shared a few comments and suggestions that might enhance readability. Most are related to writing style.
I appreciate the straightforward example you've provided. 👍

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[merveenoyan]

@rafaelpadilla @NielsRogge can we merge this if this looks good?

[rafaelpadilla]

@rafaelpadilla @NielsRogge can we merge this if this looks good?

Yes, it's OK to me.
My comments were merely about writing style

[NielsRogge]

This sentence is actually not true, ResNet and MobileNet each have their own image processors

[merveenoyan]

They do return the same thing because processor just does preprocessing on same resolution. Check this out.

from transformers import AutoFeatureExtractor
from PIL import Image
import requests
import numpy as np

teacher_extractor = AutoFeatureExtractor.from_pretrained("microsoft/resnet-50")
student_extractor = AutoFeatureExtractor.from_pretrained("google/mobilenet_v2_1.4_224")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
sample = Image.open(requests.get(url, stream=True).raw)

np.array_equal(teacher_extractor(sample),student_extractor(sample))
# True

[NielsRogge]

Thanks for writing this up! ❤️

[MKhalusova]

Great work on the guide! While reading it I had a few questions that I feel other folks may have, and it would be great to address them :)

[MKhalusova]

Would be cool to link KL loss to some page that gives a definition of what that is for people who are not familiar.

[MKhalusova]

Since you're customizing the Trainer, it would also be nice to link to this page https://huggingface.co/docs/transformers/en/main_classes/trainer#trainer

[MKhalusova]

The first sentence would be great to have somewhere in the introduction - how the distillation works. Something like: "To distill knowledge from one model to another, we take a pre-trained teacher model, and randomly initialize a student model. Next, we train the student model to minimize the difference between its outputs and the teacher's outputs, thus making it mimic the behavior. "

[MKhalusova]

Maybe also push the final model to hub?
trainer.push_to_hub()?

[merveenoyan]

I think the final model is pushed already when we set push_to_hub to True (I also have save strategy enabled for every epoch so it's triggered every epoch as well), no?

[MKhalusova]

AFAIK trainer.push_to_hub() also creates a basic model card, e.g. with metrics, and some training results.

[MKhalusova]

Thank you for iterating on this! This revision looks fantastic :)

[rafaelpadilla]

Looks good to me. :)

[merveenoyan]

@LysandreJik can you give a review or ask for another reviewer if needed?

[LysandreJik]

Thank you @merveenoyan!

[LysandreJik]

Please resolve the merge conflicts and merge @merveenoyan