Applying fusion to a ViT from a different package

[idankinderman]

Hi,

Let's say I want to apply your fusion method to a PyTorch-based ViT implemented in a package that isn't Hugging Face. Specifically, I want to merge ViTs from the open_clip package (https://github.com/mlfoundations/open_clip). What changes do I need to make in your code to make it work?

Thank you for your great work!

Edan

[moimfeld]

Hi,

Thanks for your interest in our work!

There are three tasks you need to do to add support for a new architecture:

• Write a top-level fusion function (like this) for your architecture, that makes calls to functions in otfusion_lib.py.
• Write an experiment YAML file (start from this), where you change the model type parameter to your new architecture.
• Add support for your new model type parameter in the architecture-specific functions. There are a few architecture-specific functions in main.py and utils.py.

In general, I suggest you to step through the experiment we provide in debug mode, to get a good idea of how our fusion pipeline works in detail before starting to add support for a new architecture.

– Moritz