Add CogVLM #27718
Add CogVLM #27718
Conversation
| def __init__(self, image_processor, tokenizer, image_size, patch_size): | ||
| super().__init__(image_processor, tokenizer) | ||
| self.image_size = image_size | ||
| self.patch_size = patch_size |
There was a problem hiding this comment.
cc @ydshieh for this model, I need to store 2 attributes to the processor, however we currently don't have a processor_config.json file. Can we add support for this in from_pretrained and save_pretrained?
| def __init__(self, image_processor, tokenizer, image_size, patch_size): | ||
| super().__init__(image_processor, tokenizer) | ||
| self.image_size = image_size | ||
| self.patch_size = patch_size |
There was a problem hiding this comment.
cc @ydshieh for this model, I need to store 2 attributes to the processor, however we currently don't have a processor_config.json file. Can we add support for this in from_pretrained and save_pretrained?
NielsRogge
force-pushed
the
add_cogvlm
branch
from
fc83063 to
d3c5fc3
Compare
|
A cleaner implementation I'm working on is here: https://github.com/NielsRogge/transformers/tree/add_cogvlm_cleaner. It implements the model like llava, by adding the image tokens inside the model, rather than creating them in the processor class. |
|
Closing this one in favor of the PR above. |
What does this PR do?
This PR adds CogVLM natively into the Transformers library (it's already usable with
trust_remote_code=True, but with this PR one can run it without the xformers, einops and triton dependencies).To do:
FastRotaryEmbeddingoptional)Processor#27761