Add CogVLM (cleaner) #28196
Add CogVLM (cleaner) #28196
Conversation
| return output | ||
|
|
||
|
|
||
| class CogvlmRotaryEmbedding(torch.nn.Module): |
| # and we need to add 1 to take into account the one extra token that is going to | ||
| # be sent through the model | ||
| position_ids = build_position_ids(token_type_ids, attention_mask) + 2 + 1 | ||
| position_ids = position_ids[:, -1:] |
There was a problem hiding this comment.
Would be nice if we make this dependent on input's seq length, instead of assuming 1. For cases of speculative decoding, when it starts working for VLMs
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
|
Hi @amyeroberts would be great if you could review this PR, quite some updates were made, making it ready for review. |
|
Let's have a first round of reviews from the people already familiar with the model and PR :) cc @zucchini-nlp |
| # import xformers.ops as xops | ||
|
|
||
| # out = xops.memory_efficient_attention( | ||
| # queries, | ||
| # keys, | ||
| # values, | ||
| # scale=self.scale, | ||
| # ) | ||
|
|
||
| # output = self.dense(out.view(batch_size, sequence_length, -1)) | ||
| # output = self.output_dropout(output) | ||
| # return output | ||
|
|
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
What does this PR do?
This PR adds CogVLM, in a cleaner way. Follow-up of #27718.
Debugging logits (for branch with matching logits see
add_cogvlm_cleaner_with_matching_logits):self.weight * hidden_states.to(input_dtype)instead of(self.weight * hidden_states).to(input_dtype).EagerAttentionbesidesSdpaAttention(only latter gives matching logits)