About ROPE in sample process

[Leedonus]

Hi, thanks for the interesting work. I want to know why the PE of the text token in generating process is set to zero?

[zxduan90]

@daiyixiang666 But these zeros will make the q and k of the text_token zero, according to

LlamaGen/autoregressive/models/gpt.py

Line 220 in ce98ec4

xq = apply_rotary_emb(xq, freqs_cis)

, then the attention of the output token and the text token will be the same

[daiyixiang666]

Oh， I see， so the text only make effect via xv?

[zxduan90]

@daiyixiang666 yes, i think there is a problem in it

[zxduan90]

The text token has less relevance with the image compared to stable diffusion method

[daiyixiang666]

And it will also mean the text attention mask is useless?