Code for computing visual-to-instruction information flow

[ErikZ719]

Hello author, I am now trying to get the qualitative results of Figure 1 in the paper: visualization of information flow.

Thank you very much for providing a pseudo code for computing visual-to-instruction information flow in the appendix. Would you be able to provide the full visualization code, which would be helpful for me to delve into this exciting work. Thank you.

[xing0047]

Hi @ErikZ719

Thanks for pointing this out. Code for computing visual-to-instruction information flow is available. Please run script below for a quick glance.

bash scripts/v1_5/eval/eval.cca-llava-1.5-7b.pope.coco.adv.if_vis.sh

The visualisation of information flow will be saved under outputs/pope/coco.

For details of computing this information flow, please check line 773 ~ 777 in cca.py, where text_image_if is the visualised variable.

Hope these help your further explorations.

[SakuraTroyChen]

Why do I get results that are opposite to yours when using the method you provided?

[xing0047]

Hi @SakuraTroyChen

I inference with the script above and it seems work fine from my side. Would you share more details on this so that we may further investigate?

[SakuraTroyChen]

Sorry. I made a terrible mistake.

[ErikZ719]

@xing0047 ,Thank you very much. Have a good day! 👍

[xing0047]

Hi @SakuraTroyChen

It is okay. Happy to see your issue has been addressed. Feel free to open discussion if you have further concern on this visualisation.

[xing0047]

Hi @ErikZ719

Glad I have helped on this. Have a good day to you too!

Feel free to re-open this issue if further clarifications are needed.