Support Kosmos-2.5

[tic-top]

What does this PR do?

#30877 Implementation of Kosmos-2.5 in transformers.
https://huggingface.co/kirp/kosmos2_5/blob/main/README.md

Usage

from PIL import Image
import requests
import torch
from transformers import AutoProcessor, AutoModelForVision2Seq, AutoConfig
import re

repo = "kirp/kosmos2_5"
device = "cuda:0"
config = AutoConfig.from_pretrained(repo)

NAME = {
    "f" : "flash_attention_2",
    "s" : "sdpa",
    "e" : "eager",
}

# all sdpa fp16
dtype = torch.float16
config._attn_implementation = NAME["s"]
config.vision_config._attn_implementation = NAME["s"]
config.text_config._attn_implementation = NAME["s"]

# # all sdpa fp16
# dtype = torch.float16
# config._attn_implementation = NAME["s"]
# config.text_config._attn_implementation = NAME["s"]
# config.vision_config._attn_implementation = NAME["s"]

# # all eager bf16
# dtype = torch.bfloat16
# config._attn_implementation = NAME["e"]
# config.text_config._attn_implementation = NAME["e"]
# config.vision_config._attn_implementation = NAME["e"]


model = AutoModelForVision2Seq.from_pretrained(repo, device_map = device, torch_dtype=dtype, config=config)
processor = AutoProcessor.from_pretrained(repo)

url = "https://huggingface.co/kirp/kosmos2_5/resolve/main/receipt_00008.png"
image = Image.open(requests.get(url, stream=True).raw)
prompt = "<ocr>" # <md>

inputs = processor(text=prompt, images=image, return_tensors="pt")
height, width = inputs.pop("height"), inputs.pop("width")
raw_width, raw_height = image.size
scale_height = raw_height / height
scale_width = raw_width / width

inputs = {k: v.to(device) if v is not None else None for k, v in inputs.items()}
inputs["flattened_patches"] = inputs["flattened_patches"].to(dtype)

generated_ids = model.generate(
    **inputs,
    max_new_tokens=1024,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)

def postprocess(y, scale_height, scale_width):
    y = y.replace(prompt, "")
    if "<md>" in prompt:
        return y
    pattern = r"<bbox><x_\d+><y_\d+><x_\d+><y_\d+></bbox>"
    bboxs_raw = re.findall(pattern, y)
    lines = re.split(pattern, y)[1:]
    bboxs = [re.findall(r"\d+", i) for i in bboxs_raw]
    bboxs = [[int(j) for j in i] for i in bboxs]
    info = ""
    for i in range(len(lines)):
        box = bboxs[i]
        x0, y0, x1, y1 = box
        if not (x0 >= x1 or y0 >= y1):
            x0 = int(x0 * scale_width)
            y0 = int(y0 * scale_height)
            x1 = int(x1 * scale_width)
            y1 = int(y1 * scale_height)
            info += f"{x0},{y0},{x1},{y0},{x1},{y1},{x0},{y1},{lines[i]}"
    return info

output_text = postprocess(generated_text[0], scale_height, scale_width)
print(output_text)

[amyeroberts]

cc @ydshieh

[zucchini-nlp]

Thanks a lot! ❤️ Just a few more things that can be removed now but looks nice overall to me. Looking forward for changes to accommodate generate() from mixin and should be okay

[zucchini-nlp]

hmm i've been removing deprecation for new models. Oke, if core maintainer agrees we can leave but maybe change the deprecation version to around v4.50 or smth?

[zucchini-nlp]

nit: these can be saved as special tokens in tokenizer so you can do tokenizer.boi_token_id instead of hardcoding the string of each token. Not super necessary, simply FYI :)

for that when converting you need to add

# bos/eos/pad tokens are already in tokenizer so no need to add them
extra_special_tokens = {"boi_token": "<image>", "eoi_token": </image>}
tokenizer = AutoTokenizer.from_pretrained(model_id, extra_special_tokens=extra_special_tokens)

[ydshieh]

Hi @zucchini-nlp Great reviews! Could you check the changes shown below? I think all comments addressed.

(and also the removed generate)

https://github.com/tic-top/transformers/compare/0ec499a841b35ce9c77e072362a09a20a287a2f5..8fc9699655c5b80b0c5cbb1fadfc4837daf1ad90

[zucchini-nlp]

Thanks a lot for iterating and removing the overwritten generate()!

[ydshieh]

run slow

[github-actions]

This comment contains run-slow, running the specified jobs: ['models/kosmos2_5'] ...