Fix the structure of images in PixtralProcessor

[Rocketknight1]

The PixtralProcessor has some issues regarding correct nesting depth of inputs. If we are fully explicit about input nesting, then text should be List[str] and images should be List[List[Image]]. This is because each sample only has one text, but can have multiple images.

The start of the Pixtral processor code handles cases when users don't supply fully nested inputs. However, it uses the heuristic that if the user passes List[str] for text and List[Image] for images, then this indicates one image per sample. This is very confusing for users, especially because it means output changes when batch_size==1, depending on whether that single input is passed as str or [str].

With this PR, we avoid that assumption. If the user supplies a single image or flat list of images, then we assign them all to the first sample in the event that batch_size==1. If batch_size>1 then we throw an error, asking them to supply an explicit list of lists instead. This seems much safer!

[Rocketknight1]

cc @yurkoff-mv, can you try out this PR and confirm it resolves the issues you were having? You can use it with pip install git+https://github.com/huggingface/transformers.git@pixtral_processor_structure_fix

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Rocketknight1]

also cc @zucchini-nlp and @ArthurZucker / @LysandreJik for core maintainer review

[yurkoff-mv]

@Rocketknight1, thank you for responding so quickly. Yes, these fixes work.

[ArthurZucker]

Cool, the new errors have to be tested and the doc should mention this to remove confusion!

[zucchini-nlp]

i am wondering, is there any specific reason to support such a complex processing if at the end we end up flattening it in the modeling code. I think we should support both flat list/batch list as inputs similar to most other VLMs without trying to batch-list inputs before passing them to processors

It can help us in the long run to get standardized processors

[Rocketknight1]

Hi @zucchini-nlp, how do the inputs differ here compared to most other processors? I thought text = List[str] and images = List[List[Image]] was pretty normal!

[zucchini-nlp]

@Rocketknight1 usually we support both formats, as List[List[image]] and simply List[image], except for some cases like in Idefics where the cross attention module needs to know how many images each prompt should attend to. Maybe we can also ask @yonigozlan, he's working on standardizing processor input/output format for VLMs

[Rocketknight1]

Closing because this has been included in #34801 instead