Fall back to slow image processor in ImageProcessingAuto when no fast processor available #34785

yonigozlan · 2024-11-18T16:49:44Z

What does this PR do?

Refactor parts of image_processing_auto to fall back on slow processor when use_fast is set to True and no fast processor is available.

Before, this would throw an error:

processor = AutoImageProcessor.from_pretrained("Salesforce/blip-image-captioning-large", use_fast=True)

Now the following warning is displayed

`use_fast` is set to `True` but the image processor class does not have a fast version. Falling back to the slow version.

Also add warnings to start rolling out fast image processor by default (goal is v4.48). If use_fast is not set, and the checkpoint was saved with a slow processor, display:

Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

HuggingFaceDocBuilderDev · 2024-11-18T17:17:48Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

src/transformers/models/auto/image_processing_auto.py

yonigozlan · 2024-11-18T17:58:41Z

src/transformers/models/vit/image_processing_vit_fast.py

@@ -246,6 +246,7 @@ def preprocess(
        image_mean = image_mean if image_mean is not None else self.image_mean
        image_std = image_std if image_std is not None else self.image_std
        size = size if size is not None else self.size
+        return_tensors = "pt" if return_tensors is None else return_tensors


Otherwise, the fast vit image processor will crash in the default behavior (when return_tensors is not specified). This is now a bigger problem with fast image processors used by default.

Is this kind of breaking change?

(I am also curious what if a user is using TF/Flax model while their environment has torch/torchvision installed. Is the fast image processor will be used by default and will return torch tensor by default ..?)

Oh I see, yes this might be a problem. In general, I'm not too sure why it was decided to constrain fast image processors to output only torch tensors. Would be glad to know if there was a reason for that, otherwise it might be something we would want to reconsider.

TBH I am not even sure TF is used for image models! Fine by me!

qubvel · 2024-11-18T20:30:51Z

Hi @yonigozlan! Thanks for opening a PR! Please check this comment regarding enabling use_fast=True by default. The main concern was that fast and slow image processors might be not equivalent, so instead of forcing fast image processors we notify users that fast image processor is available. WDYT?

yonigozlan · 2024-11-19T03:33:05Z

Oh, I hadn't seen this, thanks for pointing it out @qubvel!
Indeed there are tiny differences between fast and slow image processors outputs, which could be confusing for users as it will result in minor differences in model outputs (not necessarily worse though). My main concern is that most users will ignore or won't notice the warning as nothing is going clearly wrong during inference, but they will miss out on using fast processors with our current vision models. Since slow processors are a major speed bottleneck for many of them, this seems like a missed opportunity. Also it looks like there are currently no way to use fast processors in pipelines, whereas this PR would enable their used there too (I still have to check this).

I guess it comes down to prioritizing backward compatibility or inference speed. Maybe we could add a warning_once in this PR, stating that while a slow processor was saved, Transformers v4.x automatically loads fast processors, so users might see minor output differences, and insist on the gains in inference speed. Something like:

A slow processor was saved with this model. However, since Transformers v4.x, we automatically load fast processors for improved inference speed. 
This may result in minor differences in outputs. To force the use of a slow processor, set `use_fast=False`.

This way, users are aware of the potential difference but also understand the benefit, and they can still choose to use the slow processor if needed.

Happy to discuss all of this @qubvel @ydshieh @molbap.

ydshieh

Thanks!

Have some comments as a first review.

src/transformers/models/auto/image_processing_auto.py

ydshieh · 2024-11-19T10:10:38Z

src/transformers/models/vit/image_processing_vit_fast.py

@@ -246,6 +246,7 @@ def preprocess(
        image_mean = image_mean if image_mean is not None else self.image_mean
        image_std = image_std if image_std is not None else self.image_std
        size = size if size is not None else self.size
+        return_tensors = "pt" if return_tensors is None else return_tensors


Is this kind of breaking change?

(I am also curious what if a user is using TF/Flax model while their environment has torch/torchvision installed. Is the fast image processor will be used by default and will return torch tensor by default ..?)

ydshieh · 2024-11-19T10:13:59Z

src/transformers/models/detr/image_processing_detr_fast.py

@@ -863,6 +863,7 @@ def preprocess(
            input_data_format = infer_channel_dimension_format(images[0])
        if input_data_format == ChannelDimension.LAST:
            images = [image.permute(2, 0, 1).contiguous() for image in images]
+            input_data_format = ChannelDimension.FIRST


I guess this is a bug and here you fix it right?

Yes it was, sorry for the lack of explanation. It's actually now fixed by another PR

qubvel

Fast image processing is indeed a compelling feature that could benefit many users. It looks similar to setting the default to SDPA over eager attention. While I haven’t noticed complaints about making SDPA the default, there were issues with CI breaking in different projects after SDPA was added to popular models like CLIP and Siglip.

For users experiencing slow processing speeds, exploring the option to set use_fast=True might be worthwhile. However, for those not facing performance challenges, it may not be necessary.

I don’t have strong objections to this feature, but I suggest the following approach to implement it effectively:

Add tests for thorough comparison between Slow and Fast image processors to ensure nothing breaks, especially for custom processors with non-default parameters. Tests should cover all Slow and Fast image processors (maybe a common tests?).
Inform users well ahead of time about the upcoming behavior change. For instance, announce that Fast image processors will become the default in 3–5 releases. Users who wish to retain the current behavior should explicitly set use_fast=False in their code.

This gradual rollout can minimize potential issues while allowing users time to adapt. What do you think?

yonigozlan · 2024-11-19T16:17:10Z

@qubvel Agreed it might be better to do a progressive rollout and add more base tests in the meantime. I can change this PR to do that and also keep some of the refactoring of image_processing_auto in this PR as I feel there was some bits of confusing code (especially regarding not separating image_processor_type and image_processor_class).
I can also add support for fast image processors to output other format than torch tensors, unless there is any objection against that.

ydshieh · 2024-11-19T16:29:31Z

I can also add support for fast image processors to output other format than torch tensors, unless there is any objection against that.

Considering the usage of TF/Flax, I am not sure if it is worth the effort (if it will require more than expected effort). You can talk to core maintainers about this part.

qubvel · 2024-11-19T18:26:37Z

@qubvel Agreed it might be better to do a progressive rollout and add more base tests in the meantime. I can change this PR to do that and also keep some of the refactoring of image_processing_auto in this PR as I feel there was some bits of confusing code (especially regarding not separating image_processor_type and image_processor_class).

@yonigozlan sounds great!

ArthurZucker · 2024-11-28T15:11:18Z

Inform users well ahead of time about the upcoming behavior change. For instance, announce that Fast image processors will become the default in 3–5 releases. Users who wish to retain the current behavior should explicitly set use_fast=False in their code.

I think we should start rolling this out (the warnings) and reduce to 2 releases for example (2 months) giving us time to tests!

Absolutely no need to work on TF / Jax versions for now either! ➕

yonigozlan · 2024-11-29T21:16:48Z

@ArthurZucker Changed the scope of this PR to fix the error when use_fast is set to True and no fast image processor is available. Also added warnings to start rolling out fast image processors by default.

ArthurZucker

Cool, yep let's make sure 4.48 has fast by default!

ArthurZucker · 2024-12-13T10:28:29Z

src/transformers/models/auto/image_processing_auto.py

+            if use_fast:
+                if not image_processor_type.endswith("Fast"):
+                    image_processor_type += "Fast"
+                image_processor_class = get_image_processor_class_from_name(image_processor_type)
+                if image_processor_class is None:
+                    logger.warning_once(
+                        "`use_fast` is set to `True` but the image processor class does not have a fast version. "
+                        " Falling back to the slow version."
+                    )
+                    image_processor_class = get_image_processor_class_from_name(image_processor_type[:-4])
+            else:
+                image_processor_type = (
+                    image_processor_type[:-4] if image_processor_type.endswith("Fast") else image_processor_type
+                )
+                image_processor_class = get_image_processor_class_from_name(image_processor_type)


this can be simplified a bit to check first if image_processor_type + "Fast" is in the mapping, if yes we take, if no we don't. only need to call get_image_processor_class_from_name once

ArthurZucker · 2024-12-13T10:29:18Z

src/transformers/models/vit/image_processing_vit_fast.py

@@ -246,6 +246,7 @@ def preprocess(
        image_mean = image_mean if image_mean is not None else self.image_mean
        image_std = image_std if image_std is not None else self.image_std
        size = size if size is not None else self.size
+        return_tensors = "pt" if return_tensors is None else return_tensors


TBH I am not even sure TF is used for image models! Fine by me!

yonigozlan force-pushed the set-use-fast-default branch from d80d899 to 0a130f1 Compare November 18, 2024 16:50

yonigozlan commented Nov 18, 2024

View reviewed changes

src/transformers/models/auto/image_processing_auto.py Outdated Show resolved Hide resolved

yonigozlan commented Nov 18, 2024

View reviewed changes

yonigozlan requested review from ydshieh, Rocketknight1 and qubvel November 18, 2024 18:05

qubvel added Vision Processing labels Nov 18, 2024

ydshieh reviewed Nov 19, 2024

View reviewed changes

qubvel reviewed Nov 19, 2024

View reviewed changes

yonigozlan mentioned this pull request Nov 21, 2024

Add optimized PixtralImageProcessorFast #34836

Merged

5 tasks

yonigozlan force-pushed the set-use-fast-default branch from c1283fc to 95a508b Compare November 29, 2024 21:05

yonigozlan changed the title ~~Use fast image processors by default with ImageProcessingAuto~~ Fall back to slow image processor in ImageProcessingAuto when no fast processor available Nov 29, 2024

yonigozlan requested a review from ArthurZucker November 29, 2024 21:16

yonigozlan force-pushed the set-use-fast-default branch from de5613b to f107a66 Compare December 12, 2024 16:52

ArthurZucker approved these changes Dec 13, 2024

View reviewed changes

yonigozlan added 6 commits December 15, 2024 18:43

refactor image_processing_auto logic

9415a4f

fix fast image processor tests

7454c6d

Fix tests fast vit image processor

7ecf0f9

Add safeguard when use_fast True and torchvision not available

aa9b213

change default use_fast back to None, add warnings

a90f537

remove debugging print

94a9c23

call get_image_processor_class_from_name once

7c066f2

yonigozlan force-pushed the set-use-fast-default branch from 95d039a to 7c066f2 Compare December 15, 2024 18:43

yonigozlan merged commit 5615a39 into huggingface:main Dec 15, 2024
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fall back to slow image processor in ImageProcessingAuto when no fast processor available #34785

Fall back to slow image processor in ImageProcessingAuto when no fast processor available #34785

yonigozlan commented Nov 18, 2024 •

edited

Loading

HuggingFaceDocBuilderDev commented Nov 18, 2024

yonigozlan Nov 18, 2024

ydshieh Nov 19, 2024

yonigozlan Nov 19, 2024

ArthurZucker Dec 13, 2024

qubvel commented Nov 18, 2024 •

edited

Loading

yonigozlan commented Nov 19, 2024

ydshieh left a comment

ydshieh Nov 19, 2024

ydshieh Nov 19, 2024

yonigozlan Nov 19, 2024

qubvel left a comment

yonigozlan commented Nov 19, 2024

ydshieh commented Nov 19, 2024

qubvel commented Nov 19, 2024

ArthurZucker commented Nov 28, 2024

yonigozlan commented Nov 29, 2024

ArthurZucker left a comment

ArthurZucker Dec 13, 2024

ArthurZucker Dec 13, 2024

Fall back to slow image processor in ImageProcessingAuto when no fast processor available #34785

Fall back to slow image processor in ImageProcessingAuto when no fast processor available #34785

Conversation

yonigozlan commented Nov 18, 2024 • edited Loading

What does this PR do?

Who can review?

HuggingFaceDocBuilderDev commented Nov 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qubvel commented Nov 18, 2024 • edited Loading

yonigozlan commented Nov 19, 2024

ydshieh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qubvel left a comment

Choose a reason for hiding this comment

yonigozlan commented Nov 19, 2024

ydshieh commented Nov 19, 2024

qubvel commented Nov 19, 2024

ArthurZucker commented Nov 28, 2024

yonigozlan commented Nov 29, 2024

ArthurZucker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yonigozlan commented Nov 18, 2024 •

edited

Loading

qubvel commented Nov 18, 2024 •

edited

Loading