[bug]: DepthAnything has outliers

[simonfuhrmann]

Is there an existing issue for this problem?

• I have searched the existing issues

Operating system

Linux

GPU vendor

Nvidia (CUDA)

GPU model

RTX 4070

GPU VRAM

12GB

Version number

5.4.2

Browser

Chrome

Python dependencies

No response

What happened

When running DepthAnything processor (model size "base", but other model sizes are also affected), I get a depth map with many outliers. See attached image. In particular, there are white pixels (not all at 255, some have smaller values) in places where they don't make sense.

What you expected to happen

Get a depth map without outliers.

How to reproduce the problem

Create trivial workflow with "Image Primitive" -> "Depth Anything Processor" and check "save to gallery" on the depth processor.

Additional context

No response

Discord username

No response

[psychedelicious]

We run the depth model directly via transformers.

If I run the same Depth Anything v2 small model using the non-transformers implementation via this script, I do not get the artifacts.

This might be a bug with transformers: huggingface/transformers#28292

[hipsterusername]

Hey @simonfuhrmann - are these impacting generation when you use them as a control?

I’ve not found them to be super deleterious in my use, as the outlines occur on object edges, but I’ve not done explicit testing

[simonfuhrmann]

I've not been using Depth Anything for control, but Stereogram generation (https://github.com/simonfuhrmann/invokeai-stereo), and for this use-case, these artifacts are super detrimental and cause unpleasant artifacts when viewing.

It's also worth nothing that these artifacts only appear in situations where there is a large depth disparity between foreground and background (such as portraits), thus not on all scenes.

[hipsterusername]

Makes sense for the use case! As noted above, we’ll have to look upstream for specific fixes.

An alternative to look into is seeing about enhancing depth processing to a newer model like lotus.

[simonfuhrmann]

I appreciate the effort of the team to look into this.

Transformers also seems to support Depth Anything v2. Is it worth giving this a shot? This is potentially the most straightforward path to updating the depth model. There is also this issue, to it would be a feed-two-birds-with-one-scone situation.

[psychedelicious]

@simonfuhrmann We support both actually - well, only the small v2 variant due to licensing. Both have the same issue in my testing.

[simonfuhrmann]

The root cause for this seems to be bicubic interpolation which can generate values outside of the input domain and can cause an overflow. The solution is to use bilinear interpolation, or clamp before discretizing, as described here.

Is this something that can be fixed as part of InvokeAI's code?

[RyanJDick]

@simonfuhrmann Thanks for tracking down that discussion. It turns out that this has already been fixed upstream , so bumping transformers fixes the issue: #7386