Fix incorrect distance given by ViewRangefinder3d

[floppyhammer]

Objective

Fixes #11608.

Solution

This PR makes ViewRangefinder3d calculate object-to-camera distance correctly in order to sort them properly.

Migration Guide

In sturct ViewRangefinder3d, from_view_matrix has been renamed to from_view_proj_matrix.

[IDEDARY]

This PR is a major improvement and almost fixes #11608 issue. Under normal circumstances, the bug is gone, but I noticed the same behavior happens under extreme angles.

Works as expected now

But if the camera's direction vector is pointing in the direction of the plane's normal vector while the plane is still visible, the order issue is still there. (note that the middle plane is at 0,0,0, so if we see it's back that means we are looking in the direction of the plane's normal)

[floppyhammer]

But if the camera's direction vector is pointing in the direction of the plane's normal vector while the plane is still visible, the order issue is still there. (note that the middle plane is at 0,0,0, so if we see it's back that means we are looking in the direction of the plane's normal)

Sorry I don't really get what you mean. How the plane can be visible if its front face points outward from the camera?

[IDEDARY]

Sorry I don't really get what you mean. How the plane can be visible if its front face points outward from the camera?
Ah, I will showcase it here:

These diagrams are showcasing the situation from my example above in birds perspective (top down)...

As you can see, the issue was fixed under this condition (thanks so much btw, great work. Much appreciated 👍🏻 )

But in the second picture I show this situation...

It breaks things, because if not the top rectangle should have been darkened like in the first image, because there is a black square on top of it with 60% opacity.

That sums up my review.
I would 100% merge this PR (unless it breaks other stuff), but I would not consider the issue fixed.
Great job 👍🏻

[floppyhammer]

@IDEDARY Okay I get it.

[floppyhammer]

@IDEDARY Why the plane positions along X and Y are different?

For the two planes having order issue, their positions are as follows.

Plane 1 Vec3(9.0, -182.5, -170.0)
Plane 2 Vec3(-2.5, 202.5, -140.0)

From the render result, I suppose the two planes should have different Z coordinate only.

[floppyhammer]

@IDEDARY Considering the 3d/texture example is all good at any camera angle, it's very possible the issue has something to do with how you place the UI panels in your layouting system.

[IDEDARY]

I don't believe so, but I will carry out further investigation and try to create a minimal recreation project without my layouting. It should give us some insight into the origin of the problem.

Im unsure about the different cords, but I use parenting. Quads are defined through center point and they aren't perfectly centered either so it might come from that. But I will also look into it.

[IDEDARY]

@IDEDARY Considering the 3d/texture example is all good at any camera angle, it's very possible the issue has something to do with how you place the UI panels in your layouting system.

@floppyhammer I have created bare-bones reproducible (roughly 100 lines of code) without any of my layouting code present. It's the same thing, but the abstraction is removed. The issue is unfortunately still present :/

[floppyhammer]

@IDEDARY I'll look into it.

[floppyhammer]

I'm closing this as this solution is not good enough. We might need a more sophisticated solution (e.g. BSP trees) to handle the order issue.