DropInViewer Three scene always in foreground

[UX3D-haertl]

I'm currently trying to get the DropInViewer running together with a normal three.js scene.
If I use the normal Viewer class and specify the scene via the threeScene parameter, everything is working as intended. But when I use the DropInViewer and add it to scene, I can see the splat and the normal 3D mesh, but the splat is always rendered underneath the normal mesh independend of the camera position. In both cases the splat is drawn last. Do you have to provide more parameters to the DropInViewer, or compute the depth of the splat primitives?

This is basically the same issue as #5, but for the DropInViewer.

Here is the code I'm using. (I'm using three fiber):

// Working viewer
useEffect(() => {
    const viewer2 : THREE.Object3D<THREE.Object3DEventMap>= new GaussianSplats3D.Viewer({
      'gpuAcceleratedSort': false,
      'sharedMemoryForWorkers': false,
      'threeScene': scene,
    });
    viewer2.addSplatScenes([{'path' : 'garden_high.ksplat'}])
    .then(() => {
      viewer2.start();
    });
  }, [scene]);

// Depth missing
useEffect(() => {
    const viewer : THREE.Object3D<THREE.Object3DEventMap>= new GaussianSplats3D.DropInViewer({
      'gpuAcceleratedSort': false,
      'sharedMemoryForWorkers': false,
    });
    scene.add(viewer);
    viewer.addSplatScenes([{'path' : 'garden_high.ksplat'}]);
}, [scene]);

[mkkellogg]

Is your normal three.js mesh transparent? If so, you'll need to tell three.js how to explicitly order them relative to the splats for rendering. since the mesh containing the splats is also transparent. You can try setting the render order of the splats by doing something like:

   // `viewer.viewer` is not a typo :) 
   viewer.viewer.splatMesh.renderOrder = 1000;

The Viewer class explicitly renders the threeScene you pass to it before rendering the splats, so it has control over rendering order.

It's definitely worth noting that mixing normal transparent three.js objects with splats, in general, will not look correct because the splats do not render to the depth buffer (as is typical with transparent objects)

[UX3D-haertl]

The mesh is fully opaque. I also checked the drawing order in specter.js which is identicaly to the Viewer class.
I attached a video. It probably has something to do with syncing the cameras. We do modify the camera settings in our viewer.
In the video you can see that at some point the splat data is obstructing the mesh.
Do I have to sync the camera movements somehow?

2024-11-19.10-54-25.mov

[UX3D-haertl]

I debugged more into it but it seems like the correct camera is used for the splat sorting. I'm currently not really sure what the DropInViewer does differently than the normal Viewer.

[UX3D-haertl]

We managed to find the issue:
We had logarithmicDepthBuffer set to true, but this is not propagated to the drop in viewer.

It is set in this line, but parameters is currently always empty:

GaussianSplats3D/src/three-shim/WebGLCapabilities.js

Line 69 in f6f4ade

const logarithmicDepthBuffer = parameters.logarithmicDepthBuffer === true;

[mkkellogg]

Ah, that makes sense. It looks like this will require an update to multiple parts of the code. First I think a Viewer parameter will be needed to tell it to use a logarithmic depth buffer. Then this line will need to be updated to pass that parameter to the class you mentioned above:

GaussianSplats3D/src/splatmesh/SplatMesh.js

Line 1389 in f6f4ade

const capabilities = new WebGLCapabilities(gl, extensions, {});

The new Viewer parameter will also need to be passed to the splat materials (SplatMaterial, SplatMaterial2D, and SplatMaterial3D) so they can conditionally add the necessary shader code.

Would you be interested in a submitting a PR with all of that?

[Yancykeiro]

Is there a plan to resolve this issue at some point?