Merge pull request #60 from mkkellogg/feature/dropin-viewer

Feature: Drop-in/renderable viewer
mkkellogg · Dec 1, 2023 · fdfff85 · fdfff85
2 parents fe389fb + d07d582
commit fdfff85
Show file tree

Hide file tree

Showing 16 changed files with 1,055 additions and 511 deletions.
diff --git a/README.md b/README.md
@@ -14,6 +14,10 @@ When I started, web-based viewers were already available -- A WebGL-based viewer
 - Built-in viewer is self-contained so very little code is necessary to load and view a scene
 - Allows user to import `.ply` files for conversion to custom compressed `.splat` file format
 - Allows a Three.js scene or object group to be rendered along with the splats
+- Focus on optimization:
+    - Splats culled prior to sorting & rendering using a custom octree
+    - WASM splat sort: Implemented in C++ using WASM SIMD instructions
+    - Partially GPU accelerated splat sort: Uses transform feedback to pre-calculate splat distances
 
 ## Known issues
 
@@ -86,6 +90,7 @@ The demo scene data is available here: [https://projects.markkellogg.org/downloa
 <br>
 <br>
 
+
 ## Basic Usage
 
 To run the built-in viewer:
@@ -95,102 +100,97 @@ const viewer = new GaussianSplats3D.Viewer({
     'cameraUp': [0, -1, -0.6],
     'initialCameraPosition': [-1, -4, 6],
     'initialCameraLookAt': [0, 4, 0],
-    'ignoreDevicePixelRatio': false,
-    'gpuAcceleratedSort': true
+    'halfPrecisionCovariancesOnGPU': true,
 });
 viewer.loadFile('<path to .ply or .splat file>', {
-    'splatAlphaRemovalThreshold': 5, // out of 255
-    'halfPrecisionCovariancesOnGPU': true
+    'splatAlphaRemovalThreshold': 5,
+    'showLoadingSpinner': true,
+    'position': [0, 1, 0],
+    'rotation': [0, 0, 0, 1],
+    'scale': [1.5, 1.5, 1.5]
 })
 .then(() => {
     viewer.start();
 });
+
 ```
+Viewer parameters
+<br>
+
 | Parameter | Purpose
 | --- | ---
-| `cameraUp` | The natural 'up' vector for viewing the scene. Determines the scene's orientation relative to the camera and serves as the axis around which the camera will orbit.
-| `initialCameraPosition` | The camera's initial position.
-| `initialCameraLookAt` | The initial focal point of the camera and center of the camera's orbit.
-| `ignoreDevicePixelRatio` | Tells the viewer to pretend the device pixel ratio is 1, which can boost performance on devices where it is larger, at a small cost to visual quality. Defaults to `false`.
-| `gpuAcceleratedSort` | Tells the viewer to use a partially GPU-accelerated approach to sorting splats. Currently this means pre-computing splat distances is done on the GPU. Defaults to `true`.
-| `splatAlphaRemovalThreshold` | Tells `loadFile()` to ignore any splats with an alpha less than the specified value. Defaults to `1`.
+| `cameraUp` | The natural 'up' vector for viewing the scene (only has an effect when used with orbit controls and when the viewer uses its own camera). Serves as the axis around which the camera will orbit, and is used to determine the scene's orientation relative to the camera.
+| `initialCameraPosition` | The camera's initial position (only used when the viewer uses its own camera).
+| `initialCameraLookAt` | The initial focal point of the camera and center of the camera's orbit (only used when the viewer uses its own camera).
 | `halfPrecisionCovariancesOnGPU` |  Tells the viewer to use 16-bit floating point values for each element of a splat's 3D covariance matrix, instead of 32-bit. Defaults to `true`.
-
 <br>
 
-
-As an alternative to using `cameraUp` to adjust to the scene's natural orientation, you can pass an orientation (and/or position) to the `loadFile()` method to transform the entire scene:
-```javascript
-const viewer = new GaussianSplats3D.Viewer({
-    'initialCameraPosition': [-1, -4, 6],
-    'initialCameraLookAt': [0, 4, 0]
-});
-const orientation = new THREE.Quaternion();
-orientation.setFromUnitVectors(new THREE.Vector3(0, -1, -0.6).normalize(), new THREE.Vector3(0, 1, 0));
-viewer.loadFile('<path to .ply or .splat file>', {
-    'splatAlphaRemovalThreshold': 5, // out of 255
-    'halfPrecisionCovariancesOnGPU': true,
-    'position': [0, 0, 0],
-    'orientation': orientation.toArray(),
-})
-.then(() => {
-    viewer.start();
-});
-```
-
-The `loadFile()` method will accept the original `.ply` files as well as my custom `.splat` files.
+Parameters for `loadFile()`
 <br>
-<br>
-### Creating SPLAT files
-To convert a `.ply` file into the stripped-down `.splat` format (currently only compatible with this viewer), there are several options. The easiest method is to use the UI in the main demo page at [http://127.0.0.1:8080/index.html](http://127.0.0.1:8080/index.html). If you want to run the conversion programatically, run the following in a browser:
 
-```javascript
-const compressionLevel = 1;
-const splatAlphaRemovalThreshold = 5; // out of 255
-const plyLoader = new GaussianSplats3D.PlyLoader();
-plyLoader.loadFromURL('<URL for .ply file>', compressionLevel, splatAlphaRemovalThreshold)
-.then((splatBuffer) => {
-    new GaussianSplats3D.SplatLoader(splatBuffer).downloadFile('converted_file.splat');
-});
-```
-Both of the above methods will prompt your browser to automatically start downloading the converted `.splat` file.
+| Parameter | Purpose
+| --- | ---
+| `splatAlphaRemovalThreshold` | Tells `loadFile()` to ignore any splats with an alpha less than the specified value (valid range: 0 - 255). Defaults to `1`.
+| `showLoadingSpinner` | Displays a loading spinner while the scene is loading.  Defaults to `true`.
+| `position` | Position of the scene, acts as an offset from its default position. Defaults to `[0, 0, 0]`.
+| `rotation` | Rotation of the scene represented as a quaternion, defaults to `[0, 0, 0, 1]` (identity quaternion).
+| `scale` | Scene's scale, defaults to `[1, 1, 1]`.
 
-The third option is to use the included nodejs script:
+<br>
 
-```
-node util/create-splat.js [path to .PLY] [output file] [compression level = 0] [alpha removal threshold = 1]
-```
+The `loadFile()` method will accept the original `.ply` files as well as my custom `.splat` files.
 
-Currently supported values for `compressionLevel` are `0` or `1`. `0` means no compression, `1` means compression of scale, rotation, and position values from 32-bit to 16-bit.
-<br>
 <br>
+
 ### Integrating THREE.js scenes
-You can integrate your own Three.js scene into the viewer if you want rendering to be handled for you. Just pass a Three.js scene object as the 'scene' parameter to the constructor:
+You can integrate your own Three.js scene into the viewer if you want rendering to be handled for you. Just pass a Three.js scene object as the `scene` parameter to the constructor:
 ```javascript
 const scene = new THREE.Scene();
-
 const boxColor = 0xBBBBBB;
 const boxGeometry = new THREE.BoxGeometry(2, 2, 2);
 const boxMesh = new THREE.Mesh(boxGeometry, new THREE.MeshBasicMaterial({'color': boxColor}));
-scene.add(boxMesh);
 boxMesh.position.set(3, 2, 2);
+scene.add(boxMesh);
 
 const viewer = new GaussianSplats3D.Viewer({
     'scene': scene,
-    'cameraUp': [0, -1, -0.6],
-    'initialCameraPosition': [-1, -4, 6],
-    'initialCameraLookAt': [0, 4, -0]
 });
 viewer.loadFile('<path to .ply or .splat file>')
 .then(() => {
     viewer.start();
 });
 ```
+
 Currently this will only work for objects that write to the depth buffer (e.g. standard opaque objects). Supporting transparent objects will be more challenging :)
 <br>
+
+A "drop-in" mode for the viewer is also supported. The `RenderableViewer` class encapsulates `Viewer` and can be added to a Three.js scene like any other renderable:
+```javascript
+const scene = new THREE.Scene();
+const renderableViewer = new GaussianSplats3D.RenderableViewer({
+    'gpuAcceleratedSort': true
+});
+renderableViewer.addScenesFromFiles([
+                                        {
+                                            'path': '<path to .ply or .splat file>',
+                                            'splatAlphaRemovalThreshold': 5,
+                                        },
+                                        {
+                                            'path': '<path to .ply or .splat file>',
+                                            'rotation': [0, -0.857, -0.514495, 6.123233995736766e-17],
+                                            'scale': [1.5, 1.5, 1.5],
+                                            'position': [0, -2, -1.2],
+                                            'splatAlphaRemovalThreshold': 5,
+                                        }
+                                    ],
+                                    true);
+scene.add(renderableViewer);
+
+```
 <br>
-### Custom options
-The viewer allows for various levels of customization via constructor parameters. You can control when its `update()` and `render()` methods are called by passing `false` for the `selfDrivenMode` parameter and then calling those methods whenever/wherever you decide is appropriate. You can tell the viewer to not use its built-in camera controls by passing `false` for the `useBuiltInControls` parameter. You can also use your own Three.js renderer and camera by passing those values to the viewer's constructor. The sample below shows all of these options:
+
+### Advanced options
+The viewer allows for various levels of customization via constructor parameters. You can control when its `update()` and `render()` methods are called by passing `false` for the `selfDrivenMode` parameter and then calling those methods whenever/wherever you decide is appropriate. You can also use your own camera controls, as well as an your own instance of a Three.js `Renderer` or `Camera` The sample below shows all of these options:
 
 ```javascript
 const renderWidth = 800;
@@ -209,14 +209,16 @@ rootElement.appendChild(renderer.domElement);
 
 const camera = new THREE.PerspectiveCamera(65, renderWidth / renderHeight, 0.1, 500);
 camera.position.copy(new THREE.Vector3().fromArray([-1, -4, 6]));
-camera.lookAt(new THREE.Vector3().fromArray([0, 4, -0]));
 camera.up = new THREE.Vector3().fromArray([0, -1, -0.6]).normalize();
+camera.lookAt(new THREE.Vector3().fromArray([0, 4, -0]));
 
 const viewer = new GaussianSplats3D.Viewer({
     'selfDrivenMode': false,
     'renderer': renderer,
     'camera': camera,
-    'useBuiltInControls': false
+    'useBuiltInControls': false,
+    'ignoreDevicePixelRatio': false,
+    'gpuAcceleratedSort': true
 });
 viewer.loadFile('<path to .ply or .splat file>')
 .then(() => {
@@ -231,6 +233,40 @@ function update() {
     viewer.render();
 }
 ```
+Advanced `Viewer` parameters
+<br>
+| Parameter | Purpose
+| --- | ---
+| `selfDrivenMode` | If `false`, tells the viewer that you will manually call its `update()` and `render()` methods. Defaults to `true`.
+| `useBuiltInControls` | Tells the viewer to use its own camera controls. Defaults to `true`.
+| `renderer` | Pass an instance of a Three.js `Renderer` to the viewer, otherwise it will create its own. Defaults to `undefined`.
+| `camera` | Pass an instance of a Three.js `Camera` to the viewer, otherwise it will create its own. Defaults to `undefined`.
+| `ignoreDevicePixelRatio` | Tells the viewer to pretend the device pixel ratio is 1, which can boost performance on devices where it is larger, at a small cost to visual quality. Defaults to `false`.
+| `gpuAcceleratedSort` | Tells the viewer to use a partially GPU-accelerated approach to sorting splats. Currently this means pre-computing splat distances is done on the GPU. Defaults to `true`.
+<br>
+
+### Creating SPLAT files
+To convert a `.ply` file into the stripped-down `.splat` format (currently only compatible with this viewer), there are several options. The easiest method is to use the UI in the main demo page at [http://127.0.0.1:8080/index.html](http://127.0.0.1:8080/index.html). If you want to run the conversion programatically, run the following in a browser:
+
+```javascript
+const compressionLevel = 1;
+const splatAlphaRemovalThreshold = 5; // out of 255
+const plyLoader = new GaussianSplats3D.PlyLoader();
+plyLoader.loadFromURL('<URL for .ply file>', compressionLevel, splatAlphaRemovalThreshold)
+.then((splatBuffer) => {
+    new GaussianSplats3D.SplatLoader(splatBuffer).downloadFile('converted_file.splat');
+});
+```
+Both of the above methods will prompt your browser to automatically start downloading the converted `.splat` file.
+
+The third option is to use the included nodejs script:
+
+```
+node util/create-splat.js [path to .PLY] [output file] [compression level = 0] [alpha removal threshold = 1]
+```
+
+Currently supported values for `compressionLevel` are `0` or `1`. `0` means no compression, `1` means compression of scale, rotation, and position values from 32-bit to 16-bit.
+
 <br>
 
 ### CORS issues and SharedArrayBuffer

diff --git a/demo/dropin.html b/demo/dropin.html
@@ -0,0 +1,120 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <meta http-equiv="x-ua-compatible" content="ie=edge">
+  <title>3D Gaussian Splats - Drop-in example</title>
+  <script type="text/javascript" src="js/util.js"></script>
+  <script type="importmap">
+    {
+        "imports": {
+            "three": "./lib/three.module.js",
+            "gaussian-splats-3d": "./lib/gaussian-splats-3d.module.js"
+        }
+    }
+  </script>
+  <style>
+
+    body {
+      background-color: #000000;
+      height: 100vh;
+      margin: 0px;
+    }
+
+  </style>
+
+</head>
+
+<body>
+  <script type="module">
+    import * as GaussianSplats3D from 'gaussian-splats-3d';
+    import * as THREE from 'three';
+
+    function setupRenderer() {
+      const renderWidth = 800;
+      const renderHeight = 600;
+
+      const rootElement = document.createElement('div');
+      rootElement.style.width = renderWidth + 'px';
+      rootElement.style.height = renderHeight + 'px';
+      rootElement.style.position = 'relative';
+      rootElement.style.left = '50%';
+      rootElement.style.top = '50%';
+      rootElement.style.transform = 'translate(-50%, -50%)';
+      document.body.appendChild(rootElement);
+
+      const renderer = new THREE.WebGLRenderer({
+          antialias: false
+      });
+      renderer.setSize(renderWidth, renderHeight);
+      rootElement.appendChild(renderer.domElement);
+
+      return {
+        'renderer': renderer,
+        'renderWidth': renderWidth,
+        'renderHeight': renderHeight
+      }
+    }
+
+    function setupCamera(renderWidth, renderHeight) {
+      const camera = new THREE.PerspectiveCamera(65, renderWidth / renderHeight, 0.1, 500);
+      camera.position.copy(new THREE.Vector3().fromArray([-1, -4, 6]));
+      camera.lookAt(new THREE.Vector3().fromArray([0, 4, -0]));
+      camera.up = new THREE.Vector3().fromArray([0, -1, -0.6]).normalize();
+      return camera;
+    }
+
+    function setupScene() {
+      const scene = new THREE.Scene();
+      const boxColor = 0xBBBBBB;
+      const boxGeometry = new THREE.BoxGeometry(2, 2, 2);
+      const boxMesh = new THREE.Mesh(boxGeometry, new THREE.MeshBasicMaterial({'color': boxColor}));
+      scene.add(boxMesh);
+      boxMesh.position.set(3, 2, 2);
+      return scene;
+    }
+
+    function setupControls(camera, renderer) {
+      const controls = new GaussianSplats3D.OrbitControls(camera, renderer.domElement);
+      controls.rotateSpeed = 0.5;
+      controls.maxPolarAngle = Math.PI * .75;
+      controls.minPolarAngle = 0.1;
+      controls.enableDamping = true;
+      controls.dampingFactor = 0.05;
+      return controls;
+    }
+
+    const {renderer, renderWidth, renderHeight} = setupRenderer();
+    const camera = setupCamera(renderWidth, renderHeight);
+    const scene = setupScene();
+    const controls = setupControls(camera, renderer);
+
+    const renderableViewer = new GaussianSplats3D.RenderableViewer();
+    renderableViewer.addScenesFromFiles([
+                                          {
+                                            'path': 'assets/data/garden/garden.splat',
+                                            'splatAlphaRemovalThreshold': 20,
+                                          },
+                                          {
+                                            'path': 'assets/data/bonsai/bonsai_trimmed.splat',
+                                            'rotation': [-0.14724434, -0.0761755, 0.1410657, 0.976020],
+                                            'scale': [1.5, 1.5, 1.5],
+                                            'position': [-3, -2, -3.2],
+                                            'splatAlphaRemovalThreshold': 20,
+                                          }
+                                        ],
+                                        true);
+    scene.add(renderableViewer);
+
+    requestAnimationFrame(update);
+    function update() {
+      requestAnimationFrame(update);
+      controls.update();
+      renderer.render(scene, camera);
+    }
+  </script>
+</body>
+
+</html>
diff --git a/demo/garden.html b/demo/garden.html
@@ -34,13 +34,12 @@
     const viewer = new GaussianSplats3D.Viewer({
       'cameraUp': [0, -1, -0.54],
       'initialCameraPosition': [-3.15634, -0.16946, -0.51552],
-      'initialCameraLookAt': [1.52976, 2.27776, 1.65898]
+      'initialCameraLookAt': [1.52976, 2.27776, 1.65898],
+      'halfPrecisionCovariancesOnGPU': true
     });
     let path = 'assets/data/garden/garden';
     path += isMobile() ? '.splat' : '_high.splat';
-    viewer.loadFile(path, {
-        'halfPrecisionCovariancesOnGPU': true
-    })
+    viewer.loadFile(path)
     .then(() => {
         viewer.start();
     });

diff --git a/demo/index.html b/demo/index.html
@@ -444,10 +444,10 @@
       const viewerOptions = {
         'cameraUp': cameraUpArray,
         'initialCameraPosition': cameraPositionArray,
-        'initialCameraLookAt': cameraLookAtArray
-      };
-      const sceneOptions = {
+        'initialCameraLookAt': cameraLookAtArray,
         'halfPrecisionCovariancesOnGPU': true,
+      };
+      const splatBufferOptions = {
         'splatAlphaRemovalThreshold': alphaRemovalThreshold
       };
 
@@ -462,7 +462,7 @@
       document.body.style.backgroundColor = "#000000";
       history.pushState("ViewSplat", null);
       const viewer = new GaussianSplats3D.Viewer(viewerOptions);
-      viewer.loadSplatBuffer(splatBuffer, sceneOptions)
+      viewer.loadSplatBuffersIntoMesh([splatBuffer], [splatBufferOptions])
       .then(() => {
           viewer.start();
       });