We keep all the functionalities related to NeRF itself in our viewer. Thanks again to Nerfstudio team for building such an extraordinary library.
Click the "Enable SAM" checkbox, which will enable providing point prompt by clicking. The markers whill "sticks" to the object you clicked until you click "Clear SAM pins" button right under the previous one.
point_prompt_output.mov
There are two ways to use language prompts: by using a search box or filling the textbox on the config panel.
You can provide text prompt by filling the text in the "Text Prompt". Text prompt will be updated once you click the "Send Text Prompt" button. One can change the threshold of relevance and the number of prompts to SAM model by filling the corresponding fields.
text_prompt_output.mov
We also provide a search box for acquiring your text prompt. Press "Ctrl + F" ("^ + F" on MacOS) to toggle the search box. Once you finish inputing, press "Enter" will trigger the viewer to rerender with the given text prompt. Press "Ctrl + F" again to make the search box disappear and the original RGB image will be displayed. Futher, for convenience, the threshold and number of prompts in the right control panel is valid when using seach box, so you can adjust a proper parameter to fit your needs.
search_box_output.mov
Beside the aboving, we also providing a heatmap option to visualize pixel-level relevance of the providing text prompt. You can change the "Output Render" to "clipseg_feature" to enable this.
switch_output_output.mov
Since the prompts are provided in the form of 2D image coordinate, the resolution will affect the final segmentation result. To solve this, we provide a checkbox to lock the resolution. With this enabled, the output resolution will not change when dragging and moving the camera, which is helpful to observing the generated segmentation mask.
If you find something is not updated (e.g. enter a word in search box and press "Enter" but nothing happens), dragging the scene will probably solve the problem. The viewer is currently work in progress, and there may exist tons of bugs. Please let us know if you encounter something unexpected, thanks in advance for you help 🥰.