notebooks/demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MeshCat.jl\n",
    "\n",
    "This notebook demonstrates interacting with the MeshCat viewer from Julia. You'll need to have installed MeshCat.jl from <https://github.com/rdeits/MeshCat.jl>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional: \n",
    "# These commands tell the Julia package manager to use the exact\n",
    "# set of dependencies specified in the Project.toml file in this folder. \n",
    "# That should give you a nice, reproducible environment for testing. \n",
    "\n",
    "using Pkg\n",
    "Pkg.activate(@__DIR__)\n",
    "Pkg.develop(Pkg.PackageSpec(path=dirname(@__DIR__)))\n",
    "Pkg.instantiate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Activate the MeshCat package, and import some other useful functions\n",
    "using MeshCat\n",
    "using CoordinateTransformations\n",
    "using Rotations\n",
    "using GeometryBasics: HyperRectangle, Mesh, Point, TriangleFace, Vec\n",
    "using Colors: RGBA, RGB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new visualizer instance\n",
    "vis = Visualizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The `render(vis)` function will try to render a MeshCat\n",
    "# visualizer inline. In Jupyter, it will cause the visualizer\n",
    "# to be displayed in the output of the current Jupyter cell. \n",
    "# In Juno/VSCode, it will cause the visualizer to be displayed in\n",
    "# the plot pane.\n",
    "render(vis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several ways you can open the visualizer if you like:\n",
    "\n",
    "### In-Browser\n",
    "To open the meshcat viewer in your browser, you can simply do:\n",
    "\n",
    "```julia\n",
    "open(vis)\n",
    "```\n",
    "\n",
    "### Standalone (with Electron.jl)\n",
    "\n",
    "Electron.jl allows you to create standalone web apps with Electron. To use\n",
    "Electron with MeshCat, you need to install Electron:\n",
    "\n",
    "```julia\n",
    "Pkg.add(\"Electron\")\n",
    "import Electron\n",
    "```\n",
    "\n",
    "Then you can open MeshCat in an Electron window with:\n",
    "\n",
    "```julia\n",
    "open(vis, Electron.Applicaction())\n",
    "```\n",
    "\n",
    "### Embed the visualizer inside this notebook\n",
    "\n",
    "```julia\n",
    "render(vis)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We also need to check if this notebook is running on Travis CI\n",
    "# since the browser doesn't work properly there. You shouldn't \n",
    "# need this in your code, and you can just do `open(vis)`.\n",
    "\n",
    "if !haskey(ENV, \"CI\")\n",
    "    open(vis)\n",
    "end\n",
    "\n",
    "# To block execution until the visualizer is open, you can do:\n",
    "# wait(vis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, we'll create a simple geometric object\n",
    "box = HyperRectangle(Vec(0.0, 0, 0), Vec(1.0, 1, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setobject! causes the visualizer to render the given geometry. \n",
    "# Initially that geometry will be placed in the zero configuration\n",
    "# (that is, its position and rotation are zero)\n",
    "setobject!(vis, box)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can use settransform!() to tell the viewer to draw the box at a specific\n",
    "# position. Translation() creates a CoordinateTransformations.Transformation\n",
    "# corresponding to the given x; y; z translation.\n",
    "settransform!(vis, Translation(0.0, 1, 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can also rotate the model by sending a different transformation\n",
    "settransform!(vis, LinearMap(AngleAxis(pi / 4, 0, 0, 1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And we can clear the box\n",
    "delete!(vis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Next, let's add the same box, but color it green this time.\n",
    "green_material = MeshPhongMaterial(color=RGBA(0, 1, 0, 0.5))\n",
    "setobject!(vis, box, green_material)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "delete!(vis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# The Viewer Tree\n",
    "\n",
    "Now that we've introduced the basics of loading and transforming geometries, it's time to introduce the tree model used by MeshCat.jl. The idea is pretty simple:\n",
    "\n",
    "* A Visualizer contains a tree of geometries\n",
    "* We add, draw, and delete geometries by using their path, which is the path from the root of the tree to that geometry\n",
    "* We can set the transform for any node in the tree. Setting the transform for a node in the tree affects all of its descendant geometries.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can access a particular path within the visualizer with indexing notation:\n",
    "vis[\"group1\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We load geometries using the same path notation:\n",
    "green_box_vis = setobject!(vis[\"group1\"][\"greenbox\"], box, green_material)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click \"open controls\" in the MeshCat viewer. You should see a `meshcat` folder, which you can click to expand:\n",
    "\n",
    "    Scene\n",
    "      | meshcat\n",
    "          | group1\n",
    "              | greenbox\n",
    "\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And we set transforms in the same way:\n",
    "settransform!(green_box_vis, Translation(0, 0, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The same indexing notation makes it easy to get a handle to a \n",
    "# particular part of the viewer tree:\n",
    "group1 = vis[\"group1\"]\n",
    "settransform!(group1, Translation(0, 0, -1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The viewer tree becomes much more interesting (and useful) when we have multiple geometries. Let's add another geometry in the same group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "blue_material = MeshPhongMaterial(color=RGBA(0, 0, 1, 0.5))\n",
    "setobject!(vis[\"group1\"][\"bluebox\"], box, blue_material)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scene browser will now show:\n",
    "\n",
    "    Scene\n",
    "      | meshcat\n",
    "          | group1\n",
    "              | greenbox\n",
    "              | bluebox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since both the green and blue box are inside `group1`, we can move them together by transforming that whole group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "settransform!(vis[\"group1\"], Translation(1, 0, 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can move just the green box:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "settransform!(vis[\"group1\"][\"greenbox\"], Translation(0, 1, 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final pose of each geometry in the tree is just the composition of all of the transforms in the path from the root of the tree to that geometry. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, we can delete an entire group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "delete!(vis[\"group1\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Of course, we can draw much more interesting geometries than \n",
    "# just simple boxes. Let's load a 3D mesh and visualize it:\n",
    "using MeshIO\n",
    "using FileIO\n",
    "cat_mesh = load(joinpath(dirname(pathof(MeshCat)), \"..\", \"test\", \"data\", \"meshes\", \"cat.obj\"))\n",
    "setobject!(vis, cat_mesh)\n",
    "settransform!(vis, LinearMap(AngleAxis(pi / 2, 1, 0, 0)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "delete!(vis)\n",
    "\n",
    "# Next, let's create a triangulated mesh by finding the 0-level set of some function. \n",
    "\n",
    "# First, we'll define our function:\n",
    "f = x -> sum(sin, 5 * x)\n",
    "\n",
    "# Then we pick a region of interest in which to sample the function.\n",
    "# This region starts at (-1, -1, -1) and extends to (1, 1, 1):\n",
    "lower_bound = Vec(-1.0, -1, -1)\n",
    "upper_bound = Vec(1.0, 1, 1)\n",
    "\n",
    "# Those two pieces of information are all we need to construct a mesh geometry.\n",
    "\n",
    "# Using `Meshing.jl`, we can construct a mesh directly from our function:\n",
    "using Meshing: MarchingTetrahedra, isosurface\n",
    "\n",
    "xr,yr,zr = ntuple(i->LinRange(lower_bound[i], upper_bound[i],50),3)\n",
    "sdf = [f(Vec(x,y,z)) for x in xr, y in yr, z in zr]\n",
    "vts, fcs = isosurface(sdf, MarchingTetrahedra(), xr, yr, zr)\n",
    "mesh = Mesh(Point.(vts), TriangleFace.(fcs))\n",
    "\n",
    "# And now we can load that geometry into the visualizer\n",
    "setobject!(vis, mesh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More Geometry Types\n",
    "\n",
    "`MeshCat.jl` supports a few extra types of geometry that you may want to visualize. One such geometry is a point cloud: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Point Clouds\n",
    "\n",
    "A PointCloud can be constructed from a vector of points. The easiest way to represent a point is just a standard Julia vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "delete!(vis)\n",
    "pointcloud = PointCloud([[x, 0 + 0.01 * randn(), 0.5] for x in range(-1, stop=1, length=1000)])\n",
    "setobject!(vis[:pointcloud], pointcloud)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a much more efficient point cloud, try using a vector of `StaticVector`s from StaticArrays.jl. The `Point3f` is one such static vector:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "using GeometryBasics: Point3f\n",
    "points = rand(Point3f, 1000000)\n",
    "setobject!(vis[:pointcloud], PointCloud(points))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also color each point by passing a vector of `RGB{Float32}` elements to `PointCloud`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "colors = reinterpret(RGB{Float32}, points); # use the xyz value as rgb color\n",
    "setobject!(vis[:pointcloud], PointCloud(points, colors))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Textured Meshes\n",
    "\n",
    "Meshes can be textured by passing a `Texture` parameter to the material constructor. Currently, the only supported texture type is `PngImage`, which simply transmits the data from an image as base64-encoded PNG data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image = PngImage(joinpath(MeshCat.VIEWER_ROOT, \"..\", \"data\", \"HeadTextureMultisense.png\"))\n",
    "texture = Texture(image=image)\n",
    "material = MeshLambertMaterial(map=texture)\n",
    "geometry = load(joinpath(MeshCat.VIEWER_ROOT, \"..\", \"data\", \"head_multisense.obj\"))\n",
    "setobject!(vis[\"robots\", \"valkyrie\", \"head\"], geometry, material)\n",
    "settransform!(vis[\"robots\", \"valkyrie\"], Translation(0.5, -0.5, 0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mesh Files\n",
    "\n",
    "While MeshCat should be able to render any mesh you can load as any kind of `GeometryBasics.AbstractMesh`, there are a wide variety of mesh file formas, and not all of them can be easily loaded in Julia right now. For OBJ, DAE, and STL (ASCII and binary) meshes, there is an additional option: The `MeshFileGeometry` type passes a raw mesh file directly to the MeshCat viewer, letting the existing mesh decoders built into that viewer take care of loading the data for you. For example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = joinpath(@__DIR__, \"..\", \"test\", \"data\", \"meshes\", \"cube.dae\")\n",
    "setobject!(vis[\"meshes\", \"dae_file_geometry\"], MeshFileGeometry(path))\n",
    "settransform!(vis[\"meshes\", \"dae_file_geometry\"], Translation(0.0, 1.25, 0.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in this case only the *geometry* of the mesh is loaded. \n",
    "\n",
    "# Mesh File Objects\n",
    "\n",
    "Certain kinds of meshes, like OBJ and DAE files, can contain meshes with associated textures or even multiple such objects in a single file. To render a mesh file, including all of its associated materials and textures, you can use the `MeshFileObject` type, which passes the entire mesh file contents *and its associated materials* to the MeshCat viewer, again letting the built-in decoders handle all of the data loading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = joinpath(@__DIR__, \"..\", \"test\", \"data\", \"meshes\", \"cube.obj\")\n",
    "setobject!(vis[\"meshes\", \"dae_file_object\"], MeshFileObject(path))\n",
    "settransform!(vis[\"meshes\", \"dae_file_object\"], Translation(0.0, 2.5, 0.0))"
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Julia 1.11.1",
   "language": "julia",
   "name": "julia-1.11"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}