HTML content in the background?

[jimgoo]

I've got a use case where I'm trying to overlay TLDraw on top of an HTML background page. The HTML comes from a Jupyter notebook (for example: https://nbviewer.org/github/waltherg/notebooks/blob/master/2013-12-03-Crank_Nicolson.ipynb), so the HTML can contain tables, plots, equations, etc. The idea is to put TLDraw on top of the rendered notebook's HTML that so that scientist type of folks can collaborate on annotating their work in multiplayer mode. The key requirement is that the annotation shapes line up to the same locations in the underlying HTML content across mobile and desktop browsers (so if one person circles a point in a plot, everyone sees the same point circled - see screenshot).

I've got a semi-working version where I do the following:

    <div
      style = {{ 
        position: "relative", 
        width: dimensions.width, 
        height: dimensions.height 
      }}
    >
      <NotebookHTML html={html}/>   
      <div
        style = {{
          position: "absolute",
          top: 0,
          left: 0,
          width: "100%",
          height: "100%",
        }}
      >
        <Tldraw />
      </div>
    </div>

so that the Tldraw canvas div is set to be a fixed size for all users (dimensions.width x dimensions.height) that is also as large as my notebook HTML. In my fork of Tldraw, I've turned off scroll listeners and zooming so that scrolling just moves down the HTML page. The canvas is essentially finite and spans the bounds of the notebook's HTML.

The issue with this approach is that the TLDraw shape coordinates only match up between users if they're using the same type of browser. In the attached screenshot, the Chrome user is on the left drawing in blue, and the Safari user is on the right drawing in red. I think this is just due to subtle differences in how the browsers render the HTML, which throws off the locations of TLdraw things since the coordinates are all with respect to a dimensions.width x dimensions.heightcanvas. The differences increase as we go further down the page and tiny differences compound.

There's no real issue with TLDraw, my question is more how can I go about this in a better way? I've considered moving <NotebookHTML> to inside of the TLDraw canvas, but I don't think that would solve the differences in browser rendering issue.

For mobile users, the differences are even worse since I think the browser HTML rendering differences compound even more. I've also tried adding a huge screenshot of the HTML page to TLDraw as media, but the font looks super tiny and it just feel kind of unwieldily. I have no way of getting pure SVG of the notebook, plus the users have some javascript embedded in the notebook HTML for hovering over points in a plot, etc.

Is there a better way to do what I'm trying to do? Again, I have no issues with TLDraw at all. It's been wonderful to work with and learn from.

[judicaelandria]

hey, quick question, why does it need to be a background?

[jimgoo]

By background I mostly just mean that the notebook's HTML appears behind the TLDraw canvas so that any drawings, shapes, text, sticky notes, etc added by TLDraw appear on top of the notebook's contents. TLDraw is basically serving as an annotation layer that sits on top of the notebook HTML layer. Since the notebooks can have their own interactive Javascript content (plots, movie players, etc), for now I added another setting to TLDraw that sets the z-index of the TLDraw canvas to something really small if the user wants to interact with their original notebook content instead of what's on the TLDraw canvas. It definitely feels very hacky, so I'd love to hear other ideas!

[steveruizok]

I'd use css to take out the background of the tldraw canvas! But this is a common request, so noted. Interesting use case too!

[jimgoo]

Ah ok yeah that makes total sense!

[gillchristian]

Hey @jimgoo, I'm trying to do something similar (have TLDraw on top of text) so your project is a confirmation that it is possible 🚀

for now I added another setting to TLDraw that sets the z-index of the TLDraw canvas to something really small if the user wants to interact with their original notebook content instead of what's on the TLDraw canvas.

Do you mind if I ask how did you implement that (from functionality perspective, not the actual implementation details)? When do you switch the z-index of the TLDraw canvas to allow interacting with the Notebook's content?

I want to have both things being interactive (the text and the TLDraw canvas), since I would like to support doing stuff with the text (like copy) so I'm not fully convinced about making the user explicitly switch between drawing mode and interaction with text. Maybe putting the text inside the TLDraw canvas is the way to go for my case.

[jimgoo]

I added a boolean notebook_in_foreground that's true by default, then inside of Canvas.tsx I have this:

 <div id={id} className="tl-container" ref={rContainer} 
      style={{
        zIndex: notebookInForeground ? -500 : 1
      }}
 >

The user has a button that will toggle notebook_in_foreground, which needs to be true in order to do any kind drawing with TLDraw. Mostly the button is there if they need to copy some text. My text isn't in the canvas, the canvas is just sitting on top (or behind) the notebook's HTML which contains text.

[gillchristian]

Thanks!

I looked at some of he examples in your project, it makes sense to have the switch there since the canvas is the main interaction. I'm not yet clear what the best UX for my use case would be, but your example helped me a lot. Thanks again!

[Rodeoclash]

I do something similar to this with my app here:

https://www.vodon.gg/

Except TLDraw is overlayed on top of HTML video.

If you're interested in my implementation, you can view it here: https://github.com/Rodeoclash/vodon-player/blob/main/player/src/components/ui/Drawing.tsx

This also handles resizing of the screen (and will resize TLDraw to scale with it).