Compiling nim to javascript

[HugoGranstrom]

This is still bit of a mess but it's a somewhat working mess at least. Here's a fully working example:

import std/strutils
import nimib

nbInit()

nbText: "Counter POC"

template counterButton(id: string) =
  let labelId = "label-" & id
  let buttonId = "button-" & id
  nbRawOutput: """
<p id="$1">0</p>
<button id="$2">Click me</button>
""" % [labelId, buttonId]
  nbJsCode(labelId, buttonId):
    import std/dom
    echo "Hello world!"
    var label = getElementById(labelId.cstring)
    var button = getElementById(buttonId.cstring)
    var counter: int = 0
    button.addEventListener("click",
      proc (ev: Event) =
        counter += 1
        label.innerHtml = ($counter).cstring
    )


counterButton("1")
counterButton("2")

nbSave

Feedback is welcome 😄 Especially on where to put all of this code, it feels like it deserves it's own file to not clutter nimib.nim

TODO

• Test client-side Karax to not have to write HTML in strings (currently blocked by Typed macro removes std / in imports nim-lang/Nim#19904 I think) Can be worked around by using nbNewCode API with string "include karax / prelude" and adding body with script.addCode. Manually importing all karax modules does work, it's only the include which isn't working.
• Clean up in the code
• Test it
• Test it
• Test it
• Test it

Pietroppeter's suggestions

• Move as much as possible to nimib/jsutils
• Rename to nbCodeToJs, nbCodeToJsInit, addCodeToJs, addToDocAsJs.
• Use NbBlock instead of NbCodeScript
• Add rendering code for this block
• Add nbCodeToJsShowSource which should change a switch to show the Nim code.
• Use a random string for kxi_id instead of global counter. (define a nimib-specific rng in nbInit)
• Don't use randomCounter anymore, just check if key in cachetable and reuse it.
• Write counters.nim showcasing reusable widgets
• Write caesar.nim showcasing input handling

[HugoGranstrom]

I've been able to get a Karax template to work by doing the include karax / prelude in a string first and adding the untyped code afterwards:

import std/random
template karaxButton() =
  echo "CReating a new button!"
  block:
    let root {.inject.} = "root" & $rand(int.high)
    nbRawOutput: """<div id="$1"></div>""" % [root]
    let script = nbNewCode: hlNim"""
include karax / prelude   
"""
    script.addCode(root):
      var counter = 0
      proc createDom(): VNode =
        result = buildHtml(tdiv):
          p:
            text "Hello world"
          label:
            text "Counter: " & $counter
          button:
            text "Click me (karax)"
            proc onClick() =
              counter += 1
      setRenderer(createDom, root=root.cstring)
    script.addToDocAsJs()

The problem I have though which I didn't have in the raw-dom example at the top, is that the counter variables are shared between multiple instances of the button. "Shared" in the sense that the same variable name is generated for both of them. So clicking one button increases the count of all other buttons as well. In the previous example, different variables were generated.

So it's partially working if you don't have any global variables that you update.

[HugoGranstrom]

Upon further experimenting by adding a third button to the first example is also exhibiting this same behavior. The reason it didn't at first was that for some reason the first time we call the macro the code is treated as untyped but the second time as typed. And this resulted in two slightly different codes which had different names in the js. But adding a third button made the second and third button share the same names in the js. So we will have to do some manual name mangling ourselves to make this work. The complicating part is that we only have a string in the end and not the AST anymore.

[pietroppeter]

Well, one thing is that we could make explicit the limitation that you could have a single js or Karax section in the page and you should have all the logic there.

Another option that comes to mind (but I know very little about this and not so sure if it would work) is to see if an iframe would work in sandboxing different parts of JavaScript (found this ref, have not had time to read it in detail: https://web.dev/sandboxed-iframes/)

[HugoGranstrom]

Well, one thing is that we could make explicit the limitation that you could have a single js or Karan section in the page and you should have all the logic there.

That is way too limiting in my eyes, plus for Karax to work we have to compile the different blocks in separate files for them to get their own Karax instances (it was just recently implemented support for this, see karaxnim/karax#221). So putting them each into a block doesn't work (I've tried and only one of them rendered).

Another option that comes to mind (but I know very little about this and not so sure if it would work) is to see if an iframe would work in sandboxing different parts of JavaScript (found this ref, have not had time to read it in detail: https://web.dev/sandboxed-iframes/)

Perhaps, but I have an idea for how to do the mangling. It involves a CT table to keep track of the variable names for all variables in a specific script. And then when a new script is created the table is reset. This way we would keep the flexibility but isolate each script.

[HugoGranstrom]

My solution actually seems to work 🤯 Now Karax should be fully working (if used with string imports and includes).

[HugoGranstrom]

@pietroppeter Now we can start discussion namings and where to put this code. Do you have any preferences?

[pietroppeter]

Wow! I will need to digest that a moment in order to get reasonable advice on naming and stuff. You know that merging this makes you an official maintainer now? ;) I will need you to get whatever rights you need for the repo (or we could move this and other repos to a nimib organization).

Anyway, let me get to basics. The feature we want to provide with this is a way to embed Nim derived js scripts in a page AND this would work also for Karax's framework, correct?

Can you add an example doc (or more than one if it works better) where we can see both in action (standard Nim to js and an example with Karax)? You should in this case merge current master (or rebase on top of it) so that you have the new structure where the example sources for docs are in docsrc.

[HugoGranstrom]

Wow! I will need to digest that a moment in order to get reasonable advice on naming and stuff.

My only idea so far is that the templates are still kept in nimib.nim but nimToJsString and all macros associated with it are moved to a separate file.

You know that merging this makes you an official maintainer now? ;) I will need you to get whatever rights you need for the repo (or we could move this and other repos to a nimib organization).

What?! :O What an honor! 😄 Or does this just mean I can't just dump my code here and expect you to maintain it for me? 🤣
If you choose to create a nimib org, I would gladly transfer both nimiBoost and nimiSlide to it. But for you it also has downsides, making it into an org would make it less obvious that you are the master-mind behind nimib for example.

Anyway, let me get to basics. The feature we want to provide with this is a way to embed Nim derived js scripts in a page AND this would work also for Karax's framework, correct?

Exactly! Allowing us to finally add interactive elements to the output without inlining javascript (nor HTML). If you want to add a button that does something, this is exactly what you should use.

Can you add an example doc (or more than one if it works better) where we can see both in action (standard Nim to js and an example with Karax)? You should in this case merge current master (or rebase on top of it) so that you have the new structure where the example sources for docs are in docsrc.

Will do! Does it suffice if I add the counter buttons I have shown here or do you want to add something more spectacular as well? 😄 I don't have anything specific in mind but if you do, I could try implementing it.

[pietroppeter]

My only idea so far is that the templates are still kept in nimib.nim but nimToJsString and all macros associated with it are moved to a separate file.

agree on that. the non essential (non public api) should be moved to a nimib/jsutils.nim from what we should import (and export) onlt what is necessary to have it working.

on naming:

• nbJsCode, I propose to change it to nbCodeToJs. nbCode is what we use to integrate Nim code and what that block does is translating it to js code.
• nbNewCode could become nbCodeToJsInit, it initializes a block of code to js (and the init is not because it creates a object instead of a ref object but because it initializes that operation)
• addCode would become addCodeToJs
• addToDocAsJs stays as it is

I would also make another change and use the standard NbBlock instead of a new type NbCodeScript. In code field there should be the original nim code and in output the code to JS. You would need to add the render function to html backend (the command would be nbCodeToJs. I might not seem some issues that might prevent this from happening and anyway would like to hear your opinion on this. Incidentally this should make it easier to show the orginal nim code in notebook (something that you want to do in the interactivity document for example).

I still have to review more but ideally I would try to avoid as possible adding other global objects, for example kxi_id. In principle we could have a field in nb.context that is called kxinames and it could be an array of strings. Once a nbCodeToJs block is created a new kxiname is created (and added to nb.blk.context) by making it distinct from previous ones (check if $len(kxinames) is already there if not increase the counter and try again). But this is an idea to make kxiname in principle customizable (would there be any advantage?) and we would keep track of all previous names. Not sure if this overcomplicates stuff, just an idea, there might be a better way, the gist is: let's make it something not a global variable but part of nb object and of current block object. We could add a field but maybe it is just more convenient to use the context.

I am missing at the moment what is the role of randomCounter but it could be treated in the same way (and maybe this is more essential and we want instead to add field to the NbDoc and NbBlock object not sure).

What?! :O What an honor! 😄 Or does this just mean I can't just dump my code here and expect you to maintain it for me? 🤣 If you choose to create a nimib org, I would gladly transfer both nimiBoost and nimiSlide to it. But for you it also has downsides, making it into an org would make it less obvious that you are the master-mind behind nimib for example.

well, yeah. Once we merge this I would definitely need your help in maintaining it (as it is indeed the case for nimibCodeAsSource) so let's make it official :)

regarding not being recognized for originating nimib I do not care very much and I see the project as long term sustainable only if others join the effort.

Can you add an example doc (or more than one if it works better) where we can see both in action (standard Nim to js and an example with Karax)? You should in this case merge current master (or rebase on top of it) so that you have the new structure where the example sources for docs are in docsrc.

Will do! Does it suffice if I add the counter buttons I have shown here or do you want to add something more spectacular as well? 😄 I don't have anything specific in mind but if you do, I could try implementing it.

I have seen you already added an interactivity document that looks fine, maybe too many details in there but we can streamline it later. The counter examples are fine, it could be a counters.nim and it shows how to create resuable widgets (which is where the additional complexity of ids is needed).
Another though I had would be a caesar.nim where you have two textboxes and two buttons (layout could be a very simple linear layout) and the buttons implement the encode and decode of Caesar_cipher.

[HugoGranstrom]

agree on that. the non essential (non public api) should be moved to a nimib/jsutils.nim from what we should import (and export) onlt what is necessary to have it working.
on naming:

Those names and structure sounds good to me 👍

I would also make another change and use the standard NbBlock instead of a new type NbCodeScript. In code field there should be the original nim code and in output the code to JS. You would need to add the render function to html backend (the command would be nbCodeToJs. I might not seem some issues that might prevent this from happening and anyway would like to hear your opinion on this.

Yes, that sounds like a good idea. So nbCodeToJsInit returns a NbBlock but it isn't until we do script.addToDocAsJs that it is added to nb.blocks? And then the logic currently in addToDocAsJs is moved to the HTML backend which populates the blk.code field. Do I understand it correctly?

Incidentally this should make it easier to show the orginal nim code in notebook (something that you want to do in the interactivity document for example).

I'm not sure what you mean here. Or I think you mean that we want this to work:

nbCode:
  nbCodeToJs:
    echo "Hello world!"

but I'm not sure how this would help. For example, if we do:

nbCode:
  nbText: "Hello world"

The output I get is neither a code block nor a text block but the echos we are doing in newNbBlock:

[nimib] 5 nbText: -> Hello world

[nimib] 5 nbText: -> Hello world

So nestability of blocks doesn't work regardless.

I still have to review more but ideally I would try to avoid as possible adding other global objects, for example kxi_id. In principle we could have a field in nb.context that is called kxinames and it could be an array of strings. Once a nbCodeToJs block is created a new kxiname is created (and added to nb.blk.context) by making it distinct from previous ones (check if $len(kxinames) is already there if not increase the counter and try again). But this is an idea to make kxiname in principle customizable (would there be any advantage?) and we would keep track of all previous names. Not sure if this overcomplicates stuff, just an idea, there might be a better way, the gist is: let's make it something not a global variable but part of nb object and of current block object. We could add a field but maybe it is just more convenient to use the context.

I agree that using a global variable isn't good in this case. Building upon the NbBlock route, we only need a unique kxi_id for each block so I could use the context and just increase it every time the render proc is called. I don't really see a point in making it customizable though. It will only be used in a single page so there is no risk of collisions unless you combine multiple pages of course. A solution to that would be to generate a random string for kxi_id instead. And if we make it long enough we shouldn't run any risks of collisions and don't have to store previous ones. Then we don't have to store anything for this. What do you think about this?

I am missing at the moment what is the role of randomCounter but it could be treated in the same way (and maybe this is more essential and we want instead to add field to the NbDoc and NbBlock object not sure).

randomCounter is a CT variable so we can't use our RT nb variable for it. The reason I added it was that multiple bodies would cause collisions in the macrotable. For example nbCodeToJs in a template would have the same body each time we call the template. But I now realize that we only have to add it to the table once and then check if it is in the table or not. So we should be able to get rid of randomCounter entirely.

well, yeah. Once we merge this I would definitely need your help in maintaining it (as it is indeed the case for nimibCodeAsSource) so let's make it official :)
regarding not being recognized for originating nimib I do not care very much and I see the project as long term sustainable only if others join the effort.

💯 Sounds like a good plan and wise words👌 I'll happily help out where I can.

I have seen you already added an interactivity document that looks fine, maybe too many details in there but we can streamline it later. The counter examples are fine, it could be a counters.nim and it shows how to create resuable widgets (which is where the additional complexity of ids is needed).
Another though I had would be a caesar.nim where you have two textboxes and two buttons (layout could be a very simple linear layout) and the buttons implement the encode and decode of Caesar_cipher.

Hehe yeah I wasn't sure how much to say about the details, especially since nbCode didn't play well with nbRawOutput in addToDocAsJs. Maybe adding a separate section at the with the detailed parts so those who are interested could read it and the others aren't bothered.
counters.nim sounds like a separation from the interactive part. And caesar.nim does sound like a fun example to add as well :o 😄 Will be interesting to see which of Karax and pure dom that is easier for that one.

[pietroppeter]

I would also make another change and use the standard NbBlock instead of a new type NbCodeScript. In code field there should be the original nim code and in output the code to JS. You would need to add the render function to html backend (the command would be nbCodeToJs. I might not seem some issues that might prevent this from happening and anyway would like to hear your opinion on this.

Yes, that sounds like a good idea. So nbCodeToJsInit returns a NbBlock but it isn't until we do script.addToDocAsJs that it is added to nb.blocks? And then the logic currently in addToDocAsJs is moved to the HTML backend which populates the blk.code field. Do I understand it correctly?

yes, correct

Incidentally this should make it easier to show the orginal nim code in notebook (something that you want to do in the interactivity document for example).

I'm not sure what you mean here. Or I think you mean that we want this to work:

[...]

So nestability of blocks doesn't work regardless.

yes, nestability of blocks is a separate issue.

what I meant is that we could provide a nbCodeToJsShowSource functionality that if called after a nbCodeToJs (or after script.addToDocAsJs) would make the original nim code to appear in the document. The api would look like this:

nbCodeToJs:
  discard
nbCodeToJsShowSource

and the implementation could be something like this part (to be added in useHtmlBackend, note the usage of existing partial nbCodeSource):

  doc.partials["nbCodeToJs"] = """
{{>nbJsScriptNimSource}}
{{>nbJsScript}}"""
  doc.partials["nbJsScriptNimSource"] = "{{#js_show_nim_source}}{{>nbCodeSource}}{{/js_show_nim_source}}"
  doc.partials["nbJsScript"] = "<script>{{&output}}</script>"

(note: you will need to highlight nim code)

and this part:

template nbCodeToJsShowSource =
  nb.blk.context["js_show_nim_source"] = true

I still have to review more but ideally I would try to avoid as possible adding other global objects, for example kxi_id. In principle we could have a field in nb.context that is called kxinames and it could be an array of strings. Once a nbCodeToJs block is created a new kxiname is created (and added to nb.blk.context) by making it distinct from previous ones (check if $len(kxinames) is already there if not increase the counter and try again). But this is an idea to make kxiname in principle customizable (would there be any advantage?) and we would keep track of all previous names. Not sure if this overcomplicates stuff, just an idea, there might be a better way, the gist is: let's make it something not a global variable but part of nb object and of current block object. We could add a field but maybe it is just more convenient to use the context.

I agree that using a global variable isn't good in this case. Building upon the NbBlock route, we only need a unique kxi_id for each block so I could use the context and just increase it every time the render proc is called. I don't really see a point in making it customizable though. It will only be used in a single page so there is no risk of collisions unless you combine multiple pages of course. A solution to that would be to generate a random string for kxi_id instead. And if we make it long enough we shouldn't run any risks of collisions and don't have to store previous ones. Then we don't have to store anything for this. What do you think about this?

ah yes, probably better! in principle as an id you could even take len(nb.blocks) and in most cases this would work.

I am missing at the moment what is the role of randomCounter but it could be treated in the same way (and maybe this is more essential and we want instead to add field to the NbDoc and NbBlock object not sure).

randomCounter is a CT variable so we can't use our RT nb variable for it. The reason I added it was that multiple bodies would cause collisions in the macrotable. For example nbCodeToJs in a template would have the same body each time we call the template. But I now realize that we only have to add it to the table once and then check if it is in the table or not. So we should be able to get rid of randomCounter entirely.

even better :)

[HugoGranstrom]

what I meant is that we could provide a nbCodeToJsShowSource functionality that if called after a nbCodeToJs (or after script.addToDocAsJs) would make the original nim code to appear in the document. The api would look like this:

Ah ok, then I understand what you mean. Thanks for the tips :D

ah yes, probably better! in principle as an id you could even take len(nb.blocks) and in most cases this would work.

If two scripts are created right after each other and then the addToDocAsJs for both are run after that, I think they will get the same id if we do it this way. As the number of blocks doesn't change until we actually add the scripts to nb.blocks. This is actually a quite likely scenario if two different templates are building up their scripts piece by piece using addCodeToJs.

I'll start implementing all this tomorrow

[pietroppeter]

That theme could work but I don't think it will beat writing karax in a dedicated file. In nbKaraxCode we do get autocomplete but we don't get error checks so you have to run the code to get the errors in the console.

You are right about that.

Getting back to this PR. We agree that theme we do it later?
If I think of stuff left to do, probably the only thing is to change the location (and name) of code.nim, right? I think it still make sense to put it next to the processed file (in case it accesses statically some assets) and not remove it if js compilation gives error.

For the name we could use:

• for nbKarax: nbkarax3_myapp.nim (where myapp.nim would be the original name of file and 3 one of the id that could be reused).
• for nbCodeToJs (and related): nbcodetojs7_myapp.nim (same as above)

[HugoGranstrom]

wheeew, I was sweating the imposter syndrome fearing I got it wrong! 😅

No you just are thinking in less convoluted ways than me often. I'm making things a bit took complicated to what they have to be sometimes :D

Getting back to this PR. We agree that theme we do it later?

Yes, the theme can come later. But tbh I actually prefer the autostyling of water.css over manually having to assign classes using bulma (the CSS framework included by karun).

If I think of stuff left to do, probably the only thing is to change the location (and name) of code.nim, right? I think it still make sense to put it next to the processed file (in case it accesses statically some assets) and not remove it if js compilation gives error.

Yes, that sounds about right. And if we want to add moduleAvailable(karax) for now. Naming wise the renderer doesn't know the difference between nbKaraxCode and nbCodeToJs, they are both just nbCodeToJs to it. So the naming convention would simply be nbcodetojs7_myapp.nim.

Btw, I got the use-case for which I wanted to create all of this working now. coolWidget was actually typewriter for nimiSlides to get a typewriter writing your text char-by-char. 🥳

[HugoGranstrom]

TODO (I've found a few bugs):

• Guard using moduleAvailable(karax)
• Place generated file next to orignal
• Gensym symbols that aren't gensym'd in a template (proc, iterator, converter), otherwise they will get the same names in the final code.

[HugoGranstrom]

It feels kinda pointless to have tests that aren't run by the CI because it doesn't have karax installed. Would you mind if I added so nimble docsdeps was run for the tests as well?

[pietroppeter]

Yes sure

[HugoGranstrom]

If it passes the build now and the resulting pages seem to function as they should, I'd say it's finished now 🎊

[pietroppeter]

minor points

[pietroppeter]

checked everything and also gave a quick read on the docs preview to see if there were regression. everything seems fine.
Again great work on this, it has been fun!
You should have been invited to the repo, do you want to test your new powers and do the honors by merging this? :)

[HugoGranstrom]

checked everything and also gave a quick read on the docs preview to see if there were regression. everything seems fine.
Again great work on this, it has been fun!
You should have been invited to the repo, do you want to test your new powers and do the honors by merging this? :)

Did the same and everything seems fine. Yes it has been fun and we have more fun to await now that we have the power of javascript at our fingertips.

What an honor! ❤️ Which method of merging do you prefer for PRs? Saw that it was 54 commits so perhaps squashing is the best option?

[pietroppeter]

yes, squash and merge (for any PR in general)

[HugoGranstrom]

Said and done! That felt nice 🥳🎉