Bundle multi-module Scala.JS projects

[japgolly]

Currently, this plugin applies to single SBT modules in isolation. When you're working on a multi-module project and that has multiple modules exporting Scala.JS, then you need to bundle everything together.

This is the situation I'm currently in and one of the reasons I think I'm going to use webpack without scalajs-bundler. I think someone is going to have to work out what that means for this project and how best to solve it if scalajs-bundler is going to be used for Big Serious (™) work projects. My OSS quota is already full so it won't be me but I'm certainly open to share info, ideas and opinions if it can provide some value so feel free to ask.

[julienrf]

scalajs-bundler uses the output of Scala.js. If you have a multi-module build, then scalajs-bundles uses the .js file that results from the linking of all the modules. What’s the problem with that? How would you do things differently?

[japgolly]

Using webpack that way doesn't give you much value because all it would do is create big monolithic bundles, it would be better to use something like rollup.js which is faster and simpler. You could give each module custom webpack configs and instruct each one to spit out multiple modules but the scope is still just the single module each time. And what about all the assets? It's rare that you'd never share any assets between multiple bundles in the same project.

In a multi-module app, what you'd generally want is to feed the JS of each SBT module to webpack as separate entry points, configure it to have access to your webapp assets and 3rd-party libs, then let it do it's thing with a view of the project as a whole. This allows webpack to identify common code amongst the various entry points (SJS modules) and better extract shared bundles reducing the total size of all JS through smarter splitting and sharing. It also allows you to infuse bundles into the HTML without having to worry about dependencies or links going stale. The list goes on and it's all genuinely useful real-world stuff; the reason I'm raising this issue is because I'm not sure how scalajs-bundler's current approach facilitates those kind of needs. If anything it seems to hinder it and I think that's a huge show-stopper.

Also I think that if a change along the lines of scala-js/scala-js#2833 is made to Scala.JS itself, solving this issue will become much, much easier.

[evbo]

Is the basic sbt configuration for multi-module builds documented?

The closest to documentation I found was the approach in this PR by setting dependsOn to another scala project that had scalaJSPlugin enabled.

However, unlike regular (non-js) sbt multi-module projects, Intellij doesn't seem recognize imports from my other scalajs modules like it usually does when dependsOn is set (I noticed this is due to the other module's sources not getting saved to my main module's target/scala-2.12/classes folder)?

So instead, until I see a cleaner approach I'm currently copying my otherProject source into sourceManaged of my main, dependent project. Not pretty, but at least I have syntax highlighting and it's only 3 lines of code instead of 2:

sourceGenerators in Compile += Def.task {
      val destination: File = sourceManaged.value / "main" / "otherProject"
      IO.copyDirectory(sourceDirectory.in(otherProject).value / "main" / "scala", destination)
      (destination ** "*" filter(_.isFile)).get.map(_.getAbsoluteFile)
    }

The benefits to separating projects into multiple modules is of course immeasurably huge. I like to keep pure scala functions in my otherProject module so that client-side business logic unit tests (no references to js objects) can run much faster without any javascript compilation.

Does my approach make sense? Is there a cleaner way to bundle multi-module scalajs projects?

[julienrf]

I used to work on a multi-(sbt)-modules project, where module dependencies were expressed using dependsOn. I didn’t have the problem you mention: IntelliJ was able to resolve the code coming from the depended module.

I think this issue was initially about producing multiple JavaScript modules, not about managing multiple sbt modules. I suggest that you open a different issue to report the problem you are facing when using dependsOn.

[evbo]

I agree the OP use-case is more focused on pure scalajs modules, whereas mine is more focused on the special case of packaging scalajs code with pure scala code or "backend" in this case. In that example special case, unlike usual sbt multi-module projects, you don't use backend.dependsOn(frontend), but instead actually call fastOptJS or fullOptJS as a sourceGenerator in backend.

So my comment is more about integrating scalajs as part of a multi-module scala build. So if that's outside the scope of scalajs-bundler, then I'm happy to just live with my workarounds! Otherwise, let me know and I'll try submitting a new issue if there's interest in this use-case, and why my use of dependsOn isn't behaving intuitively when scalajs project is compiled as part of sourceGeneration.

[julienrf]

Your backend has to use the output of the webpack task applied to your frontend. Or you can just use sbt-web.

[evbo]

Yes, I am using the webpack task as I showed in the link. But in doing so dependsOn isn't resulting in necessary classes being copied over to the target folder of the scalajs project the webpack task processes.

pseudo-code representation (if it helps):
backend.sourceGenerate(frontend.dependsOn(otherModule)) (doesn't work)
backend.sourceGenerate(frontend.sourceGenerate(otherModule)) (works)

My work around is to let the scalajs submodule use sourceGeneration to copy other module classes it depends on into its target classes folder (from sourceManaged).

Sorry if this is further polluting this issue with outside topics...

[viktor-podzigun]

Hi @evbo,
if you need some classes from your scala.js/ui sub-module to be visible in your backend sub-module, then you can extract them into a third common/shared sub-module and make two other dependsOn it. You can find many examples online (as in the link you've posted above) how to achieve this, but here is the main idea:

lazy val common = (crossProject.crossType(CrossType.Pure) in file ("common"))
  .settings(...)

lazy val ui = project.in(file("ui"))
  .settings(...)
  .dependsOn(common.js)

lazy val backend = project.in(file("backend"))
  .settings(...)
  .dependsOn(common.jvm)

Please, note the common.js and common.jvm this is important to make it depends on the right sub-module type.

[evbo]

Thanks, yes dependsOn works now that I've correctly set the project as a crossProject:

lazy val common = crossProject(JSPlatform, JVMPlatform).crossType(CrossType.Pure).in(file("common"))

Thanks, and sorry if I've hijacked this issue ;)

[fdietze]

Hi, what needs to be done to support multi-module webpack builds?

My use-case is: I want backend and frontend code to live in the same sbt-subproject and then create two separate javascript bundles, one for the browser, one for a node backend.

Example:

package my.app

import scala.scalajs.js.annotation._
import io.scalajs.nodejs.fs

object Backend {
  @JSExportTopLevel(name = "start", moduleID = "backend")
  def start(): Unit = {
    println("hello from backend: " + Shared.x)
    fs.Fs.mkdirSync("a")
  }
}

object WebApp {
  @JSExportTopLevel(name = "main", moduleID = "frontend")
  def main(): Unit = {
    println("hello browser: " + Shared.x)
  }
}

object Shared {
  val x = 1337
}

build.sbt

import org.scalajs.linker.interface.ModuleInitializer
import org.scalajs.linker.interface.OutputPatterns

ThisBuild / scalaVersion := "3.0.0"

lazy val main = project
  .enablePlugins(ScalaJSPlugin, ScalaJSBundlerPlugin)
  .in(file("main"))
  .settings(
    scalaJSLinkerConfig ~= { _.withModuleKind(ModuleKind.CommonJSModule) },
    libraryDependencies ++= Seq(
      ("net.exoego" %%% "scala-js-nodejs-v14" % "0.13.0").cross(CrossVersion.for3Use2_13),
    ),
    Compile / scalaJSModuleInitializers += {
      ModuleInitializer.mainMethod("my.app.WebApp", "main").withModuleID("frontend")
    },
  )

The error message I get:

[error] (webClient / Compile / fastOptJS) org.scalajs.linker.interface.ReportToLinkerOutputAdapter$UnsupportedLinkerOutputException: Linking returned more than one public module. Full report:
[error] Report(
[error]   publicModules = [
[error]     Module(
[error]   moduleID      = frontend,
[error]   jsFileName    = frontend.js,
[error]   sourceMapName = Some(frontend.js.map),
[error]   moduleKind    = CommonJSModule,
[error] ),
[error] Module(
[error]   moduleID      = backend,
[error]   jsFileName    = backend.js,
[error]   sourceMapName = Some(backend.js.map),
[error]   moduleKind    = CommonJSModule,
[error] )
[error]   ],
[error] )

The message is thrown here:
https://github.com/scala-js/scala-js/blob/master/linker-interface/shared/src/main/scala/org/scalajs/linker/interface/ReportToLinkerOutputAdapter.scala#L61

[viktor-podzigun]

@fdietze as error message suggests

Linking returned more than one public module.

try to comment out one of the methods annotated with @JSExportTopLevel, or remove moduleID prop from it.

You can also try to structure it bit differently, instead of having one sbt module and generate two different bundles out of it, you can extract most of the code into common shared sbt module, and have two additional sbt modules (backend, frontend) that depend on common. See one of my previous comments with idea how to do it in sbt.

[fdietze]

@viktor-podzigun Yes, that's what I have right now and I find it a bit inflexible. Especially if I want to output multiple js bundles for different aws lambda deployments.

Removing one @JSExportTopLevel, e.g. for the frontend and using just a main-function-entrypoint produces the same error.

[fdietze]

Thinking about it, @viktor-podzigun you made me realize that I could have all scala code in a single sbt-subproject and only have separate sbt projects for the different js modules. This way I can easily have different webpack configs for each module.

[viktor-podzigun]

@fdietze happy it was helpful for you :)

This way you will have your code more structured and will overcome the above issue.

Please, don't hesitate to ask if you need help to set it up (one common sbt module and several others that depends on it).

[fdietze]

I just created a proof of concept, let's see how it works out in a bigger project. I still like the idea of a single subproject and ScalaJS itself already supports this. It's just scalajs-bundler which cannot handle multi-module output yet.