Support for .mill and .mill.scala files in VSCode/Metals (1000USD Bounty)

[lihaoyi]

---

From the maintainer Li Haoyi: I'm putting a 1000USD bounty on this issue, payable by bank transfer on a merged PR implementing this.

---

Mill 0.12.0 is moving towards using build.mill/package.mill files rather than build.sc. Basic IntelliJ support landed in JetBrains/intellij-scala#667, so the next thing to do is get them working in VSCode/Metals. .mill and .mill.scala files already mostly work in IntelliJ/BSP, so this ticket is to get the same support for .mill and .mill.scala files in VSCode/Metals/BSP as well.

The acceptance criteria is as follows:

• Download and unpack https://github.com/com-lihaoyi/mill/releases/download/0.12.0-RC3/0.12.0-RC3-example-depth-large-10-multi-file-builds.zip
• Load the folder's Mill build into Metals
• Inside bar/qux/package.mill
  • Jumping to build.MyModule should bring you to build.mill
  • Jumping to build.MyModule should bring you to mill-scalalib_2.13-0.12.0-RC3-sources.jar/mill/scalalib/package.scala (or equivalent)
• Inside foo/package.mill
  • Jumping to build.MyModule should bring you to build.mill
  • Jumping to build.bar.qux.mymodule should bring you to bar/qux/package.mill
  • Jumping to RootModule should bring you to mill-main_2.13-0.12.0-RC3-sources.jar/mill/package.scala (or equivalent)
• Inside build.mill
  • Jumping to ScalaModule should bring you to mill-scalalib_2.13-0.12.0-RC3-sources.jar/mill/scalalib/ScalaModule.scala (or equivalent)
• Everything should still work if the .mill files are renamed to .mill.scala

For the purposes of this ticket, treating .mill files as vanilla .scala files (with the appropriate classpath and sources) is sufficient. It's not 100% precise, but is good enough for 95% of scenarios, and we can sort out the last 5% in a followup. Notably, we do not want to treat them as .sc files, since things like ScalaCLI directives and other scripty things don't work

Prior Art: scalameta/metals#6752

Dependencies

• [BSP] semanticDbData fails on meta-build #3666
• Add mill extension to supported ones scalameta/scalameta#3989

[lolgab]

For the purposes of this ticket, treating .mill files as vanilla .scala files (with the appropriate classpath and sources) is sufficient. It's not 100% precise, but is good enough for 95% of scenarios, and we can sort out the last 5% in a followup. Notably, we do not want to treat them as .sc files, since things like ScalaCLI directives and other scripty things don't work

I'm not sure that's possible. I tried and Metals can't really get the MyModule definition from package.mill files.
My guess is that we miss the semanticdb files since they are generated for out/generatedSources.dest/build_/... and not for the actual source files. I think we need to properly communicate the link between the generated files and their sources with something like build-server-protocol/build-server-protocol#673
What do you think @lihaoyi?

[lihaoyi]

My guess is that we miss the semanticdb files since they are generated for out/generatedSources.dest/build_/... and not for the actual source files. I think we need to properly communicate the link between the generated files and their sources with something like build-server-protocol/build-server-protocol#673

I think it's worth investigating this theory further. We do control the entire compilation pipeline, and we already patch the ASTs post-parser via a compiler plugin to point at the original non-codegenned files. I remember in earlier testing that I could get navigation working for build/package files if using a .scala file extension back when I initially worked on #3426, which indicates that the code-gen nature of pipeline shouldn't be a blocker

[lihaoyi]

@lolgab just to check, does the navigate/peek on .mymodule work for you, like it did in the screenshot in #3426?

[lihaoyi]

Long term having proper support for wrapped/transformed source files as part of BSP makes sense. Short term, I'm curious how far we can get without it.

[lolgab]

Long term having proper support for wrapped/transformed source files as part of BSP makes sense. Short term, I'm curious how far we can get without it.

@lihaoyi I'm trying to hack it on the Metals side by reading the generated files and parsing the //MILL_ORIGINAL_FILE_PATH= and //MILL_USER_CODE_START_MARKER magic comments.
Let's see if I can manage to make it work and if the Metals maintainers accept the temporary hack.

[lihaoyi]

@lolgab how does metals get the file name and line number information normally? Does it get it from the Scala AST in the compiler? Could it be we're not running the linenumberplugin when generating semanticdb files, so they Scala ASTs are not picking up the fixed filenames and line numbets?

[lolgab]

I'm not sure, but do we even compile the build files before processing? They aren't even valid Scala files.
From what I see we only compile the generated files and so we only have semanticdb data for them.
About line numbers, I think metals has its own way to process line numbers.
I managed to pass the mapping to metals by reading the generated files and parsing the magic comments, but, still, metals doesn't work properly. I will ask for help from Metals' maintainers.

[lihaoyi]

We do not compile the build.mill files before processing, but during compilation the linenumberplugin is meant to go and fix up all the line numbers and file names in the scala AST to point at the original un-processed files.

[lihaoyi]

Going to call this done for now with scalameta/metals#6752 and scalameta/metals-vscode#1543. The MyModule jump doesn't work yet, but all the other stuff does, so although not perfect it's definitely very usable which was the spirit of the original ticket. We can open follow up tickets for improvements. @lolgab I'll close out the bounty using the bank transfer details you shared previously if that works for you