Current LSP backend is not compositional

[favonia]

It should be noted that autocompletion, semantic tokens, document synchronization, go to definition/declaration, find references, hover, etc. are all out of the scope of asai. The current LSP backend is exclusive and makes it difficult to create an LSP server that has error reporting and other features. This has to be changed.

[kentookura]

After having implemented quite a few LSP features I can give some feedback on this.

In a nutshell, I don't know what a compositional asai lsp handler would look like. Any serious language server implementation would need to do so much custom stuff that I don't know if it makes sense for the asai library to contain the main loop. I think that asai should ship with the Lsp_shims module (probably renamed) and provide useful handler combinators for fine-grained accumulation and reporting of diagnostics, see #185, #186.

Now granted, the current lsp implementation contains a lot of useful code, but I think it makes sense to publish this separately. Lsp_eio would be extremely handy even outside of asai. This repository could then contain an example lsp implementation shows how to use this hypothetical lsp library together with the diagnostic capabilities of asai. I think this would be a valuable resource for language implementers, whether they use asai or not!

[TOTBWF]

For some context: the original design I had for Asai had it tracking arbitrary source-span related information, not just diagnostics. The idea was the Reporter would always know your current source location, so you could handle things like identifier references/type information/semantic tokens/diagnostics/etc all with the same machinery.

[TOTBWF]

One possible point in the design space is to factor out the Range machinery into its own package, along with a glorified Reader effect that lets you track the current span. You could then have a zoo of packages for each individual language "feature" (identifier references, etc).

Tying it all back into an LSP server would be tricky though. I see a couple of options:

• Have a big LSP package that provides handlers for all the per-feature libraries. Note that this requires us to depend on the whole zoo.
• Have an lsp-eio-core package that creates effects on a per-message basis, and then provide libraries that act as a sort of mixin system. The dream code is something like

Lsp_core.default_handlers @@ fun () -> (* Big effect handler that ignores all messages we haven't handled already *)
Lsp_declarations.handle_hover @@ fun () ->
Lsp_identifiers.handle_goto_definition @@ fun () ->
Lsp_asai.handle_diagnostics @@ fun () ->
Lsp_core.run_event_loop
  ... (* All the stuff that actually does file loading etc goes here *)

[kentookura]

The idea was the Reporter would always know your current source location, so you could handle things like identifier references/type information/semantic tokens/diagnostics/etc all with the same machinery.

In that case, I think that this goal has (to some extent) been achieved. We keep track of location info up until the final phase of compilation, and it allows me to write, say, a document symbols handler like this:

https://git.sr.ht/~jonsterling/ocaml-forester/tree/82ef7b96419e41776fdb58b083682a845df5c754/item/lib/language_server/Document_symbols.ml

https://asciinema.org/a/N0HQ0dVWpbisUrsZggdjxYncI