Add support for semantic tokens delta

[vinistock]

While investigating performance issues in very large files (3k+ lines), I discovered something which had been off our radar. We always wondered why the Ruby LSP was not performing well in super large files despite our benchmarks showing that the server would respond really fast.

I believe the issue we've been facing is that, for very large files, the number of semantic tokens returned from the server is insane and then editors like VS Code lag when trying to apply all of them in such long files.

By simply not returning any tokens from the server, the lag is gone and the Ruby LSP becomes responsive in large files just like it is in smaller ones.

If this hypothesis is correct, then the resolution is to add support for semantic tokens delta. Essentially, instead of returning all tokens on every request, we return all tokens only once and then the editor will ask for the delta (the difference) in tokens between the previous request and the current one.

This is potentially related to #1756 because since the editor falls behind in processing incoming responses, it may be the culprit for dropping diagnostics coming from the server. By reducing the pressure in the editor, it will not fall behind and might resolve that problem too.

Notes

• Not every editor supports delta requests. We have to check for the client capability during initialization, save it in the global state and pass that along to semantic highlighting
• Currently, we cache the encoded tokens in the document they belong to. I think we may be able to implement this in a somewhat straight forward way if we instead cache the tokens before they are encoded. That will make it significantly easier to compute the delta