Add initial Semantic token

[6cdh]

Supported tokens: (variable function string number regexp)
Supported modifiers: (definition)
Supported method: textDocument/semanticTokens/range and textDocument/semanticTokens/full.

There are some difficulties to support tokens like comment, struct, class, etc.

I don't know how to implement a embeded sexp comment reader. And struct, class, etc requires a full functional type infer.

image:

image with a semantic highlight plugin which gives different variable different color:

[6cdh]

This PR has a serious problems, causes latency problem and makes other requests slower to response because it does not run concurrently.

[6cdh]

I have no idea how to make it runs concurrently. That method needs to access doc-text which is a lsp-editor% instance. It's basically a text buffer implementation. But later requests might modify that text buffer. So we need to lock it, then it leaves no room to runs the semantic token request concurrently.

[dannypsnl]

How about let others task can interrupt "semantic tokens"?

[6cdh]

It needs to access the text buffer during the whole process.

It obtains the text contained in the text buffer, calculate the tokens, then also use the text buffer class method to convert the token positions from character position format into line/character position format.

If it got interrupted in the middle of calculate tokens then the text buffer might got updated and produce unpredictable result.

[6cdh]

I found we can use line/character style position directly after I states the whole process before.

And there is another problem that if people types quickly, then it can produce multiple requests. I guess we can always process only one request, and let new request replace the old processing one, like how we did for the traverse that send diagnostics.

[6cdh]

It simply copy the text buffer and use it during processing.

It has usable latency now, although the fan is running loudly.

[6cdh]

The design that new incoming task replace old running task can't apply. Because client can reuse the result of old requests, for example, insert some whitespaces, then client can calculate the offset and keeps old content highlight. But if the interrupted task returns an empty result, then the highlight would lost. The interrupted task can't just give up to return, because client might wait that request forever and cause resource leak.

I think this PR is completed now.

[jeapostrophe]

Thank you!