Introduce tokenizing options for full and partial mode

[martin-fleck-at]

This is based on a discussion that started in #1653 where it was discussed that auto-completing the dedent tokens in the indentation-aware lexer messes with the auto-completion in the editor.

Add tokenizing mode to tokenizing method

• Full: We get the full text to tokenize
• Partial: We get only a portion of the text to tokenize

In indentation lexing, we do not auto-complete dedents for partial mode

Please note this PR is based on #1664 which was already approved but has yet to be merged.

[martin-fleck-at]

@aabounegm @msujew FYI

[msujew]

Can you add a completion test to this as well? Just to confirm that it really fixes the completion issue :)

[msujew]

@martin-fleck-at By the way, after this change do you consider this feature to be "complete", at least for most common use cases? I would like to perform the release soon (ideally today, since I'm on a team event next week).

[martin-fleck-at]

@msujew Yes, if that is working, I think we can consider this complete. I'd still love to try it out on our use case first though but I'll try to get this done today. Unfortunately, I am a bit struggling with creating the completion test case. Somehow, the tokens do not appear as I hoped they would but maybe it has something to do with the test function.

[martin-fleck-at]

@msujew I pushed an update with the failing test cases and some other minor comments I received on #1668. I'll quickly see if I can try the changes locally on my use case, i.e., whether it is really just a testing issue or the issue is not properly resolved.

[msujew]

@martin-fleck-at Looking into this, this seems like a more complicated issue. Essentially, in the position with index 1 the completion provider identifies that we're "within" an INDENT token and attempts to "complete" it. This of course doesn't yield anything and the test fails. The completion provider was written in the assumption that whitespace tokens are all hidden.

[martin-fleck-at]

@msujew Yeah, I also had a feeling that something was off. Curiously, when I try that code (I published it locally to a verdaccio npm registry to consume it correctly without copying code), the completion provider I have in my project works. Without this code, it does not. That is why I was not sure whether it was something in the test code or not.

Not sure if we can merge this as it does at least not make things worse it seems. But I am still a bit baffled by the difference in behavior between my use case in the test case.

[msujew]

We can probably fix the outstanding completion issue later. See below. I think we can merge this.

[martin-fleck-at]

Perfect, will push an update in a second. Actually, found another minor issue in the indentation token builder where I used a 0-based column index instead of a 1-based (chevrotain).

[msujew]

the completion provider I have in my project works

Note that in our case, the completion provider fails because it's being triggered right after a INDENT or DEDENT token. If you trigger it after a different token, the change you did makes it behave as expected. That's why I'm approving this in the first place. See also #1653 (reply in thread).