Monarch: Parentheses are grouped as a single token

[vmlopezr]

Hello, i'm trying to write some logic for custom auto-complete but am having issues with the tokenizer. Whenever adjacent parantheses are encountered, they are grouped as one token.

This causes issues when cases such as 'function(function2())' come up. Is there a way to separate each parenthesis character as separate tokens?

monaco-editor version: 0.30.1
Browser: Firefox, Chrome, Safari
OS: MacOS
Playground code that reproduces the issue:
See:
https://microsoft.github.io/monaco-editor/monarch.html

• Press F1 and select 'Developer: Inspect Tokens'.
• Move the cursor to the parentheses at 'main()'. See that '()' is listed as a single token.

Expected:

• Each '(', ')' should be a individual token.

[rcjsuen]

Unrelated but looks like there's a totally different where the text's X offset is off as the ()) starts outside the hover... 🤔