Lexer
This article is a draft and work in progress. Feel free to comment and discuss on the issue tracker.
The lexer provides a stream of token available one by one with the method:
public Token getNextToken();
A token represents a lexem combined with kind and position inside the source code. The token definition can be found inside the repositry.