Refine Lexer API and write tests

[MatthewKosloski]

Currently, there aren't any tests for the compiler. To improve the quality of the software and reduce the introduction of bugs, we should start testing the compiler. Tests should be introduced gradually.

Acceptance Criteria:

• Write rigorous tests that exercise all scenarios of the lexer
• Refine Lexer API if needed to make it easier to test (e.g., introduce interfaces, dependency injection, etc.).