Support the whole Rust grammar

[scrabsha]

expandable works quite well, but it only supports a very reduced subset of Rust.

In order to make it complete, we have to write a state machine that accepts every syntactically valid Rust code (and nothing more!). This is currently done thanks to a 260 line-long invocation of the generate_grammar macro. This is very painful to understand and update properly. It certainly won't scale to the whole Rust grammar.

The other solution is to write a parser for Rust in a dsl that is then turned into a state machine by a custom compiler, and replace the generate_grammar stuff with this shiny state machine. This means more code to maintain, document and test, but this would make future development so much easier that it's worth it.

Here are the different steps we will follow:

• feat: Add parser DSL compiler code #30 Write a compiler that generates the state machine. The output language will be plain Rust code that is (hopefully) debuggable. The input language won't be fancy:
  • No variables.
  • No nested expressions, except blocks (needed for conditions).
  • Very limited conditionals (just a bunch of builtins).
  • Primitive loops (is it really needed?).
  • No function return value.
• feat: converge on transitions instead of converging on final state #34 Modify the state machine handling logic so that it converges on transitions instead of converging on states. This will allow us to accommodate cases where the generated state machine is suboptimal and pushes unnecessary values onto its stack (otherwise, this would result in never-ending checking).
• feat: add a subset of the Rust language as grammar #35 Write a parser of the Rust language in the DSL. We could take inspiration from the Ferrocene Language Specifications, the The Rust Reference, rustc_parse or even syn.
• refactor: replace the state machine with the generated parser machinery #36 Get rid of the generate_grammar state machine, use the generated state machine instead.
• feat: allow || operator in grammar_gen conditions #39 Add support for the || operator in grammar_gen
• Add the whole Rust grammar to grammar.rs
  • feat: add a subset of the Rust language as grammar #35
  • feat: add support for unqualified path expressions #40
  • feat: add support for tuple expressions and grouped expressions #42
  • feat: add support for index expression, loop, while and for loops  #43
  • feat: add (almost) full pattern support #44
  • feat: add path parsing #47
  • feat: add macro call support #52
  • test: add test for underscore expression #54
  • feat: add support for visibilty and struct definitions #55
  • feat: allow grammar-gen functions to return atoms #56
  • feat: add support for missing fragments #57
  • feat: match expressions #58
• Write custom tests.
• Set up a solid test harness. For instance, we could parse code from crates.io, or the rustc test suite. weird_exprs.rs may be a challenging one :D
• ...
• Profit!

It could be interesting to benchmark the parser, but I don't consider it an important item.