Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: add variable support in rust-grammar-dpdfa/grammar-gen #61

Merged
merged 14 commits into from
Sep 16, 2024
Merged

refactor: add variable support in rust-grammar-dpdfa/grammar-gen #61

merged 14 commits into from
Sep 16, 2024

Conversation

scrabsha
Copy link
Owner

@scrabsha scrabsha commented Sep 15, 2024 •
edited
Loading

Well, it turns out it is not that convenient to write a Rust parser without using variables. Who could predict? 🤡

This MR completely changes grammar-gen and rust-grammar-dpdfa so that functions can take arguments. Internally, rust-grammar-dpdfa is now backed by a virtual machine that runs its own bytecode, and grammar-gen is a compiler that targets this very bytecode. I may add loops in the future, I don't know.

For what it's worth, the previous approach was to use a pushdown automation, which did not allow for variables and return values.

I'm actually surprised that most of the debugging I had to do was in the virtual machine implementation, and I got the codegen correct on the first attempt.

This is going to make my work on #29 massively simpler.

@scrabsha scrabsha changed the title REFACTOR: add variable support in rust-grammar-dpdfa/grammar-gen refactor: add variable support in rust-grammar-dpdfa/grammar-gen Sep 15, 2024
scrabsha
scrabsha commented Sep 15, 2024
View reviewed changes
grammar-gen/src/parse.rs Outdated Show resolved Hide resolved
rust-grammar-dpdfa/src/rt.rs Show resolved Hide resolved
rust-grammar-dpdfa/src/rt.rs Outdated Show resolved Hide resolved
rust-grammar-dpdfa/src/rt.rs Outdated Show resolved Hide resolved
rust-grammar-dpdfa/src/rt.rs Show resolved Hide resolved
rust-grammar-dpdfa/src/rt.rs Outdated Show resolved Hide resolved
grammar-gen/src/codegen.rs Outdated Show resolved Hide resolved
@scrabsha scrabsha merged commit 3abd754 into main Sep 16, 2024
11 checks passed
@scrabsha scrabsha deleted the push-upvuxxwzqwwq branch September 16, 2024 20:08
@scrabsha scrabsha mentioned this pull request Oct 10, 2024
Merged
scrabsha added a commit that referenced this pull request Oct 20, 2024
@scrabsha
chore: release v0.1.1 (#76)
bdfa65f 
## 🤖 New release
* `expandable`: 0.1.0 -> 0.1.1
* `expandable-impl`: 0.1.0 -> 0.1.1
* `rust-grammar-dpdfa`: 0.1.0 -> 0.1.1 (✓ API compatible changes)

<details><summary><i><b>Changelog</b></i></summary><p>

## `expandable`
<blockquote>

##
[0.1.1](expandable-v0.1.0...expandable-v0.1.1)
- 2024-10-20

### Added

- add release-plz worklow
([#75](#75))
- use the "vec! with -> instead of ." example in the doc
([#73](#73))
- add new background illustration
([#62](#62))
- match expressions
([#58](#58))
- add support for missing fragments
([#57](#57))
- add support for visibilty and struct definitions
([#55](#55))
- add missing [expandable::*] macros
([#53](#53))
- add macro call support
([#52](#52))
- add path parsing
([#47](#47))
- add (almost) full pattern support
([#44](#44))
- rework ordering in the "expected XXX, YYY, ZZZ" error message
([#45](#45))
- add support for index expression, loop, while and for loops
([#43](#43))
- add support for tuple expressions and grouped expressions
([#42](#42))
- avoid printing a comically large amount of tokens on error
([#41](#41))
- add support for unqualified path expressions
([#40](#40))
- add a subset of the Rust language as grammar
([#35](#35))
- converge on transitions instead of converging on final state
([#34](#34))
- add support for logic and, or and range expressions
([#33](#33))
- sort expected tokens lexicographically before printing them
([#32](#32))
- Add parser DSL compiler code
([#30](#30))
- parse let statements in block expression and function body
([#28](#28))
- parse statements
([#27](#27))
- parse break and return expressions
([#26](#26))
- parse block expressions and const generics
([#24](#24))
- parse generics in function and method calls
([#23](#23))
- parse <expr> . <something>
([#22](#22))
- add support for array expressions
([#20](#20))
- parse all the tokens of the Rust language
([#16](#16))
- add arithmetic, bit and comparison expression parsing
([#14](#14))

### Fixed

- add link to docs, add missing readme link
([#82](#82))
- add the include field to all published crates
([#80](#80))
- handle newer rustdoc-json output
([#81](#81))
- set up workspace-level dependencies
([#78](#78))
- Add description and license field to the crates
([#77](#77))
- propagate "no repetitions" when checking 0-n repetitions
([#59](#59))
- honour newer rustc/clippy warnings
([#31](#31))
- add license file
([#19](#19))

### Other

- fix latest clippy lints
([#74](#74))
- update to the new rustfmt style
([#64](#64))
- add variable support in `rust-grammar-dpdfa`/`grammar-gen`
([#61](#61))
- add test for underscore expression
([#54](#54))
- bump MSRV to 1.70
([#51](#51))
- check that generated.rs is up to date in the CI
([#38](#38))
- add rustfmt.toml and cool formatting settings
([#18](#18))
- add backend documentation
([#11](#11))
- add semantic.yml file
([#13](#13))
- Don't lower `=` to `EqualEqual` in the frontend
([#12](#12))
- Add subtraction, multiplication and equality, rework function
arguments ([#10](#10))
- Add function call parsing
([#8](#8))
- Bring back the repetition stack check
([#7](#7))
- Improve the crate-level doc and readme
([#4](#4))
- Update README and crate docs to show 1.65 as MSRV, not 1.56
([#5](#5))
- Add a MSRV policy
([#3](#3))
- Set 1.65 as MSRV in CI
- Only check repetition nesting
([#2](#2))
- Add parsing for if/if-else expressions
([#1](#1))
- find find -> find
- Format trybuild tests
- Check trybuild tests as well
- Force nigthtly clippy install
- Nightly?
- New cache (we need nightly 🥺)
- CI?
- I forgot another part of the frontend :/
- Update trybuild oracles
- Make the test cuter
- I forgot the frontend :/
- State the architecture invariants
- Rework the grammar and transition machinery
- Fix alignment (2)
- Fix alignment
- Update root readme
- Make entry point macro expansion smaller
- Let's make this image funnier
- Improve doc
- Add top image
- Oops - I forgot to add the error message
- Report errors when the repetition nesting does not match
- Reduce our usage of syn
- Ok I actually missed a file
- Add README
- Actually there were more usages of `join`
- Don't rely too much on the `proc_macro_span` feature
- Le justfile
- Very bad stack machine -> full-blown stack machine
- Rework user-facing api, add top-level documentation
- Update tests
- Add tld doc (STILL NEED HUGE WORK)
- Remove warnings
- Rework tests
- Remove warnings
- Add test
- Better error message
- Add the proc_macro entry point
- Add "the whole point" mp4 file
- Fix Cargo.toml
- Documentation + plenty of smol fixes
- Error i guess
- Update test cases
- Make it slightly easier to transform TokenTreeKind -> TokenTree
- Spans everywhere
- Cleaning
- Documentation + remove useless tests
- Work
- Cool checkpoint
- I should have committed earlier 😅
</blockquote>

## `expandable-impl`
<blockquote>

##
[0.1.1](expandable-impl-v0.1.0...expandable-impl-v0.1.1)
- 2024-10-20

### Added

- add support for missing fragments
([#57](#57))
- allow grammar-gen functions to return atoms
([#56](#56))
- add missing [expandable::*] macros
([#53](#53))
- generate a possible invalid expansion when an error is found
([#49](#49))
- add path parsing
([#47](#47))
- add (almost) full pattern support
([#44](#44))
- rework ordering in the "expected XXX, YYY, ZZZ" error message
([#45](#45))
- add support for unqualified path expressions
([#40](#40))
- converge on transitions instead of converging on final state
([#34](#34))
- add support for logic and, or and range expressions
([#33](#33))
- parse let statements in block expression and function body
([#28](#28))
- parse statements
([#27](#27))
- parse break and return expressions
([#26](#26))
- parse block expressions and const generics
([#24](#24))
- parse generics in function and method calls
([#23](#23))
- parse <expr> . <something>
([#22](#22))
- add support for array expressions
([#20](#20))
- parse all the tokens of the Rust language
([#16](#16))
- add arithmetic, bit and comparison expression parsing
([#14](#14))

### Fixed

- symlink the doc directory in the expandable-impl crate root
([#83](#83))
- add link to docs, add missing readme link
([#82](#82))
- add the include field to all published crates
([#80](#80))
- set up workspace-level dependencies
([#78](#78))
- Add description and license field to the crates
([#77](#77))
- propagate "no repetitions" when checking 0-n repetitions
([#59](#59))
- honour newer rustc/clippy warnings
([#31](#31))
- add license file
([#19](#19))
- Add the missing AfterIf transitions
([#17](#17))

### Other

- fix latest clippy lints
([#74](#74))
- update to the new rustfmt style
([#64](#64))
- add variable support in `rust-grammar-dpdfa`/`grammar-gen`
([#61](#61))
- bump MSRV to 1.70
([#51](#51))
- make the expansion check fully deterministic
([#50](#50))
- replace the state machine with the generated parser machinery
([#36](#36))
- add a transition inheritance system
([#21](#21))
- add rustfmt.toml and cool formatting settings
([#18](#18))
- compute the token descriptions before checking the expansion
([#15](#15))
- add backend documentation
([#11](#11))
- Don't lower `=` to `EqualEqual` in the frontend
([#12](#12))
- Add subtraction, multiplication and equality, rework function
arguments ([#10](#10))
- Add function call parsing
([#8](#8))
- Fix an embarassing bug in the "stack size reduction" optimization
([#9](#9))
- Bring back the repetition stack check
([#7](#7))
- Add debug-only span
([#6](#6))
- Add a MSRV policy
([#3](#3))
- Only check repetition nesting
([#2](#2))
- Add parsing for if/if-else expressions
([#1](#1))
- Make FragmentKind public, fix public API on stable
- Make sure we don't accidentally keep some stack symbols
- I forgot the macros :/
- Add all the keywords in the Rust language
- State the architecture invariants
- Rework the grammar and transition machinery
- Remove intermediate test
- Report errors when the repetition nesting does not match
- Keep the repetition stack when parsing the macro matcher
- Very bad stack machine -> full-blown stack machine
- Remove warnings
- Gitigrone
- Check that macro expansion is complete
- Better error message
- Move the impl into a separate impl crate
</blockquote>

## `rust-grammar-dpdfa`
<blockquote>

##
[0.1.1](rust-grammar-dpdfa-v0.1.0...rust-grammar-dpdfa-v0.1.1)
- 2024-10-20

### Fixed

- add the include field to all published crates
([#80](#80))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/MarcoIeni/release-plz/).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@scrabsha