Warn about performance regressions

[hanno-becker]

It would be nice if the benchmarking CI could compare new vs. main performance numbers and warn, through a comment on the PR, if there's a target experiencing a major (>5%, say?) regression.

[hanno-becker]

@potsrevennil Do you know how to do this?

[potsrevennil]

I'm thinking about using git note although I've never used it before

For example, we could add summary of the benchmarking result (whenever there's a new commit on main) via git note, which add notes to commit without touching commit itself, thus would not pollute the history.
Then we could just simply diff the notes between different commits on the main branch

But this might required some times to trial and error

[potsrevennil]

Oh, maybe there's an easier way
the benchmarkaction/github-action-benchmark provided alert-threshold and comment-on-alert, this might be able to achieve what we want ?

[hanno-becker]

@potsrevennil That sounds promising indeed!

[mkannwischer]

Done by #424