Moved benchmarking project go Github Actions #2129
Conversation
|
As discussed with @nirinchev, it would also be possible to make it run on PR, but there is no way of constraining it on the PR labels. We can still do this check inside the workflow, but the workflow will run nevertheless, and we can make it fail, or in the best case cancel. |
.github/workflows/benchmark-test.yml
Outdated
| with: | ||
| tool: 'benchmarkdotnet' | ||
| # Where the output from the benchmark tool is stored | ||
| output-file-path: .\BenchmarkDotNet.Artifacts\results\PerformanceTests.QueryTests-report-full-compressed.json |
There was a problem hiding this comment.
Should this include the other tests as well?
There was a problem hiding this comment.
Yes it should, I was testing on a subset and I forgot :)
Tests/PerformanceTests/Program.cs
Outdated
| @@ -49,33 +41,9 @@ public static void Main(string[] args) | |||
| .WithArtifactsPath(defaultConfig.ArtifactsPath) | |||
| .AddDiagnoser(MemoryDiagnoser.Default) | |||
| .AddJob(Job.ShortRun.WithToolchain(InProcessEmitToolchain.Instance)) | |||
There was a problem hiding this comment.
TODO: check if these parameters make sense - I used them to get quick results locally, but maybe on CI we'd want a default/long job. The toolchain is also something I used as a bit of a workaround and we may want to adjust later. The problem that I hit was that it looked like the weaver is not running for the generated assembly. I'm guessing there are ways around that, but I didn't feel investigating is justified. We can keep it for now and revisit in another PR.
|
@nirinchev Regarding making the graphs for the different test sets, it's not possible at the moment with that action. Only one file can be specified as the benchmark output, and if I try to run that step multiple times with different |
|
Wouldn't using |
You're correct, I didn't find it in the documentation for |
|
@nirinchev regarding the Jenkins job, I removed the exporter for that from the benchmark. Do you think I should put it back to keep both? |
|
Hm, good point - I guess that it will now fail, so let's remove it. |
Co-authored-by: Nikola Irinchev <[email protected]>
Description
Moved benchmarking project to Github Actions. The workflow takes care of building the wrappers, and cache the results to speed up subsequent runs. In case the wrappers have been modified, those are built again.
For now the benchmarks run only on Windows, and a series of graphs is shown on Github Pages.
Fixes #2119