Benchmark Regression alert

[potsrevennil]

• Resolves Show performance difference to main in benchmark output of CI #223

• The alert threshold for performance regression is set to 103% for now, let me know if there's any better suggestion for the threshold.

• If the threshold is exceeded, the bench bot will automatically leave a comment on PR like this:

potsrevennil#27

• There will also be GH job summary generated.

• There's some other options provided by the benchmark action (but not set in this pr) which one might be useful as well:

  • fail-on-alert
  • fail-threshold
  • alert-comment-cc-users

[hanno-becker]

It's not yet clear to me that this will trigger at PR-time not only post-merge. @potsrevennil can you explain?

[oqs-bot]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Intel Xeon 4th gen (c7i) (no-opt)'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.03.

Benchmark suite	Current: 372c3ef	Previous: ea2193a	Ratio
ML-KEM-512 keypair	37589 cycles	34692 cycles	1.08
ML-KEM-512 encaps	46894 cycles	43244 cycles	1.08
ML-KEM-512 decaps	62890 cycles	58124 cycles	1.08
ML-KEM-768 keypair	62723 cycles	57750 cycles	1.09
ML-KEM-768 encaps	75054 cycles	68963 cycles	1.09
ML-KEM-768 decaps	96720 cycles	88937 cycles	1.09
ML-KEM-1024 keypair	92949 cycles	85238 cycles	1.09
ML-KEM-1024 encaps	107828 cycles	98422 cycles	1.10
ML-KEM-1024 decaps	141106 cycles	128871 cycles	1.09

This comment was automatically generated by workflow using github-action-benchmark.

[oqs-bot]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Intel Xeon 3rd gen (c6i) (no-opt)'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.03.

Benchmark suite	Current: 372c3ef	Previous: ea2193a	Ratio
ML-KEM-512 keypair	37535 cycles	35693 cycles	1.05
ML-KEM-512 encaps	46908 cycles	44458 cycles	1.06
ML-KEM-512 decaps	63045 cycles	59592 cycles	1.06
ML-KEM-768 keypair	62515 cycles	59467 cycles	1.05
ML-KEM-768 encaps	74847 cycles	71086 cycles	1.05
ML-KEM-768 decaps	96717 cycles	91771 cycles	1.05
ML-KEM-1024 keypair	92823 cycles	88374 cycles	1.05
ML-KEM-1024 encaps	107514 cycles	102244 cycles	1.05
ML-KEM-1024 decaps	141220 cycles	133901 cycles	1.05

This comment was automatically generated by workflow using github-action-benchmark.

[oqs-bot]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Intel Xeon 3rd gen (c6i)'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.15.

Benchmark suite	Current: 10479e5	Previous: ea2193a	Ratio
ML-KEM-512 keypair	16249 cycles	13949 cycles	1.16
ML-KEM-512 encaps	21007 cycles	18084 cycles	1.16
ML-KEM-512 decaps	28477 cycles	23909 cycles	1.19
ML-KEM-768 keypair	26449 cycles	22692 cycles	1.17
ML-KEM-768 encaps	29274 cycles	24353 cycles	1.20
ML-KEM-768 decaps	38266 cycles	32332 cycles	1.18
ML-KEM-1024 keypair	37967 cycles	32392 cycles	1.17
ML-KEM-1024 encaps	41924 cycles	35802 cycles	1.17
ML-KEM-1024 decaps	56315 cycles	47500 cycles	1.19

This comment was automatically generated by workflow using github-action-benchmark.

[oqs-bot]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'AMD EPYC 4th gen (c7a)'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.02.

Benchmark suite	Current: 10479e5	Previous: ea2193a	Ratio
ML-KEM-512 encaps	20412 cycles	19713 cycles	1.04

This comment was automatically generated by workflow using github-action-benchmark.

[mkannwischer]

Okay, we have to resolve the variance first before proceeding with the alerts.
I opened an issue for that: #429

Turning this into a draft PR for now.

[hanno-becker]

Variance should be fixed now that we have moved c7i to a metal instance. @potsrevennil Can you update this PR?

[mkannwischer]

LGTM - thanks @potsrevennil ! I tested this 3 times now and it's stable.

resolved.

[hanno-becker]

Thanks @potsrevennil !