Fix `tests bench` mac-taskpolicy handling in benchmarks #499

potsrevennil · 2024-12-04T03:47:21Z

Add riscv64 toolchain support #449 accidentally broke tests bench cli parsing on mac when mac-taskpolicy option is presented.
tests bench -c M1 --run-as-root would still work as expected

mkannwischer

Please clarify your commit message (also the first line is too long) and describe the bug this is fixing.

My understanding is that --mac-taskpolicy didnt work at all anymore.
This doesn't affect the (more common) benchmarks on the high-performance cores of the M1, as for that you would just run

tests bench -c M1 --run-as-root

Corrrect?

potsrevennil · 2024-12-04T04:07:00Z

My understanding is that --mac-taskpolicy didnt work at all anymore. This doesn't affect the (more common) benchmarks on the high-performance cores of the M1, as for that you would just run
tests bench -c M1 --run-as-root
Corrrect?

Yes

Fixed a bug introduced in PR #449, where the exec_wrapper variable was no longer in scope. This caused the script to abort if --mac-taskpolicy was provided, as it attempted to check the value of exec_wrapper. Signed-off-by: Thing-han, Lim <[email protected]>

mkannwischer

Thanks @potsrevennil for the minmal change!

I have checked that

tests bench -c M1 --run-as-root --mac-taskpolicy background

works now.

I also checked that

tests bench -c M1 --run-as-root

still works.

oqs-bot

Arm Cortex-A76 (Raspberry Pi 5) benchmarks

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`29196` cycles	`29176` cycles	`1.00`
`ML-KEM-512 encaps`	`35819` cycles	`35851` cycles	`1.00`
`ML-KEM-512 decaps`	`46703` cycles	`46652` cycles	`1.00`
`ML-KEM-768 keypair`	`49182` cycles	`49175` cycles	`1.00`
`ML-KEM-768 encaps`	`55793` cycles	`55841` cycles	`1.00`
`ML-KEM-768 decaps`	`71077` cycles	`71007` cycles	`1.00`
`ML-KEM-1024 keypair`	`72096` cycles	`72212` cycles	`1.00`
`ML-KEM-1024 encaps`	`81627` cycles	`81519` cycles	`1.00`
`ML-KEM-1024 decaps`	`101987` cycles	`102034` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 4th gen (c7i)

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`14116` cycles	`13888` cycles	`1.02`
`ML-KEM-512 encaps`	`17988` cycles	`18214` cycles	`0.99`
`ML-KEM-512 decaps`	`24290` cycles	`24098` cycles	`1.01`
`ML-KEM-768 keypair`	`22522` cycles	`22457` cycles	`1.00`
`ML-KEM-768 encaps`	`24652` cycles	`24537` cycles	`1.00`
`ML-KEM-768 decaps`	`32837` cycles	`32556` cycles	`1.01`
`ML-KEM-1024 keypair`	`32025` cycles	`32065` cycles	`1.00`
`ML-KEM-1024 encaps`	`35686` cycles	`35693` cycles	`1.00`
`ML-KEM-1024 decaps`	`47180` cycles	`47173` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 3rd gen (c6i)

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`20329` cycles	`20323` cycles	`1.00`
`ML-KEM-512 encaps`	`27081` cycles	`27096` cycles	`1.00`
`ML-KEM-512 decaps`	`36452` cycles	`36212` cycles	`1.01`
`ML-KEM-768 keypair`	`34851` cycles	`34888` cycles	`1.00`
`ML-KEM-768 encaps`	`38222` cycles	`38241` cycles	`1.00`
`ML-KEM-768 decaps`	`51366` cycles	`51401` cycles	`1.00`
`ML-KEM-1024 keypair`	`47866` cycles	`47930` cycles	`1.00`
`ML-KEM-1024 encaps`	`54085` cycles	`54126` cycles	`1.00`
`ML-KEM-1024 decaps`	`72068` cycles	`72068` cycles	`1`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 3rd gen (c6a)

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`18188` cycles	`18109` cycles	`1.00`
`ML-KEM-512 encaps`	`23094` cycles	`23084` cycles	`1.00`
`ML-KEM-512 decaps`	`30398` cycles	`30397` cycles	`1.00`
`ML-KEM-768 keypair`	`31014` cycles	`31016` cycles	`1.00`
`ML-KEM-768 encaps`	`33970` cycles	`33970` cycles	`1`
`ML-KEM-768 decaps`	`44750` cycles	`44785` cycles	`1.00`
`ML-KEM-1024 keypair`	`44433` cycles	`44515` cycles	`1.00`
`ML-KEM-1024 encaps`	`49837` cycles	`50033` cycles	`1.00`
`ML-KEM-1024 decaps`	`64362` cycles	`64316` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 4th gen (c7i) (no-opt)

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`36339` cycles	`36221` cycles	`1.00`
`ML-KEM-512 encaps`	`46344` cycles	`46462` cycles	`1.00`
`ML-KEM-512 decaps`	`61562` cycles	`61957` cycles	`0.99`
`ML-KEM-768 keypair`	`58992` cycles	`59001` cycles	`1.00`
`ML-KEM-768 encaps`	`72719` cycles	`73079` cycles	`1.00`
`ML-KEM-768 decaps`	`91294` cycles	`91877` cycles	`0.99`
`ML-KEM-1024 keypair`	`88806` cycles	`88481` cycles	`1.00`
`ML-KEM-1024 encaps`	`109163` cycles	`109310` cycles	`1.00`
`ML-KEM-1024 decaps`	`132965` cycles	`133641` cycles	`0.99`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton4

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`18272` cycles	`18264` cycles	`1.00`
`ML-KEM-512 encaps`	`22489` cycles	`22477` cycles	`1.00`
`ML-KEM-512 decaps`	`29415` cycles	`29397` cycles	`1.00`
`ML-KEM-768 keypair`	`30758` cycles	`30774` cycles	`1.00`
`ML-KEM-768 encaps`	`34061` cycles	`34058` cycles	`1.00`
`ML-KEM-768 decaps`	`43874` cycles	`43909` cycles	`1.00`
`ML-KEM-1024 keypair`	`44474` cycles	`44486` cycles	`1.00`
`ML-KEM-1024 encaps`	`50218` cycles	`50208` cycles	`1.00`
`ML-KEM-1024 decaps`	`63657` cycles	`63635` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A55 (Snapdragon 888) benchmarks

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`58093` cycles	`58015` cycles	`1.00`
`ML-KEM-512 encaps`	`65727` cycles	`65686` cycles	`1.00`
`ML-KEM-512 decaps`	`84446` cycles	`84357` cycles	`1.00`
`ML-KEM-768 keypair`	`98477` cycles	`98206` cycles	`1.00`
`ML-KEM-768 encaps`	`110269` cycles	`111131` cycles	`0.99`
`ML-KEM-768 decaps`	`137228` cycles	`136396` cycles	`1.01`
`ML-KEM-1024 keypair`	`149673` cycles	`148760` cycles	`1.01`
`ML-KEM-1024 encaps`	`167153` cycles	`166100` cycles	`1.01`
`ML-KEM-1024 decaps`	`202921` cycles	`202266` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 3rd gen (c6i) (no-opt)

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`56662` cycles	`56727` cycles	`1.00`
`ML-KEM-512 encaps`	`71746` cycles	`71804` cycles	`1.00`
`ML-KEM-512 decaps`	`96279` cycles	`96365` cycles	`1.00`
`ML-KEM-768 keypair`	`91808` cycles	`91904` cycles	`1.00`
`ML-KEM-768 encaps`	`111435` cycles	`111518` cycles	`1.00`
`ML-KEM-768 decaps`	`144403` cycles	`144547` cycles	`1.00`
`ML-KEM-1024 keypair`	`134500` cycles	`134749` cycles	`1.00`
`ML-KEM-1024 encaps`	`159819` cycles	`160160` cycles	`1.00`
`ML-KEM-1024 decaps`	`201262` cycles	`201556` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 3rd gen (c6a) (no-opt)

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`52553` cycles	`52310` cycles	`1.00`
`ML-KEM-512 encaps`	`67423` cycles	`67744` cycles	`1.00`
`ML-KEM-512 decaps`	`92717` cycles	`92563` cycles	`1.00`
`ML-KEM-768 keypair`	`85083` cycles	`84628` cycles	`1.01`
`ML-KEM-768 encaps`	`104147` cycles	`104797` cycles	`0.99`
`ML-KEM-768 decaps`	`137437` cycles	`138104` cycles	`1.00`
`ML-KEM-1024 keypair`	`124893` cycles	`125419` cycles	`1.00`
`ML-KEM-1024 encaps`	`150100` cycles	`149648` cycles	`1.00`
`ML-KEM-1024 decaps`	`191959` cycles	`192189` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 4th gen (c7a)

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`15114` cycles	`15090` cycles	`1.00`
`ML-KEM-512 encaps`	`19796` cycles	`19751` cycles	`1.00`
`ML-KEM-512 decaps`	`26408` cycles	`26385` cycles	`1.00`
`ML-KEM-768 keypair`	`25628` cycles	`25737` cycles	`1.00`
`ML-KEM-768 encaps`	`28211` cycles	`28202` cycles	`1.00`
`ML-KEM-768 decaps`	`38205` cycles	`38214` cycles	`1.00`
`ML-KEM-1024 keypair`	`35580` cycles	`35507` cycles	`1.00`
`ML-KEM-1024 encaps`	`40809` cycles	`40674` cycles	`1.00`
`ML-KEM-1024 decaps`	`54393` cycles	`54642` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton3

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`19028` cycles	`19031` cycles	`1.00`
`ML-KEM-512 encaps`	`23865` cycles	`23843` cycles	`1.00`
`ML-KEM-512 decaps`	`31235` cycles	`31227` cycles	`1.00`
`ML-KEM-768 keypair`	`32330` cycles	`32334` cycles	`1.00`
`ML-KEM-768 encaps`	`36077` cycles	`36082` cycles	`1.00`
`ML-KEM-768 decaps`	`46483` cycles	`46487` cycles	`1.00`
`ML-KEM-1024 keypair`	`46951` cycles	`46967` cycles	`1.00`
`ML-KEM-1024 encaps`	`53088` cycles	`53083` cycles	`1.00`
`ML-KEM-1024 decaps`	`67379` cycles	`67398` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton2

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`29203` cycles	`29177` cycles	`1.00`
`ML-KEM-512 encaps`	`35818` cycles	`35861` cycles	`1.00`
`ML-KEM-512 decaps`	`46723` cycles	`46667` cycles	`1.00`
`ML-KEM-768 keypair`	`49202` cycles	`49191` cycles	`1.00`
`ML-KEM-768 encaps`	`55815` cycles	`55863` cycles	`1.00`
`ML-KEM-768 decaps`	`71064` cycles	`70978` cycles	`1.00`
`ML-KEM-1024 keypair`	`72106` cycles	`72180` cycles	`1.00`
`ML-KEM-1024 encaps`	`81617` cycles	`81626` cycles	`1.00`
`ML-KEM-1024 decaps`	`102046` cycles	`102071` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton4 (no-opt)

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`41965` cycles	`41910` cycles	`1.00`
`ML-KEM-512 encaps`	`51804` cycles	`51761` cycles	`1.00`
`ML-KEM-512 decaps`	`69319` cycles	`69327` cycles	`1.00`
`ML-KEM-768 keypair`	`69081` cycles	`69066` cycles	`1.00`
`ML-KEM-768 encaps`	`82695` cycles	`82684` cycles	`1.00`
`ML-KEM-768 decaps`	`106407` cycles	`106477` cycles	`1.00`
`ML-KEM-1024 keypair`	`102217` cycles	`102241` cycles	`1.00`
`ML-KEM-1024 encaps`	`120589` cycles	`120658` cycles	`1.00`
`ML-KEM-1024 decaps`	`150488` cycles	`150670` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 4th gen (c7a) (no-opt)

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`45752` cycles	`45701` cycles	`1.00`
`ML-KEM-512 encaps`	`58825` cycles	`58771` cycles	`1.00`
`ML-KEM-512 decaps`	`80125` cycles	`80002` cycles	`1.00`
`ML-KEM-768 keypair`	`74656` cycles	`74611` cycles	`1.00`
`ML-KEM-768 encaps`	`91566` cycles	`91454` cycles	`1.00`
`ML-KEM-768 decaps`	`120271` cycles	`120186` cycles	`1.00`
`ML-KEM-1024 keypair`	`109905` cycles	`109878` cycles	`1.00`
`ML-KEM-1024 encaps`	`131081` cycles	`131020` cycles	`1.00`
`ML-KEM-1024 decaps`	`167542` cycles	`167852` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton3 (no-opt)

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`45368` cycles	`45360` cycles	`1.00`
`ML-KEM-512 encaps`	`56182` cycles	`56169` cycles	`1.00`
`ML-KEM-512 decaps`	`75118` cycles	`75087` cycles	`1.00`
`ML-KEM-768 keypair`	`74836` cycles	`74867` cycles	`1.00`
`ML-KEM-768 encaps`	`89152` cycles	`89190` cycles	`1.00`
`ML-KEM-768 decaps`	`114539` cycles	`114549` cycles	`1.00`
`ML-KEM-1024 keypair`	`111073` cycles	`111084` cycles	`1.00`
`ML-KEM-1024 encaps`	`129969` cycles	`129926` cycles	`1.00`
`ML-KEM-1024 decaps`	`162529` cycles	`162491` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton2 (no-opt)

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`71188` cycles	`71123` cycles	`1.00`
`ML-KEM-512 encaps`	`87806` cycles	`87833` cycles	`1.00`
`ML-KEM-512 decaps`	`118215` cycles	`118150` cycles	`1.00`
`ML-KEM-768 keypair`	`117614` cycles	`117374` cycles	`1.00`
`ML-KEM-768 encaps`	`139483` cycles	`139258` cycles	`1.00`
`ML-KEM-768 decaps`	`180195` cycles	`180017` cycles	`1.00`
`ML-KEM-1024 keypair`	`174952` cycles	`175258` cycles	`1.00`
`ML-KEM-1024 encaps`	`202826` cycles	`202709` cycles	`1.00`
`ML-KEM-1024 decaps`	`254472` cycles	`254727` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A72 (Raspberry Pi 4) benchmarks

Benchmark suite	Current: `28988c9`	Previous: `f875b7f`	Ratio
`ML-KEM-512 keypair`	`51463` cycles	`51295` cycles	`1.00`
`ML-KEM-512 encaps`	`58500` cycles	`58341` cycles	`1.00`
`ML-KEM-512 decaps`	`75049` cycles	`75202` cycles	`1.00`
`ML-KEM-768 keypair`	`87772` cycles	`87188` cycles	`1.01`
`ML-KEM-768 encaps`	`96204` cycles	`96260` cycles	`1.00`
`ML-KEM-768 decaps`	`121041` cycles	`120407` cycles	`1.01`
`ML-KEM-1024 keypair`	`131526` cycles	`131465` cycles	`1.00`
`ML-KEM-1024 encaps`	`145938` cycles	`146288` cycles	`1.00`
`ML-KEM-1024 decaps`	`176860` cycles	`178004` cycles	`0.99`

This comment was automatically generated by workflow using github-action-benchmark.

potsrevennil force-pushed the fix-tests-bench-mac branch from e803c01 to 1aa7b88 Compare December 4, 2024 03:49

potsrevennil marked this pull request as ready for review December 4, 2024 03:53

potsrevennil requested a review from a team as a code owner December 4, 2024 03:53

mkannwischer requested changes Dec 4, 2024

View reviewed changes

potsrevennil force-pushed the fix-tests-bench-mac branch from 1aa7b88 to d783ca5 Compare December 4, 2024 04:06

potsrevennil force-pushed the fix-tests-bench-mac branch from d783ca5 to ed56b4e Compare December 4, 2024 04:20

mkannwischer force-pushed the fix-tests-bench-mac branch from ed56b4e to 0dcd556 Compare December 4, 2024 05:03

potsrevennil force-pushed the fix-tests-bench-mac branch from 0dcd556 to 4c04613 Compare December 4, 2024 05:23

potsrevennil force-pushed the fix-tests-bench-mac branch from 4c04613 to 28988c9 Compare December 4, 2024 05:28

mkannwischer approved these changes Dec 4, 2024

View reviewed changes

mkannwischer added the benchmark this PR should be benchmarked in CI label Dec 4, 2024

oqs-bot reviewed Dec 4, 2024

View reviewed changes

mkannwischer changed the title ~~Fix tests bench command on mac when parsing mac-taskpolicy and exec-wrapper~~ Fix tests bench mac-taskpolicy handling in benchmarks Dec 4, 2024

mkannwischer merged commit 1f864c7 into main Dec 4, 2024
72 checks passed

mkannwischer deleted the fix-tests-bench-mac branch December 4, 2024 05:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `tests bench` mac-taskpolicy handling in benchmarks #499

Fix `tests bench` mac-taskpolicy handling in benchmarks #499

potsrevennil commented Dec 4, 2024 •

edited

Loading

mkannwischer left a comment

potsrevennil commented Dec 4, 2024

mkannwischer left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

Fix tests bench mac-taskpolicy handling in benchmarks #499

Fix tests bench mac-taskpolicy handling in benchmarks #499

Conversation

potsrevennil commented Dec 4, 2024 • edited Loading

mkannwischer left a comment

Choose a reason for hiding this comment

potsrevennil commented Dec 4, 2024

mkannwischer left a comment

Choose a reason for hiding this comment

oqs-bot left a comment

Choose a reason for hiding this comment

Arm Cortex-A76 (Raspberry Pi 5) benchmarks

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 4th gen (c7i)

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 3rd gen (c6i)

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 3rd gen (c6a)

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 4th gen (c7i) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton4

oqs-bot left a comment

Choose a reason for hiding this comment

Arm Cortex-A55 (Snapdragon 888) benchmarks

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 3rd gen (c6i) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 3rd gen (c6a) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 4th gen (c7a)

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton3

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton2

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton4 (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 4th gen (c7a) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton3 (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton2 (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Arm Cortex-A72 (Raspberry Pi 4) benchmarks

Fix `tests bench` mac-taskpolicy handling in benchmarks #499

Fix `tests bench` mac-taskpolicy handling in benchmarks #499

potsrevennil commented Dec 4, 2024 •

edited

Loading