Add `tests examples` command #621

hanno-becker · 2025-01-07T07:16:09Z

Fix Add support for running all examples intests script #620

This example is meant to demonstrate how to use mlkem-native with a custom FIPS202 impleemntation, so it makes more sense to not even include the default FIPS202 implementation. Signed-off-by: Hanno Becker <[email protected]>

Signed-off-by: Hanno Becker <[email protected]>

We have enough warnings in other places, and those clutter the build and lead to failures with `-Werror`. We also don't have them for the test builds. Signed-off-by: Hanno Becker <[email protected]>

This paves the way for the integration of the examples into `tests`, which is run with cross builds and custom cflags as well. Signed-off-by: Hanno Becker <[email protected]>

oqs-bot

Graviton2 (no-opt)

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`71334` cycles	`71209` cycles	`1.00`
`ML-KEM-512 encaps`	`85172` cycles	`85148` cycles	`1.00`
`ML-KEM-512 decaps`	`112672` cycles	`112558` cycles	`1.00`
`ML-KEM-768 keypair`	`117881` cycles	`117649` cycles	`1.00`
`ML-KEM-768 encaps`	`135496` cycles	`135259` cycles	`1.00`
`ML-KEM-768 decaps`	`172139` cycles	`172092` cycles	`1.00`
`ML-KEM-1024 keypair`	`175547` cycles	`175074` cycles	`1.00`
`ML-KEM-1024 encaps`	`197558` cycles	`197158` cycles	`1.00`
`ML-KEM-1024 decaps`	`243879` cycles	`243409` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

This was flagged by the monobuild, which marks this function as static and realizes that it's never used. Signed-off-by: Hanno Becker <[email protected]>

tiny_sha3 uses a byte-reversed Keccakf1600 presentation for big endian targets, but mlkem-native's FIPS202 frontend uses the standard presentation. This commit fixes this; luckily, the fix merely involves _removing_ byte reversal functions already present in the tiny_sha3 backend implementation of the Keccakf1600 permutation. Signed-off-by: Hanno Becker <[email protected]>

oqs-bot

Intel Xeon 4th gen (c7i)

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`13496` cycles	`13905` cycles	`0.97`
`ML-KEM-512 encaps`	`17479` cycles	`17183` cycles	`1.02`
`ML-KEM-512 decaps`	`22750` cycles	`23015` cycles	`0.99`
`ML-KEM-768 keypair`	`22513` cycles	`22516` cycles	`1.00`
`ML-KEM-768 encaps`	`24463` cycles	`24551` cycles	`1.00`
`ML-KEM-768 decaps`	`32506` cycles	`32511` cycles	`1.00`
`ML-KEM-1024 keypair`	`30017` cycles	`31498` cycles	`0.95`
`ML-KEM-1024 encaps`	`33515` cycles	`35002` cycles	`0.96`
`ML-KEM-1024 decaps`	`44252` cycles	`45920` cycles	`0.96`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 3rd gen (c6a)

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`18146` cycles	`18153` cycles	`1.00`
`ML-KEM-512 encaps`	`23288` cycles	`23220` cycles	`1.00`
`ML-KEM-512 decaps`	`30490` cycles	`30488` cycles	`1.00`
`ML-KEM-768 keypair`	`31080` cycles	`31093` cycles	`1.00`
`ML-KEM-768 encaps`	`34157` cycles	`34161` cycles	`1.00`
`ML-KEM-768 decaps`	`44736` cycles	`44756` cycles	`1.00`
`ML-KEM-1024 keypair`	`44732` cycles	`44612` cycles	`1.00`
`ML-KEM-1024 encaps`	`50030` cycles	`49946` cycles	`1.00`
`ML-KEM-1024 decaps`	`64397` cycles	`64420` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 3rd gen (c6i)

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`20344` cycles	`20337` cycles	`1.00`
`ML-KEM-512 encaps`	`27008` cycles	`27004` cycles	`1.00`
`ML-KEM-512 decaps`	`35798` cycles	`35804` cycles	`1.00`
`ML-KEM-768 keypair`	`34900` cycles	`34889` cycles	`1.00`
`ML-KEM-768 encaps`	`38139` cycles	`38134` cycles	`1.00`
`ML-KEM-768 decaps`	`50964` cycles	`50948` cycles	`1.00`
`ML-KEM-1024 keypair`	`47982` cycles	`47942` cycles	`1.00`
`ML-KEM-1024 encaps`	`54225` cycles	`54087` cycles	`1.00`
`ML-KEM-1024 decaps`	`71649` cycles	`71650` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 4th gen (c7a)

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`15062` cycles	`15061` cycles	`1.00`
`ML-KEM-512 encaps`	`19658` cycles	`19678` cycles	`1.00`
`ML-KEM-512 decaps`	`26302` cycles	`26307` cycles	`1.00`
`ML-KEM-768 keypair`	`25633` cycles	`25623` cycles	`1.00`
`ML-KEM-768 encaps`	`28112` cycles	`28237` cycles	`1.00`
`ML-KEM-768 decaps`	`37900` cycles	`37842` cycles	`1.00`
`ML-KEM-1024 keypair`	`35707` cycles	`35778` cycles	`1.00`
`ML-KEM-1024 encaps`	`40989` cycles	`40988` cycles	`1.00`
`ML-KEM-1024 decaps`	`54514` cycles	`54624` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 4th gen (c7i) (no-opt)

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`34866` cycles	`35009` cycles	`1.00`
`ML-KEM-512 encaps`	`44996` cycles	`44885` cycles	`1.00`
`ML-KEM-512 decaps`	`58905` cycles	`58666` cycles	`1.00`
`ML-KEM-768 keypair`	`59190` cycles	`59076` cycles	`1.00`
`ML-KEM-768 encaps`	`71871` cycles	`71668` cycles	`1.00`
`ML-KEM-768 decaps`	`89402` cycles	`89197` cycles	`1.00`
`ML-KEM-1024 keypair`	`87569` cycles	`87735` cycles	`1.00`
`ML-KEM-1024 encaps`	`104607` cycles	`104117` cycles	`1.00`
`ML-KEM-1024 decaps`	`127761` cycles	`126800` cycles	`1.01`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 3rd gen (c6a) (no-opt)

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`52371` cycles	`52204` cycles	`1.00`
`ML-KEM-512 encaps`	`65393` cycles	`65820` cycles	`0.99`
`ML-KEM-512 decaps`	`88517` cycles	`88470` cycles	`1.00`
`ML-KEM-768 keypair`	`84246` cycles	`84782` cycles	`0.99`
`ML-KEM-768 encaps`	`101980` cycles	`101874` cycles	`1.00`
`ML-KEM-768 decaps`	`131239` cycles	`132136` cycles	`0.99`
`ML-KEM-1024 keypair`	`124570` cycles	`124371` cycles	`1.00`
`ML-KEM-1024 encaps`	`145057` cycles	`145699` cycles	`1.00`
`ML-KEM-1024 decaps`	`182606` cycles	`183707` cycles	`0.99`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 3rd gen (c6i) (no-opt)

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`56608` cycles	`56616` cycles	`1.00`
`ML-KEM-512 encaps`	`69451` cycles	`69468` cycles	`1.00`
`ML-KEM-512 decaps`	`91402` cycles	`91431` cycles	`1.00`
`ML-KEM-768 keypair`	`91889` cycles	`91894` cycles	`1.00`
`ML-KEM-768 encaps`	`107830` cycles	`107791` cycles	`1.00`
`ML-KEM-768 decaps`	`136391` cycles	`136407` cycles	`1.00`
`ML-KEM-1024 keypair`	`134744` cycles	`134894` cycles	`1.00`
`ML-KEM-1024 encaps`	`155292` cycles	`155113` cycles	`1.00`
`ML-KEM-1024 decaps`	`191564` cycles	`191462` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 4th gen (c7a) (no-opt)

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`45715` cycles	`45726` cycles	`1.00`
`ML-KEM-512 encaps`	`56870` cycles	`56884` cycles	`1.00`
`ML-KEM-512 decaps`	`76208` cycles	`76271` cycles	`1.00`
`ML-KEM-768 keypair`	`74484` cycles	`74497` cycles	`1.00`
`ML-KEM-768 encaps`	`88535` cycles	`88577` cycles	`1.00`
`ML-KEM-768 decaps`	`114391` cycles	`114385` cycles	`1.00`
`ML-KEM-1024 keypair`	`109366` cycles	`109408` cycles	`1.00`
`ML-KEM-1024 encaps`	`127488` cycles	`127276` cycles	`1.00`
`ML-KEM-1024 decaps`	`159974` cycles	`159979` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A72 (Raspberry Pi 4) benchmarks

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`51656` cycles	`51888` cycles	`1.00`
`ML-KEM-512 encaps`	`58150` cycles	`58607` cycles	`0.99`
`ML-KEM-512 decaps`	`74784` cycles	`74353` cycles	`1.01`
`ML-KEM-768 keypair`	`87885` cycles	`88452` cycles	`0.99`
`ML-KEM-768 encaps`	`96380` cycles	`97121` cycles	`0.99`
`ML-KEM-768 decaps`	`120250` cycles	`120155` cycles	`1.00`
`ML-KEM-1024 keypair`	`132271` cycles	`131483` cycles	`1.01`
`ML-KEM-1024 encaps`	`144898` cycles	`144906` cycles	`1.00`
`ML-KEM-1024 decaps`	`176067` cycles	`176884` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton3

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`18971` cycles	`18964` cycles	`1.00`
`ML-KEM-512 encaps`	`23554` cycles	`23587` cycles	`1.00`
`ML-KEM-512 decaps`	`30655` cycles	`30671` cycles	`1.00`
`ML-KEM-768 keypair`	`32308` cycles	`32288` cycles	`1.00`
`ML-KEM-768 encaps`	`35884` cycles	`35879` cycles	`1.00`
`ML-KEM-768 decaps`	`46010` cycles	`46054` cycles	`1.00`
`ML-KEM-1024 keypair`	`46613` cycles	`46612` cycles	`1.00`
`ML-KEM-1024 encaps`	`52451` cycles	`52448` cycles	`1.00`
`ML-KEM-1024 decaps`	`66290` cycles	`66249` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton4

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`18135` cycles	`18132` cycles	`1.00`
`ML-KEM-512 encaps`	`22146` cycles	`22162` cycles	`1.00`
`ML-KEM-512 decaps`	`28775` cycles	`28771` cycles	`1.00`
`ML-KEM-768 keypair`	`30569` cycles	`30566` cycles	`1.00`
`ML-KEM-768 encaps`	`33639` cycles	`33619` cycles	`1.00`
`ML-KEM-768 decaps`	`43162` cycles	`43143` cycles	`1.00`
`ML-KEM-1024 keypair`	`44204` cycles	`44192` cycles	`1.00`
`ML-KEM-1024 encaps`	`49675` cycles	`49660` cycles	`1.00`
`ML-KEM-1024 decaps`	`62641` cycles	`62653` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton3 (no-opt)

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`45382` cycles	`45392` cycles	`1.00`
`ML-KEM-512 encaps`	`54230` cycles	`54249` cycles	`1.00`
`ML-KEM-512 decaps`	`71164` cycles	`71163` cycles	`1.00`
`ML-KEM-768 keypair`	`74932` cycles	`74901` cycles	`1.00`
`ML-KEM-768 encaps`	`86199` cycles	`86138` cycles	`1.00`
`ML-KEM-768 decaps`	`108671` cycles	`108641` cycles	`1.00`
`ML-KEM-1024 keypair`	`111108` cycles	`111100` cycles	`1.00`
`ML-KEM-1024 encaps`	`125932` cycles	`125930` cycles	`1.00`
`ML-KEM-1024 decaps`	`154592` cycles	`154638` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton4 (no-opt)

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`42018` cycles	`41955` cycles	`1.00`
`ML-KEM-512 encaps`	`50102` cycles	`50109` cycles	`1.00`
`ML-KEM-512 decaps`	`66103` cycles	`66079` cycles	`1.00`
`ML-KEM-768 keypair`	`69180` cycles	`69182` cycles	`1.00`
`ML-KEM-768 encaps`	`79934` cycles	`79959` cycles	`1.00`
`ML-KEM-768 decaps`	`101127` cycles	`101140` cycles	`1.00`
`ML-KEM-1024 keypair`	`102232` cycles	`102518` cycles	`1.00`
`ML-KEM-1024 encaps`	`117211` cycles	`117437` cycles	`1.00`
`ML-KEM-1024 decaps`	`143664` cycles	`143367` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Bananapi bpi-f3 benchmarks

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`335143` cycles	`335242` cycles	`1.00`
`ML-KEM-512 encaps`	`445850` cycles	`445868` cycles	`1.00`
`ML-KEM-512 decaps`	`593938` cycles	`594184` cycles	`1.00`
`ML-KEM-768 keypair`	`556065` cycles	`556626` cycles	`1.00`
`ML-KEM-768 encaps`	`698088` cycles	`698603` cycles	`1.00`
`ML-KEM-768 decaps`	`889639` cycles	`891242` cycles	`1.00`
`ML-KEM-1024 keypair`	`821474` cycles	`822216` cycles	`1.00`
`ML-KEM-1024 encaps`	`998321` cycles	`999804` cycles	`1.00`
`ML-KEM-1024 decaps`	`1231297` cycles	`1230677` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A55 (Snapdragon 888) benchmarks

Benchmark suite	Current: `d82c33e`	Previous: `66dc45e`	Ratio
`ML-KEM-512 keypair`	`58353` cycles	`58371` cycles	`1.00`
`ML-KEM-512 encaps`	`65778` cycles	`65774` cycles	`1.00`
`ML-KEM-512 decaps`	`84576` cycles	`84596` cycles	`1.00`
`ML-KEM-768 keypair`	`99098` cycles	`99063` cycles	`1.00`
`ML-KEM-768 encaps`	`110378` cycles	`110262` cycles	`1.00`
`ML-KEM-768 decaps`	`137365` cycles	`137503` cycles	`1.00`
`ML-KEM-1024 keypair`	`150125` cycles	`150178` cycles	`1.00`
`ML-KEM-1024 encaps`	`166880` cycles	`166580` cycles	`1.00`
`ML-KEM-1024 decaps`	`202933` cycles	`202401` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

mkannwischer

Thank you @hanno-becker. Mostly looks good.
I found some small things.

examples/bring_your_own_fips202/Makefile

examples/monolithic_build/README.md

examples/custom_backend/mlkem_native/fips202/native/custom/src/sha3.c

examples/bring_your_own_fips202/Makefile

Signed-off-by: Hanno Becker <[email protected]>

mkannwischer

Thanks for the changes!

hanno-becker force-pushed the make_examples branch 6 times, most recently from bd97c45 to d560856 Compare January 7, 2025 13:03

hanno-becker marked this pull request as ready for review January 7, 2025 13:03

hanno-becker requested a review from a team as a code owner January 7, 2025 13:03

hanno-becker force-pushed the make_examples branch 10 times, most recently from 458f73c to 1afa965 Compare January 7, 2025 20:56

hanno-becker changed the title ~~Add makefile targets for examples, and add tests examples command~~ Add tests examples command Jan 7, 2025

hanno-becker force-pushed the make_examples branch from 1afa965 to b1a02d5 Compare January 7, 2025 21:02

hanno-becker marked this pull request as draft January 7, 2025 21:31

hanno-becker force-pushed the make_examples branch 2 times, most recently from ede6d04 to f9951ae Compare January 8, 2025 04:37

hanno-becker added 3 commits January 8, 2025 04:37

Examples: Remove default FIPS202 from bring-your-own-FIPS202

fc15317

This example is meant to demonstrate how to use mlkem-native with a custom FIPS202 impleemntation, so it makes more sense to not even include the default FIPS202 implementation. Signed-off-by: Hanno Becker <[email protected]>

Examples: Fix some signed vs unsigned comparison warnings

5c9400c

Signed-off-by: Hanno Becker <[email protected]>

Examples: Remove #warning message about test RNG

a3ffe92

We have enough warnings in other places, and those clutter the build and lead to failures with `-Werror`. We also don't have them for the test builds. Signed-off-by: Hanno Becker <[email protected]>

hanno-becker force-pushed the make_examples branch 3 times, most recently from 711431c to 5670760 Compare January 8, 2025 04:49

Examples: Use pedantic CFLAGS, allow CROSS_PREFIX and EXEC_WRAPPER

59c1696

This paves the way for the integration of the examples into `tests`, which is run with cross builds and custom cflags as well. Signed-off-by: Hanno Becker <[email protected]>

oqs-bot reviewed Jan 8, 2025

View reviewed changes

hanno-becker added 2 commits January 8, 2025 09:40

Drop unused poly_xxx code

c0fe62c

This was flagged by the monobuild, which marks this function as static and realizes that it's never used. Signed-off-by: Hanno Becker <[email protected]>

hanno-becker force-pushed the make_examples branch from 52610d7 to d82c33e Compare January 8, 2025 09:40

hanno-becker added benchmark this PR should be benchmarked in CI and removed benchmark this PR should be benchmarked in CI labels Jan 8, 2025

oqs-bot reviewed Jan 8, 2025

View reviewed changes

mkannwischer requested changes Jan 8, 2025

View reviewed changes

examples/bring_your_own_fips202/Makefile Outdated Show resolved Hide resolved

examples/monolithic_build/README.md Show resolved Hide resolved

examples/custom_backend/mlkem_native/fips202/native/custom/src/sha3.c Show resolved Hide resolved

mkannwischer reviewed Jan 8, 2025

View reviewed changes

examples/bring_your_own_fips202/Makefile Outdated Show resolved Hide resolved

hanno-becker requested a review from mkannwischer January 8, 2025 13:38

hanno-becker added 2 commits January 8, 2025 13:59

Examples: Make all the default target, map to build

70f039b

Signed-off-by: Hanno Becker <[email protected]>

Examples: Fix duplicates source paths

0d32f96

Signed-off-by: Hanno Becker <[email protected]>

hanno-becker force-pushed the make_examples branch from c88778e to 0d32f96 Compare January 8, 2025 13:59

mkannwischer approved these changes Jan 8, 2025

View reviewed changes

hanno-becker merged commit 5ba851a into main Jan 8, 2025
122 checks passed

hanno-becker deleted the make_examples branch January 8, 2025 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `tests examples` command #621

Add `tests examples` command #621

hanno-becker commented Jan 7, 2025 •

edited

Loading

oqs-bot left a comment •

edited

Loading

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

oqs-bot left a comment

mkannwischer left a comment

mkannwischer left a comment

Add tests examples command #621

Add tests examples command #621

Conversation

hanno-becker commented Jan 7, 2025 • edited Loading

oqs-bot left a comment • edited Loading

Choose a reason for hiding this comment

Graviton2 (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 4th gen (c7i)

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 3rd gen (c6a)

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 3rd gen (c6i)

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 4th gen (c7a)

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 4th gen (c7i) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 3rd gen (c6a) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Intel Xeon 3rd gen (c6i) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

AMD EPYC 4th gen (c7a) (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Arm Cortex-A72 (Raspberry Pi 4) benchmarks

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton3

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton4

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton3 (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Graviton4 (no-opt)

oqs-bot left a comment

Choose a reason for hiding this comment

Bananapi bpi-f3 benchmarks

oqs-bot left a comment

Choose a reason for hiding this comment

Arm Cortex-A55 (Snapdragon 888) benchmarks

mkannwischer left a comment

Choose a reason for hiding this comment

mkannwischer left a comment

Choose a reason for hiding this comment

Add `tests examples` command #621

Add `tests examples` command #621

hanno-becker commented Jan 7, 2025 •

edited

Loading

oqs-bot left a comment •

edited

Loading