Skip to content

Commit

Permalink
Merge pull request #596 from pq-code-package/exec_wrapper_logic
Browse files Browse the repository at this point in the history 
Tests/CI: Move QEMU choice logic from `tests` into workflow
  • Loading branch information
@mkannwischer
mkannwischer authored Dec 31, 2024
2 parents 5b4572e + e340a14 commit 13ca898
Show file tree
Hide file tree
Showing 5 changed files with 8 additions and 26 deletions.
5 changes: 4 additions & 1 deletion .github/actions/functest/action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ inputs:
cross_prefix:
description: Binary prefix for cross compilation
default: ""
exec_wrapper:
description: Binary wrapper for execution (e.g. QEMU)
default: ""
opt:
description: Whether to build opt/non-opt binaries or all (all | opt | no_opt)
default: "all"
Expand Down Expand Up @@ -85,7 +88,7 @@ runs:
- name: ${{ env.MODE }} ${{ inputs.opt }} tests (${{ env.FUNC }}, ${{ env.KAT }}, ${{ env.NISTKAT }})
shell: ${{ env.SHELL }}
run: |
./scripts/tests all --cross-prefix="${{ inputs.cross_prefix }}" --cflags="${{ inputs.cflags }}" --opt=${{ inputs.opt }} --${{ env.FUNC }} --${{ env.KAT }} --${{ env.NISTKAT }} --${{ env.ACVP }} -v
./scripts/tests all --exec-wrapper="${{ inputs.exec_wrapper }}" --cross-prefix="${{ inputs.cross_prefix }}" --cflags="${{ inputs.cflags }}" --opt=${{ inputs.opt }} --${{ env.FUNC }} --${{ env.KAT }} --${{ env.NISTKAT }} --${{ env.ACVP }} -v
- name: Check namespacing ${{ env.MODE }} ${{ inputs.opt }} tests (${{ env.FUNC }}, ${{ env.KAT }}, ${{ env.NISTKAT }})
shell: ${{ env.SHELL }}
run: |
Expand Down
4 changes: 4 additions & 0 deletions .github/actions/multi-functest/action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ runs:
custom_shell: ${{ inputs.custom_shell }}
cflags: ${{ inputs.cflags }}
cross_prefix: x86_64-unknown-linux-gnu-
exec_wrapper: qemu-x86_64
opt: ${{ inputs.opt }}
func: ${{ inputs.func }}
kat: ${{ inputs.kat }}
Expand All @@ -85,6 +86,7 @@ runs:
custom_shell: ${{ inputs.custom_shell }}
cflags: ${{ inputs.cflags }}
cross_prefix: aarch64-unknown-linux-gnu-
exec_wrapper: qemu-aarch64
opt: ${{ inputs.opt }}
func: ${{ inputs.func }}
kat: ${{ inputs.kat }}
Expand All @@ -101,6 +103,7 @@ runs:
custom_shell: ${{ inputs.custom_shell }}
cflags: "${{ inputs.cflags }} -static"
cross_prefix: aarch64_be-none-linux-gnu-
exec_wrapper: qemu-aarch64_be
opt: ${{ inputs.opt }}
func: ${{ inputs.func }}
kat: ${{ inputs.kat }}
Expand All @@ -117,6 +120,7 @@ runs:
custom_shell: ${{ inputs.custom_shell }}
cflags: ${{ inputs.cflags }}
cross_prefix: riscv64-unknown-linux-gnu-
exec_wrapper: qemu-riscv64
opt: ${{ inputs.opt }}
func: ${{ inputs.func }}
kat: ${{ inputs.kat }}
Expand Down
5 changes: 0 additions & 5 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,11 +74,6 @@ kat: \
$(MLKEM768_DIR)/bin/gen_KAT768 \
$(MLKEM1024_DIR)/bin/gen_KAT1024

# emulate ARM64 binary on x86_64 machine
emulate:
$(Q)$(MAKE) --quiet CROSS_PREFIX=aarch64-none-linux-gnu- $(TARGET)
$(Q)$(QEMU) $(TARGET)

clean:
-$(RM) -rf *.gcno *.gcda *.lcov *.o *.so
-$(RM) -rf $(BUILD_DIR)
1 change: 0 additions & 1 deletion mk/config.mk
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@ CFLAGS := \
# Some Variables #
##################
Q ?= @
QEMU = qemu-aarch64

HOST_PLATFORM := $(shell uname -s)-$(shell uname -m)
# linux x86_64
Expand Down
19 changes: 0 additions & 19 deletions scripts/lib/mlkem_test.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -324,25 +324,6 @@ def __init__(self, opts):
if opts.exec_wrapper:
logging.info(f"Running with customized wrapper {opts.exec_wrapper}")
self.cmd_prefix = self.cmd_prefix + opts.exec_wrapper.split(" ")
elif opts.cross_prefix and platform.system() != "Darwin":
logging.info(f"Emulating with QEMU")
if "x86_64" in opts.cross_prefix:
self.cmd_prefix.append("qemu-x86_64")
elif "aarch64_be" in opts.cross_prefix:
self.cmd_prefix.append("qemu-aarch64_be")
elif "aarch64" in opts.cross_prefix:
self.cmd_prefix.append("qemu-aarch64")
elif "riscv64" in opts.cross_prefix:
self.cmd_prefix.append("qemu-riscv64")
else:
logging.info(
f"Emulation for {opts.cross_prefix} on {platform.system()} not supported",
)
elif opts.cross_prefix:
logging.error(
f"Emulation for {opts.cross_prefix} on {platform.system()} not supported",
)
sys.exit(1)

def _run_func(self, opt):
"""Underlying function for functional test"""
Expand Down

18 comments on commit 13ca898

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arm Cortex-A76 (Raspberry Pi 5) benchmarks

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 29019 cycles 29018 cycles 1.00
ML-KEM-512 encaps 35397 cycles 35398 cycles 1.00
ML-KEM-512 decaps 45884 cycles 45885 cycles 1.00
ML-KEM-768 keypair 49351 cycles 49354 cycles 1.00
ML-KEM-768 encaps 55575 cycles 55573 cycles 1.00
ML-KEM-768 decaps 70334 cycles 70333 cycles 1.00
ML-KEM-1024 keypair 72072 cycles 72070 cycles 1.00
ML-KEM-1024 encaps 80841 cycles 80842 cycles 1.00
ML-KEM-1024 decaps 100675 cycles 100679 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intel Xeon 4th gen (c7i)

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 13527 cycles 13943 cycles 0.97
ML-KEM-512 encaps 17277 cycles 17223 cycles 1.00
ML-KEM-512 decaps 22790 cycles 23029 cycles 0.99
ML-KEM-768 keypair 22504 cycles 22522 cycles 1.00
ML-KEM-768 encaps 24471 cycles 24590 cycles 1.00
ML-KEM-768 decaps 32452 cycles 32646 cycles 0.99
ML-KEM-1024 keypair 31376 cycles 31403 cycles 1.00
ML-KEM-1024 encaps 34960 cycles 34911 cycles 1.00
ML-KEM-1024 decaps 45909 cycles 45844 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMD EPYC 3rd gen (c6a)

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 18126 cycles 18139 cycles 1.00
ML-KEM-512 encaps 23183 cycles 23201 cycles 1.00
ML-KEM-512 decaps 30491 cycles 30493 cycles 1.00
ML-KEM-768 keypair 31063 cycles 31067 cycles 1.00
ML-KEM-768 encaps 34173 cycles 34176 cycles 1.00
ML-KEM-768 decaps 44728 cycles 44728 cycles 1
ML-KEM-1024 keypair 44882 cycles 44590 cycles 1.01
ML-KEM-1024 encaps 50129 cycles 49889 cycles 1.00
ML-KEM-1024 decaps 64397 cycles 64360 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intel Xeon 3rd gen (c6i)

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 20337 cycles 20337 cycles 1
ML-KEM-512 encaps 27002 cycles 26997 cycles 1.00
ML-KEM-512 decaps 35851 cycles 35820 cycles 1.00
ML-KEM-768 keypair 34903 cycles 34875 cycles 1.00
ML-KEM-768 encaps 38208 cycles 38165 cycles 1.00
ML-KEM-768 decaps 50964 cycles 50895 cycles 1.00
ML-KEM-1024 keypair 47945 cycles 48008 cycles 1.00
ML-KEM-1024 encaps 54152 cycles 54211 cycles 1.00
ML-KEM-1024 decaps 71696 cycles 71730 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMD EPYC 4th gen (c7a)

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 15081 cycles 15106 cycles 1.00
ML-KEM-512 encaps 19919 cycles 19685 cycles 1.01
ML-KEM-512 decaps 26318 cycles 26333 cycles 1.00
ML-KEM-768 keypair 25626 cycles 25604 cycles 1.00
ML-KEM-768 encaps 28172 cycles 28163 cycles 1.00
ML-KEM-768 decaps 37875 cycles 37887 cycles 1.00
ML-KEM-1024 keypair 35758 cycles 35660 cycles 1.00
ML-KEM-1024 encaps 41107 cycles 40970 cycles 1.00
ML-KEM-1024 decaps 54508 cycles 54485 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intel Xeon 4th gen (c7i) (no-opt)

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 34892 cycles 34901 cycles 1.00
ML-KEM-512 encaps 45125 cycles 45096 cycles 1.00
ML-KEM-512 decaps 59050 cycles 58976 cycles 1.00
ML-KEM-768 keypair 59155 cycles 59147 cycles 1.00
ML-KEM-768 encaps 71781 cycles 71793 cycles 1.00
ML-KEM-768 decaps 89277 cycles 89280 cycles 1.00
ML-KEM-1024 keypair 87519 cycles 87836 cycles 1.00
ML-KEM-1024 encaps 104619 cycles 104139 cycles 1.00
ML-KEM-1024 decaps 127703 cycles 126880 cycles 1.01

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMD EPYC 3rd gen (c6a) (no-opt)

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 52144 cycles 52126 cycles 1.00
ML-KEM-512 encaps 65749 cycles 65719 cycles 1.00
ML-KEM-512 decaps 88365 cycles 88326 cycles 1.00
ML-KEM-768 keypair 84776 cycles 84812 cycles 1.00
ML-KEM-768 encaps 101446 cycles 101513 cycles 1.00
ML-KEM-768 decaps 132046 cycles 132144 cycles 1.00
ML-KEM-1024 keypair 123981 cycles 124063 cycles 1.00
ML-KEM-1024 encaps 145667 cycles 146002 cycles 1.00
ML-KEM-1024 decaps 183563 cycles 183702 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arm Cortex-A55 (Snapdragon 888) benchmarks

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 58364 cycles 58343 cycles 1.00
ML-KEM-512 encaps 65784 cycles 65760 cycles 1.00
ML-KEM-512 decaps 84548 cycles 84552 cycles 1.00
ML-KEM-768 keypair 98912 cycles 98940 cycles 1.00
ML-KEM-768 encaps 110226 cycles 110166 cycles 1.00
ML-KEM-768 decaps 137095 cycles 137215 cycles 1.00
ML-KEM-1024 keypair 150149 cycles 150195 cycles 1.00
ML-KEM-1024 encaps 166281 cycles 166586 cycles 1.00
ML-KEM-1024 decaps 202259 cycles 202638 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton4

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 18136 cycles 18137 cycles 1.00
ML-KEM-512 encaps 22154 cycles 22155 cycles 1.00
ML-KEM-512 decaps 28791 cycles 28790 cycles 1.00
ML-KEM-768 keypair 30573 cycles 30570 cycles 1.00
ML-KEM-768 encaps 33636 cycles 33633 cycles 1.00
ML-KEM-768 decaps 43174 cycles 43172 cycles 1.00
ML-KEM-1024 keypair 44214 cycles 44213 cycles 1.00
ML-KEM-1024 encaps 49658 cycles 49649 cycles 1.00
ML-KEM-1024 decaps 62658 cycles 62654 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMD EPYC 4th gen (c7a) (no-opt)

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 45735 cycles 45719 cycles 1.00
ML-KEM-512 encaps 56864 cycles 56860 cycles 1.00
ML-KEM-512 decaps 76274 cycles 76229 cycles 1.00
ML-KEM-768 keypair 74510 cycles 74512 cycles 1.00
ML-KEM-768 encaps 88559 cycles 88571 cycles 1.00
ML-KEM-768 decaps 114406 cycles 114397 cycles 1.00
ML-KEM-1024 keypair 109367 cycles 109387 cycles 1.00
ML-KEM-1024 encaps 127417 cycles 127235 cycles 1.00
ML-KEM-1024 decaps 159996 cycles 159962 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton3

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 18968 cycles 18968 cycles 1
ML-KEM-512 encaps 23557 cycles 23557 cycles 1
ML-KEM-512 decaps 30666 cycles 30667 cycles 1.00
ML-KEM-768 keypair 32294 cycles 32295 cycles 1.00
ML-KEM-768 encaps 35884 cycles 35891 cycles 1.00
ML-KEM-768 decaps 46026 cycles 46037 cycles 1.00
ML-KEM-1024 keypair 46628 cycles 46630 cycles 1.00
ML-KEM-1024 encaps 52447 cycles 52446 cycles 1.00
ML-KEM-1024 decaps 66248 cycles 66243 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intel Xeon 3rd gen (c6i) (no-opt)

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 56570 cycles 56612 cycles 1.00
ML-KEM-512 encaps 69407 cycles 69431 cycles 1.00
ML-KEM-512 decaps 91357 cycles 91412 cycles 1.00
ML-KEM-768 keypair 91889 cycles 91848 cycles 1.00
ML-KEM-768 encaps 107840 cycles 107774 cycles 1.00
ML-KEM-768 decaps 136362 cycles 136291 cycles 1.00
ML-KEM-1024 keypair 134781 cycles 134777 cycles 1.00
ML-KEM-1024 encaps 155260 cycles 155233 cycles 1.00
ML-KEM-1024 decaps 191523 cycles 191574 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton2

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 29020 cycles 29057 cycles 1.00
ML-KEM-512 encaps 35402 cycles 35444 cycles 1.00
ML-KEM-512 decaps 45887 cycles 45952 cycles 1.00
ML-KEM-768 keypair 49402 cycles 49446 cycles 1.00
ML-KEM-768 encaps 55644 cycles 55700 cycles 1.00
ML-KEM-768 decaps 70400 cycles 70502 cycles 1.00
ML-KEM-1024 keypair 72012 cycles 72061 cycles 1.00
ML-KEM-1024 encaps 80789 cycles 80857 cycles 1.00
ML-KEM-1024 decaps 100656 cycles 100751 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton4 (no-opt)

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 41980 cycles 41974 cycles 1.00
ML-KEM-512 encaps 50156 cycles 50159 cycles 1.00
ML-KEM-512 decaps 66048 cycles 66048 cycles 1
ML-KEM-768 keypair 69051 cycles 69043 cycles 1.00
ML-KEM-768 encaps 79760 cycles 79762 cycles 1.00
ML-KEM-768 decaps 101011 cycles 101013 cycles 1.00
ML-KEM-1024 keypair 102209 cycles 102455 cycles 1.00
ML-KEM-1024 encaps 117193 cycles 117435 cycles 1.00
ML-KEM-1024 decaps 143703 cycles 143371 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton3 (no-opt)

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 45392 cycles 45390 cycles 1.00
ML-KEM-512 encaps 54217 cycles 54217 cycles 1
ML-KEM-512 decaps 71150 cycles 71153 cycles 1.00
ML-KEM-768 keypair 74823 cycles 74832 cycles 1.00
ML-KEM-768 encaps 86059 cycles 86071 cycles 1.00
ML-KEM-768 decaps 108802 cycles 108818 cycles 1.00
ML-KEM-1024 keypair 111129 cycles 111125 cycles 1.00
ML-KEM-1024 encaps 125930 cycles 125929 cycles 1.00
ML-KEM-1024 decaps 154627 cycles 154629 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graviton2 (no-opt)

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 71253 cycles 71185 cycles 1.00
ML-KEM-512 encaps 85054 cycles 85068 cycles 1.00
ML-KEM-512 decaps 112737 cycles 112757 cycles 1.00
ML-KEM-768 keypair 117664 cycles 117228 cycles 1.00
ML-KEM-768 encaps 135324 cycles 135210 cycles 1.00
ML-KEM-768 decaps 171956 cycles 171943 cycles 1.00
ML-KEM-1024 keypair 174763 cycles 175127 cycles 1.00
ML-KEM-1024 encaps 196905 cycles 197155 cycles 1.00
ML-KEM-1024 decaps 243058 cycles 243401 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bananapi bpi-f3 benchmarks

Benchmark suite Current: 13ca898 Previous: 5b4572e Ratio
ML-KEM-512 keypair 335164 cycles 335210 cycles 1.00
ML-KEM-512 encaps 445882 cycles 446012 cycles 1.00
ML-KEM-512 decaps 593973 cycles 594119 cycles 1.00
ML-KEM-768 keypair 556332 cycles 556367 cycles 1.00
ML-KEM-768 encaps 698191 cycles 698263 cycles 1.00
ML-KEM-768 decaps 889951 cycles 890040 cycles 1.00
ML-KEM-1024 keypair 821702 cycles 822140 cycles 1.00
ML-KEM-1024 encaps 998669 cycles 999635 cycles 1.00
ML-KEM-1024 decaps 1230945 cycles 1230548 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@oqs-bot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arm Cortex-A72 (Raspberry Pi 4) benchmarks

Benchmark suite Current: 13ca898 Previous: ceb3e2d Ratio
ML-KEM-512 keypair 51885 cycles 51689 cycles 1.00
ML-KEM-512 encaps 58387 cycles 58815 cycles 0.99
ML-KEM-512 decaps 74346 cycles 74660 cycles 1.00
ML-KEM-768 keypair 87995 cycles 88023 cycles 1.00
ML-KEM-768 encaps 96550 cycles 96055 cycles 1.01
ML-KEM-768 decaps 120075 cycles 120005 cycles 1.00
ML-KEM-1024 keypair 131672 cycles 131141 cycles 1.00
ML-KEM-1024 encaps 144604 cycles 144371 cycles 1.00
ML-KEM-1024 decaps 175365 cycles 175853 cycles 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Please sign in to comment.