From 556c518af79f4d324e14013bb9f5004e76caf086 Mon Sep 17 00:00:00 2001 From: Samir Droubi Date: Tue, 28 May 2024 13:32:06 -0400 Subject: [PATCH 1/4] Add more instructions about testing --- README.md | 10 ++---- TESTING.md | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 8 deletions(-) create mode 100644 TESTING.md diff --git a/README.md b/README.md index 3a0ecb1a..ec98dafb 100644 --- a/README.md +++ b/README.md @@ -57,13 +57,7 @@ For example, to use `apple-silicon`, you would run: ``` cmake --preset apple-silicon cmake --build build/apple-silicon -ctest --test-dir build/apple-silicon ``` -## Troubleshooting - -If there is an error on apple-silicon, try running - -``` -export SDKROOT=$(xcrun --sdk macosx --show-sdk-path) -``` +## Testing ExoBLAS +For more detailed on building and testing ExoBLAS, please read [here](TESTING.md). diff --git a/TESTING.md b/TESTING.md new file mode 100644 index 00000000..91d95d70 --- /dev/null +++ b/TESTING.md @@ -0,0 +1,93 @@ +# Testing ExoBLAS + +## Building ExoBLAS + +We use CMake and its presets feature to ease configuration. There +are currently three presets, but `cmake --list-presets` will always +show the current list: + +``` +$ cmake --list-presets +Available configure presets: + + "apple-silicon" - Apple M1 or M2 mac + "linux-arm64" - Linux AArch64 + "avx2" - Intel AVX2 + "avx512" - Intel AVX512 +``` + +For example, to use `avx2`, you can setup a build directory using: + +``` +cmake --preset avx2 +``` +You can build the whole library with tests using: +``` +cmake --build build/avx2 +``` + +If you want to compile an individual target you can run: +``` +cmake --build build/avx2 exo_[KERNEL] # to build the binary for this kernel +cmake --build build/avx2 [KERNEL]_correctness # to build the correctness test for this kernel +cmake --build build/avx2 [KERNEL]_bench # to build the bench test for this kernel +``` +If you want to benchmark against a new library, you need to run: +``` +cmake --preset avx2 -DBLA_VENDOR=[OTHER_LIB] +``` +Then, recompile the benchmark test: +``` +cmake --build build/avx2 [KERNEL]_bench +``` + +## Running the correctness tests +To run the correctness test for all kernels: +``` +ctest --test-dir ./build/avx2 -V -R correctness +``` +To run the correctness for an individual kernel: +``` +ctest --test-dir ./build/avx2 -V -R [KERNEL]_correctness +``` + +## Running the benchmark tests +To run the benchmark test for all kernels: +``` +ctest --test-dir ./build/avx2 -V -R bench +``` +To run the benchmark for an individual kernel: +``` +ctest --test-dir ./build/avx2 -V -R [KERNEL]_bench +``` +To run the benchmark for ExoBLAS only: +``` +ctest --test-dir ./build/avx2 -V -R exo_[KERNEL]_bench # Run ExoBLAS benchmark for [KERNEL] +ctest --test-dir ./build/avx2 -V -R exo_ # Run ExoBLAS benchmark for all kernels +``` +To run the benchmark for the reference BLAS library only: +``` +ctest --test-dir ./build/avx2 -V -R cblas_[KERNEL]_bench # Run reference benchmark for [KERNEL] +ctest --test-dir ./build/avx2 -V -R cblas_ # Run the reference benchmark for all kernels +``` +Note that benchmark results are cached, so you can run the reference benchmarks once when you start development. Remember to rerun them if you change the inputs being passed by the benchmark. + +## Running the graphing script +To plot one kernel results, you can run: +``` +python analytics_tools/graphing/graph.py [KERNEL] +``` +To plot summary results for all kernels, you can run: +``` +python analytics_tools/graphing/graph.py all +``` +You will be able to find the results in: +``` +python analytics_tools/graphing/graphs +``` + +## Lines of code summary +To get a summary of the lines of code in `src/`, you can run: +``` +python analytics_tools/loc/count_loc.py +``` From bb55c05aff5a92e3b00992dd9af0bdef1901356a Mon Sep 17 00:00:00 2001 From: Samir Droubi Date: Tue, 28 May 2024 14:13:33 -0400 Subject: [PATCH 2/4] Add docs directory --- README.md | 5 ++++- TESTING.md => docs/TESTING.md | 0 docs/TROUBLESHOOTING.md | 7 +++++++ 3 files changed, 11 insertions(+), 1 deletion(-) rename TESTING.md => docs/TESTING.md (100%) create mode 100644 docs/TROUBLESHOOTING.md diff --git a/README.md b/README.md index ec98dafb..ab5bb218 100644 --- a/README.md +++ b/README.md @@ -60,4 +60,7 @@ cmake --build build/apple-silicon ``` ## Testing ExoBLAS -For more detailed on building and testing ExoBLAS, please read [here](TESTING.md). +For more detailed on building and testing ExoBLAS, please read [here](docs/TESTING.md). + +## Troubleshooting +Some troubleshooting instructions can be found [here](docs/TROUBLESHOOTING.md). diff --git a/TESTING.md b/docs/TESTING.md similarity index 100% rename from TESTING.md rename to docs/TESTING.md diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md new file mode 100644 index 00000000..2b9c3593 --- /dev/null +++ b/docs/TROUBLESHOOTING.md @@ -0,0 +1,7 @@ +# Troubleshooting + +If there is an error on apple-silicon, try running + +``` +export SDKROOT=$(xcrun --sdk macosx --show-sdk-path) +``` From 6c103d6e786c63da152b879b2a12614714aefd51 Mon Sep 17 00:00:00 2001 From: Samir Droubi Date: Tue, 28 May 2024 14:25:44 -0400 Subject: [PATCH 3/4] Fix --- TESTING.md | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 TESTING.md diff --git a/TESTING.md b/TESTING.md new file mode 100644 index 00000000..70511d9a --- /dev/null +++ b/TESTING.md @@ -0,0 +1,93 @@ +# Testing ExoBLAS + +## Building ExoBLAS + +We use CMake and its presets feature to ease configuration. There +are currently three presets, but `cmake --list-presets` will always +show the current list: + +``` +$ cmake --list-presets +Available configure presets: + + "apple-silicon" - Apple M1 or M2 mac + "linux-arm64" - Linux AArch64 + "avx2" - Intel AVX2 + "avx512" - Intel AVX512 +``` + +For example, to use `avx2`, you can setup a build directory using: + +``` +cmake --preset avx2 +``` +You can build the whole library with tests using: +``` +cmake --build build/avx2 +``` + +If you want to compile an individual target you can run: +``` +cmake --build build/avx2 --target exo_[KERNEL] # to build the binary for this kernel +cmake --build build/avx2 --target [KERNEL]_correctness # to build the correctness test for this kernel +cmake --build build/avx2 --target [KERNEL]_bench # to build the bench test for this kernel +``` +If you want to benchmark against a new library, you need to run: +``` +cmake --preset avx2 -DBLA_VENDOR=[OTHER_LIB] +``` +Then, recompile the benchmark test: +``` +cmake --build build/avx2 --target [KERNEL]_bench +``` + +## Running the correctness tests +To run the correctness test for all kernels: +``` +ctest --test-dir ./build/avx2 -V -R correctness +``` +To run the correctness for an individual kernel: +``` +ctest --test-dir ./build/avx2 -V -R [KERNEL]_correctness +``` + +## Running the benchmark tests +To run the benchmark test for all kernels: +``` +ctest --test-dir ./build/avx2 -V -R bench +``` +To run the benchmark for an individual kernel: +``` +ctest --test-dir ./build/avx2 -V -R [KERNEL]_bench +``` +To run the benchmark for ExoBLAS only: +``` +ctest --test-dir ./build/avx2 -V -R exo_[KERNEL]_bench # Run ExoBLAS benchmark for [KERNEL] +ctest --test-dir ./build/avx2 -V -R exo_ # Run ExoBLAS benchmark for all kernels +``` +To run the benchmark for the reference BLAS library only: +``` +ctest --test-dir ./build/avx2 -V -R cblas_[KERNEL]_bench # Run reference benchmark for [KERNEL] +ctest --test-dir ./build/avx2 -V -R cblas_ # Run the reference benchmark for all kernels +``` +Note that benchmark results are cached, so you can run the reference benchmarks once when you start development. Remember to rerun them if you change the inputs being passed by the benchmark. + +## Running the graphing script +To plot one kernel results, you can run: +``` +python analytics_tools/graphing/graph.py [KERNEL] +``` +To plot summary results for all kernels, you can run: +``` +python analytics_tools/graphing/graph.py all +``` +You will be able to find the results in: +``` +python analytics_tools/graphing/graphs +``` + +## Lines of code summary +To get a summary of the lines of code in `src/`, you can run: +``` +python analytics_tools/loc/count_loc.py +``` From b64ba93011f8a6ea9ebd8b8fd415b4579ec75496 Mon Sep 17 00:00:00 2001 From: Samir Droubi Date: Tue, 28 May 2024 14:56:43 -0400 Subject: [PATCH 4/4] Add codegen testing instructions --- TESTING.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/TESTING.md b/TESTING.md index 70511d9a..ee9e92db 100644 --- a/TESTING.md +++ b/TESTING.md @@ -72,6 +72,36 @@ ctest --test-dir ./build/avx2 -V -R cblas_ # Run the reference benchmark for all ``` Note that benchmark results are cached, so you can run the reference benchmarks once when you start development. Remember to rerun them if you change the inputs being passed by the benchmark. +## Checking codegen consistency +It is valuable when doing development to be able to compare against the code-generation at the commit before you started you made any changes. + +Before you start development, make sure to build the entire library using: +``` +cmake --build build/avx2 +``` +Then, run the following command to generate a cache of the generated C code: +``` +ctest --test-dir ./build/avx2 -V -R update_ +``` +This will create a local untracked copy of all the generated C sources in `test/codegen/reference/sources/`. + +As you change the scheduling code or the library code, you can run the following command to check that the codegen is still the same (after rebuilding the library): +``` +ctest --test-dir ./build/avx2 -V -R codegen_nonverbose +``` +This will fail if the new files are different from the cached source and print the diff. + +If you want to check an individual kernel's codegen, you can run: +``` +ctest --test-dir ./build/avx2 -V -R [KERNEL]_codegen_nonverbose +``` + +If a kernel's codegen has changed, but this is expected. You can update its codegen again by running: +``` +ctest --test-dir ./build/avx2 -V -R [KERNEL]_update_reference +``` +This will update the cached sources, and the hash of the sources in the json stored in `test/codegen/reference/sha256/`. The hashes jsons are tracked files and they are what we use to verify the codegen output. The cached sources are only relevant so that we can show the diff when the hash is inconsistent. + ## Running the graphing script To plot one kernel results, you can run: ```