Skip to content

Latest commit

 

History

History
189 lines (145 loc) · 9.03 KB

FUZZING.md

File metadata and controls

189 lines (145 loc) · 9.03 KB
Raw

Fuzz testing

Modern fuzz testers are very effective and we wish to use them to ensure that no silly bugs creep into BoringSSL.

We use Clang's libFuzzer for fuzz testing and there are a number of fuzz testing functions in fuzz/. They are not built by default because they require that the rest of BoringSSL be built with some changes that make fuzzing much more effective, but are completely unsafe for real use.

In order to build the fuzz tests you will need at least Clang 6.0. Pass -DFUZZ=1 on the CMake command line to enable building BoringSSL with coverage and AddressSanitizer, and to build the fuzz test binaries. You'll probably need to set the CC and CXX environment variables too, like this:

mkdir build
cd build
CC=clang CXX=clang++ cmake -GNinja -DFUZZ=1 ..
ninja

From the build/ directory, you can then run the fuzzers. For example:

./fuzz/cert -max_len=10000 -jobs=32 -workers=32 ../fuzz/cert_corpus/

The arguments to jobs and workers should be the number of cores that you wish to dedicate to fuzzing. By default, libFuzzer uses the largest test in the corpus (or 64 if empty) as the maximum test case length. The max_len argument overrides this.

The recommended values of max_len for each test are:

Test max_len value
bn_div 384
bn_mod_exp 4096
cert 10000
client 20000
pkcs8 2048
privkey 2048
server 4096
session 8192
spki 1024
read_pem 512
ssl_ctx_api 256

These were determined by rounding up the length of the largest case in the corpus.

There are directories in fuzz/ for each of the fuzzing tests which contain seed files for fuzzing. Some of the seed files were generated manually but many of them are “interesting” results generated by the fuzzing itself. (Where “interesting” means that it triggered a previously unknown path in the code.)

Minimising the corpora

When a large number of new seeds are available, it's a good idea to minimise the corpus so that different seeds that trigger the same code paths can be deduplicated.

In order to minimise all the corpora, build for fuzzing and run ./fuzz/minimise_corpora.sh. Note that minimisation is, oddly, often not idempotent for unknown reasons.

Fuzzer mode

When -DFUZZ=1 is passed into CMake, BoringSSL builds with BORINGSSL_UNSAFE_FUZZER_MODE and BORINGSSL_UNSAFE_DETERMINISTIC_MODE defined. This modifies the library to be more friendly to fuzzers. If BORINGSSL_UNSAFE_DETERMINISTIC_MODE is set, BoringSSL will:

  • Replace RAND_bytes with a deterministic PRNG. Call RAND_reset_for_fuzzing() at the start of fuzzers which use RAND_bytes to reset the PRNG state.

  • Use a hard-coded time instead of the actual time.

Additionally, if BORINGSSL_UNSAFE_FUZZER_MODE is set, BoringSSL will:

  • Modify the TLS stack to perform all signature checks (CertificateVerify and ServerKeyExchange) and the Finished check, but always act as if the check succeeded.

  • Treat every cipher as the NULL cipher.

  • Tickets are unencrypted and the MAC check is performed but ignored.

  • renegotiation_info checks are ignored.

This is to prevent the fuzzer from getting stuck at a cryptographic invariant in the protocol.

TLS transcripts

The client and server corpora are seeded from the test suite. The test suite has a -fuzzer flag which mirrors the fuzzer mode changes above and a -deterministic flag which removes all non-determinism on the Go side. Not all tests pass, so ssl/test/runner/fuzzer_mode.json contains the necessary suppressions. The run_tests target will pass appropriate command-line flags.

There are separate corpora, client_corpus_no_fuzzer_mode and server_corpus_no_fuzzer_mode. These are transcripts for fuzzers with only BORINGSSL_UNSAFE_DETERMINISTIC_MODE defined. To build in this mode, pass -DNO_FUZZER_MODE=1 into CMake. This configuration is run in the same way but without -fuzzer and -shim-config flags.

If both sets of tests pass, refresh the fuzzer corpora with refresh_ssl_corpora.sh:

cd fuzz
./refresh_ssl_corpora.sh /path/to/fuzzer/mode/build /path/to/non/fuzzer/mode/build

Adding New Fuzz Test Targets

When adding new functionality, adding new fuzz tests are important to provide additional testing and verification that we are correct.

Steps of generating Fuzz corpus

  1. NEW_FUNCTION='name_of_new_fuzzing_target'
  2. touch fuzz/${NEW_FUNCTION}.cc
    Write fuzzing code for libfuzzer to parse in fuzz/${NEW_FUNCTION}.cc. The code in this file will be how Libfuzzer identifies how to parse any data inputs the fuzzer gives. Fuzz tests test upon random data inputs, so in most cases we write code to test on data parsing (and reserializing the data again if applicable).
  3. mkdir fuzz/${NEW_FUNCTION}_corpus_raw
    Import any existing unit test files into the fuzz/${NEW_FUNCTION}_corpus_raw directory. These will be used to create our new fuzzing corpus.
  4. Add fuzzer('name_of_new_fuzzing_target') target in the fuzz/CmakeLists.txt file.
  5. Build AWS-LC with fuzzing enabled (only buildable with clang) 
    mkdir test_build_dir
cmake ../aws-lc -GNinja "-B`pwd`/test_build_dir" "-DCMAKE_INSTALL_PREFIX=`pwd`/test_build_dir/" \
-DCMAKE_BUILD_TYPE=RelWithDebInfo -DFUZZ=1 -DASAN=1 -DBUILD_TESTING=OFF
ninja -C test_build_dir
  6. Run the fuzzer with our import test files by running:
    Note:
    • max_len: set for maximum test file size if needed
    • max_total_time: can be adjusted to how long you'd like the fuzzer to run.
    • workers: should be set accordingly to how many cpu processors are available.
    mkdir fuzz/${NEW_FUNCTION}_corpus_temp
NUM_CPU_THREADS=$(grep -c ^processor /proc/cpuinfo)
./test_build_dir/fuzz/${NEW_FUNCTION} -jobs=${NUM_CPU_THREADS} -workers=${NUM_CPU_THREADS} -timeout=5 \
-print_final_stats=1 -max_total_time=1800 fuzz/${NEW_FUNCTION}_corpus_temp fuzz/${NEW_FUNCTION}_corpus_raw
  7. By default, the fuzzing process will continue until max_total_time has been reached. When a bug is found, any crashes or sanitizer failures will be reported in the console and the particular corpus file that triggered the bug will be written into the root directory (as crash-<sha1>, leak-<sha1>, or timeout-<sha1>). If any of these files appear, we should look into what is causing the failure.
  8. If no failures emerge, we can merge the generated corpus to minimize the amount of files, while still providing full coverage. 
    mkdir fuzz/${NEW_FUNCTION}_corpus
./test_build_dir/fuzz/${NEW_FUNCTION} -merge=1 fuzz/${NEW_FUNCTION}_corpus fuzz/${NEW_FUNCTION}_corpus_temp
  9. Remove original directories, files, and fuzz logs used to generate the orginal corpus. 
    rmdir fuzz/${NEW_FUNCTION}_corpus_temp
rmdir fuzz/${NEW_FUNCTION}_corpus_raw
rm -r fuzz-*.log

Steps of verifying the generated Fuzz corpus.

  1. Verify if the generated corpus has good coverage.
  • 1.1. Add temporary patch.

For example, below patch adds some print statements to check if the fuzz inputs check the code blocks.

$ git diff fuzz/ssl_serialization.cc
diff --git a/fuzz/ssl_serialization.cc b/fuzz/ssl_serialization.cc
index 9b4bc804d..c8b3208d8 100644
--- a/fuzz/ssl_serialization.cc
+++ b/fuzz/ssl_serialization.cc
@@ -24,6 +24,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t *buf, size_t len) {

   // If the format was invalid, just return.
   if (!ssl) {
+    fprintf(stderr, "SSL Serialization fuzz executed code block 1.\n");
     return 0;
   }

@@ -31,6 +32,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t *buf, size_t len) {
   size_t encoded_len;
   uint8_t *encoded;
   if (!SSL_to_bytes(ssl.get(), &encoded, &encoded_len)) {
+    fprintf(stderr, "SSL Serialization fuzz executed code block 2.\n");
     uint32_t e = ERR_get_error();
     if (e == 0) {
       fprintf(stderr, "In Fuzz, SSL_to_bytes failed without giving a error code.\n");
@@ -46,6 +48,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t *buf, size_t len) {
     return 1;
   }

+  fprintf(stderr, "SSL Serialization fuzz executed code block 3.\n");
   OPENSSL_free(encoded);
   return 0;
 }
  • 1.2. Run fuzz script.
rm -rf nohup.out && bash -c "nohup sh -c './tests/ci/run_fuzz_tests.sh' &" && sleep 2 && chmod a+rw nohup.out
  • 1.3. Check the fuzz log.

All the injected statements should be printed. That means the fuzz inputs cover the code blocks we expect.

$ cat nohup.out | grep "SSL Serialization fuzz executed code block 1" | wc -l
13545099
$ cat nohup.out | grep "SSL Serialization fuzz executed code block 2" | wc -l
12303
$ cat nohup.out | grep "SSL Serialization fuzz executed code block 3" | wc -l
68918

Fuzz corpus added by aws-lc

  1. ssl_serialization: detailed steps are in CryptoAlg-850?selectedConversation=4ec5f34a-451e-4ae0-ae4c-0322b9f22108.