Compilation fails with -O3 and GCC >= 11

[oynqr]

Compiling with -O3 on GCC tries to inline these function calls:

libxcrypt/lib/alg-sha1.c

Line 265 in 72f75aa

sha1_process_bytes ((const uint8_t *)"\200", ctx, 1);

libxcrypt/lib/alg-sha1.c

Line 267 in 72f75aa

sha1_process_bytes ((const uint8_t *)"\0", ctx, 1);

This triggers an overflow warning on this line, which fails the build due to -Werror:

libxcrypt/lib/alg-sha1.c

Line 244 in 72f75aa

sha1_do_transform (ctx->state, (const uint8_t*)buffer + i);

Looking at the code a bit, I don't think this line is actually reachable in this case, so it's probably a false positive.

[zackw]

At present this is a "don't do that then" situation. There are several places in libxcrypt where the code technically has undefined behavior but the compiler will generate correct machine code as long as the inlining level is not too high. You have found one of them. This is one of the reasons why the documentation says not to use -flto, and I guess we should add -O3 to that as well.

It is a bug and we will fix it eventually, but we have very limited time to do any work on libxcrypt presently, and this class of issues is considered low priority, especially the subset that affects obsolete hashing methods only.

I would be happy to review a patch that restructured alg-sha1.c so that it compiled cleanly with -O3, especially if you also adjust things so that when compiled with -O2 -S -fsanitize=undefined -fno-sanitize-recover=all the generated assembly code contains no calls to any function whose name begins with __ubsan (this is an incomplete but practically useful approximation to "this source code is well-defined according to the C standard").