Speed up identifier detection

[jserv]

Both 'is_ident1' and 'is_ident2' are now macros instead of function calls, and they are tweaked for ASCII detection in advance with the fallback to table lookup for non-ASCII characters.

[fuhsnn]

Thank you for the patches. Honestly performance hasn't been a priority, I consider the lexer less significant in the total pipeline compared to cache-misses from leaking and the overhead of external assembler. I do have some ideas as well, so a branch for performance experiments should make sense.

In terms of performance I want to ensure a common basis to work on: which projects to benchmark and what kind of environment, is it self-hosted, another simple C compiler or full-on gcc/clang optimizations with custom allocator etc. Do you have a preference?

[fuhsnn]

On the pull request: I think the short-circuiting of ASCII characters can be done earlier, skipping decode_utf8() altogether.

[jserv]

In terms of performance I want to ensure a common basis to work on: which projects to benchmark and what kind of environment, is it self-hosted, another simple C compiler or full-on gcc/clang optimizations with custom allocator etc.

Using uftrace, I identified the performance bottlenecks in the compilation process.

Here's what I found:
First, apply the following changes:

--- a/GNUmakefile
+++ b/GNUmakefile
@@ -1,4 +1,4 @@
-CFLAGS=-std=c99 -g -fno-common -Wall -pedantic -Wno-switch
+CFLAGS=-std=c99 -g -pg -fno-common -Wall -pedantic -Wno-switch
 
 SRCS=$(wildcard *.c)
 OBJS=$(SRCS:.c=.o)

After running uftrace ./slimcc -c -o stage2/parse.o parse.c, the trace showed the following call stack:

Copy[6] is_ident2
[5] read_ident
[4] tokenize
[3] tokenize_file
[2] must_tokenize_file
[1] cc1
[0] main

Based on this analysis, I tweaked the is_ident1 and is_ident2 functions.

[jserv]

By the way, I am working with my students to enhance the shecc project, a self-compiling C compiler with some optimizations. Since shecc only implements a subset of C, we are planning to modify slimcc to output compatible C code that shecc can process, rather than having slimcc generate x86-64 assembly directly. This way, slimcc would serve as a full C language frontend for shecc, while allowing shecc to focus on optimization and backend support. To achieve this, we plan to contribute improvements to slimcc's parsing, tokenization, and preprocessing components.

[fuhsnn]

I'm thrilled to know that the project has some real world value and am happy to target different backends as long as the IR is stable, but I imagine memory usage will be a major pain point on embedded targets that shecc currently self-hosts on. I thought about re-architecting, but then the code wouldn't be as accessible and one might as well work on other more optimized C frontend like tcc, cproc, tyfkda/xcc, libfirm/cparser etc.

If C++ is allowed in your classroom, I do have an unfinished C++ port sitting around that use RAII for nearly identical code with much better memory consumption, a minimal C++ front-end bootstrappable with C99 is also something I've always wanted to work on.

[fuhsnn]

The C++ version is uploaded in partial_c++_port branch, it's less C++ than I remembered and still able to build as C with just a little macro and typedef's.

If that particular style is okay to you, I'd prefer basing performance optimizations on that version, since it feels a bit bike shedding to optimize parsing while compiling larger projects hit OOM on an embedded device.

[jserv]

I defer this pull request to @ChAoSUnItY, who is assigned to proceed with the above research prototype under my supervision.

[fuhsnn]

I believe identifier detection is of decent speed after efcbb85, feel free to open new issues regarding the C backend.

Btw thanks for mentioning uftrace, logging exact call counts is incredible. Can't believe it's not more well known.

[ChAoSUnItY]

I've adopted identifier reading logic from efcbb85 and further refine it with the macro functions introduced previously in this PR, and also applied fool-proofing logic to the function. This way, it improves readability.

[jserv]

Let's rebase the latest main branch and squash the commits.