[RFC][PoC] Zephyr PreProcessor (ZPP) framework

[pdgendt]

Introduction

This is a proof-of-concept, namely the Zephyr Pre-Processor (ZPP) framework, an alternative to the syscall parsing for gathering function and struct information.

Problem description

The https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/build/parse_syscalls.py script has a few drawbacks that this PR tries to improve upon.

• It's currently (ab)used to find __subsystem and __net_socket "tags" but this is a hidden implementation detail
• It operates on header/source files before the C pre-processer so we can't rely on macro expansion
• It generates more boiler plate than is needed

Proposed change

Run the C pre-processor, in the pre-build stage, on files that are part of the build and have the -DZPP compile definition (opt-in), and parse the output for annotated functions or structs.
Gather the information in a formatted (json) file that can be fed into other build steps.

Detailed RFC

In the aforementioned C pre-processor run, the compile definition -D__ZPP__ is added which enables the C23 annotations (not part of the regular build), for example:

#ifdef __ZPP__
#define __zpp(x) [[zephyr::x]]
#else
#define __zpp(...)
#endif

#define __syscall __zpp(func("syscall", __FILE__, __LINE__))
#define __subsystem __zpp(struct("__subsystem"))

The newly introduced scripts/zpp.py scans for these annotations and can output the structured data needed for the next build steps.

[
  {
    "attr": "func",
    "args": [
      "syscall",
      "WEST_TOPDIR/zephyr/include/zephyr/kernel.h",
      537
    ],
    "data": {
      "type": "int",
      "name": "k_thread_join",
      "args": "struct k_thread *thread, k_timeout_t timeout"
    }
  },
  {
    "attr": "struct",
    "args": [
      "__subsystem"
    ],
    "data": {
      "name": "gpio_driver_api"
    }
  }
]

Proposed change (Detailed)

• Convert the __subsystem tags with ZPP (part of the PR already)
• Convert the __net_socket tags with ZPP
• Convert the __syscall tags with ZPP
• Introduce __section_iterable tags for structs to allow creating linker sections from source files, without the need for additional linker scripts (cleanup all common-rom.ld and common-rom.cmake files)

Dependencies

Affects downstream usage of the syscall mechanism.

Concerns and Unresolved Questions

• Portability for different toolchains
• Opt-in vs opt-out
• Generated headers need include guards when running ZPP (part of the PR already)

Alternatives

Don't do this