Implementing multi-prog attach support

[saxena-anurag]

This is in context of adding a generic support to attach multiple programs to an attach point (hook) for any program type, a subset of which is currently tracked by #2626.

Looking at the current available solutions on Linux, it looks like there are 2 possible designs we can use:

OPTION 1: XDP dispatcher approach
In Linux, libxdp uses XDP dispatcher program to "chain" multiple programs to a hook. Salient features:

1. A program defines a priority and chain call action. Lower number means higher priority. Chain call actions are return codes that the program indicates for packets that should continue processing.
2. This feature is specific to XDP only and uses new (libxdp) UM APIs.
3. Priority / chain call actions can be passed either by UM APIs or embedded in BTF data.
4. Default behavior defaults to default priority and default chain call actions for XDP.
5. Programs with same priority are chained as FIFO.

Windows Implementation:

1. Add new eBPF API to also take priority and chain call actions as input (chain call actions will be uint32, opaque to ebpf runtime)
2. Add support to parse the new BTF data, and embed in native module.
3. eBPF runtime will pass the priority and chain call actions to the extension. Extension has to implement chaining / dispatcher functionality.

References:

1. https://github.com/xdp-project/xdp-tools/blob/master/lib/libxdp/README.org
2. https://www.youtube.com/watch?v=Fu4L8ewcO70
3. https://lpc.events/event/7/contributions/671/attachments/561/992/xdp-multiprog.pdf

OPTION 2: Generic attach / detach APIs
This is the second and newer approach in Linux. Salient features:

1. New flags have been added which can be used with BPF syscalls and libbpf APIs (e.g. bpf_prog_attach()).
2. Flags include:
   1. BPF_F_BEFORE
   2. BPF_F_AFTER
   3. BPF_F_ID
   4. BPF_F_LINK
3. There is no priority for each attach. Each program can specify whether it wants to be before or after another program fd / id, or if it wants to be first or last globally.
4. The implementation "adds a generic layer called bpf_mprog which can be reused by different attachment layers to enable multi-program attachment and dependency resolution".

Windows implementation:

1. Add support for the new flags for existing libbpf APIs, and support for "default" behavior for backward compatibility.
2. As with Linux, prefereably keep the "chaining" logic in ebpf runtime so that each extension does not need to re-implement the same thing. This may require some design as to how will extensions invoke the programs when there are more than 1 programs attached, and do extensions get to make any decision after every program invocation to decide whether to continue program invocation.

References:

1. https://www.youtube.com/watch?v=EM09k066pB8
2. libbpf/libbpf@d7e583a

Question: Which design approach should we take?

Interim solution proposal: Since we urgently need support for multi-prog attach for at least SOCK_ADDR hook, while we decide on the approach and design, we can implement this only for SOCK_ADDR in netebbpfext with FIFO logic.

[saxena-anurag]

Design Proposal:

We implement option 2. ebpfcore translates the UM flags to a 64-bit priority number. Since new programs can be inserted in the middle of the "chain", runtime may need to sometimes reallocate priorities. As an alternative, we can restrict the max chain count to say, 32. Then by using 64-bit numbers for priority, runtime does not need to recompute / update priority for any of the attached programs.

eBPF runtime will pass this computed priority to the extensions while attaching the programs. Extensions can then implement their logic of invoking all the programs. Examples:

1. All netebpfext hooks use WFP filters. netebpfext can configure attach priority as filter priority. This offloads chaining logic to WFP, which will now invoke the callout corresponding to each program in the order of priority.
2. For XDP, all the attached eBPF programs can be added in a sorted list (not a hot path). When a packet arrives on an interface, all the programs in the sorted list can be invoked one by one.

[mtfriesen]

eBPF runtime will pass this computed priority to the extensions while attaching the programs. Extensions can then implement their logic of invoking all the programs

What is the reasoning behind requiring each extension to handle the complexity of multiple programs, instead of the Linux approach of containing that complexity within the common ebpf runtime?

[saxena-anurag]

I think the extensions need to be aware of multiple programs attached to the hook. This is also needed for the extension to decide after every program invocation whether to continue invoking further programs in the chain, or to terminate. This logic will be extension dependent and will be opaque to ebpf runtime.

Also, each extension may require a different way to handle multiple programs attached to the "hook". For example, for SOCK_ADDR, this can be handled using WFP filter priority, vs. XDP where I think a sorted list of programs is sufficient.

If we keep the logic in ebpfcore, then I think there are 2 possible options:

1. ebpfcore "hides" multiple attaches from the extension -- which means ebpfcore exposes a single "fake" / "placeholder" attach to extension, and still somehow notifies that there is actually a chain of programs attached. This approach I think will be messy and complicated.
2. ebpfcore exposes multiple attaches to the extension -- if this is the case, then it is better to also pass the attach priority extension and let extensions choose how they want to implement the ordering.

This is still a proposal, and we can absolutely discuss if there are better ways to implement this.

[mtfriesen]

Oh, perhaps I misunderstood the text in the original proposal:

As with Linux, prefereably keep the "chaining" logic in ebpf runtime so that each extension does not need to re-implement the same thing.

On Linux, is the ebpf runtime exposing multiple programs to the hooks themselves? So the Linux tc hook might treat ACT_SHOT from the first program as a terminating action and skip the rest of the programs?

[saxena-anurag]

Looking at the presentation, it looks like Linux kernel iterates over all the programs that are attached, and in some cases does terminate in the middle of the loop.

There are some details about bpf_mprog framework also in the presentation and in the PR / commit description. We can do something similar to bpf_mprog which maintains a sorted list of all the programs (but will need to decide who maintains this list -- runtime or extensions).

[shpalani]

new eBPF API to also take priority and chain call actions

How does the extension know (or notify at compile time) if there is any conflict (same number used by two different programs) in the priority of the ebpf program specified?

[saxena-anurag]

Priority is needed at runtime when a program is being attached. In case there is a priority conflict, IIRC the approach is to invoke the programs in the order they were attached.

[saxena-anurag]

Option 2 above supports multiple flags, like:

BPF_F_{BEFORE | AFTER}
BPF_F_FIRST
BPF_F_LAST

ID or FD -- controlled by BPF_F_ID
program or link -- controlled by BPF_F_LINK

Multiple combinations of the above flags are possible.
Example:

1. BPF_F_BEFORE | BPF_F_ID --> attach before the provided ID.
2. BPF_F_FIRST | BPF_F_LAST --> only a single program can be attached at the hook.

Questions:

1. If an app uses the default APIs, then the behavior is to "append" the program in the list of programs attached to the hook. Typically, for unrelated products attaching programs to the same hook will use this behavior. In such a case, the order of invocation depends on the order of attach and is not deterministic. Is this desirable?
2. If we implement the above logic of relative ordering, should these flags and program and link IDs be exposed to each extension or should be contained within ebpfcore instead of each extension to implement this ordering logic.
3. In libbpf, till now, this is only supported for TCX and netkit, hence libbpf APIs specific to only these 2 program types have been added / updated. How do we generalize this? specifically for SOCK_ADDR, as we want to first implement it for SOCK_ADDR.

[saxena-anurag]

libbpf: https://github.com/libbpf/libbpf/blob/42065ea6627ff6e1ab4c65e51042a70fbf30ff7c/src/libbpf.h#L825

[saxena-anurag]

Discussion outcome:

1. "Default" behavior can be different from Linux behavior, but then there may be a compat issue between Linux and Windows
2. Pass "program ID" to extensions, and then extensions can generate the list.
3. Implement a new Windows specific eBPF API, and maybe remove it in future when libbpf supports a more generic multi-prog attach API.