flb_lib: allow public-API use without dependencies via fluent-bit-minimal.h

[nrdvana]

Currently, fluent-bit.h and flb_lib.h are un-usable by third-parties
becuase the fluent-bit build process does not install all of the
required header files for fluent-bit's dependencies into the Debian
packages. Skipping that problem for the moment, I'm proposing that
users of the public API don't actually need or want all the headers
required for the interenals of fluent-bit, and there should just be
a simple header that declares the public API.

This new header sets a preprocessor variable and then includes the
normal flb_lib.h, which defines the public API. flb_lib.h sees the
preprocessor variable and skips loading any of the dependency
headers, instead just declaring the referenced types without
defining them.

To verify that fluent-bit-minimal.h works, I updated one of the
existing unit tests to use it. Many more unit tests could be
switched, or a new unit test entirely could be added, but this
seemed like the simplest solution for testing.

I also updated the examples to use the new header, since this
is what third party users will be looking at first.

Fixes #7165

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• [N/A] Example configuration file for the change
• Debug log output from testing the change
• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[nrdvana]

Test Output

Output of the modified unit test:

      Start  3: flb-rt-in_event_test
 3/80 Test  #3: flb-rt-in_event_test .......................   Passed    9.00 sec

Snippet from running make install

-- Installing: /usr/local/bin/luajit
-- Installing: /usr/local/include/fluent-bit.h
-- Installing: /usr/local/include/fluent-bit-minimal.h
-- Installing: /usr/local/include/fluent-bit/flb_api.h

Verified hello_world.c still compiles and runs:

bin/hello_world >/dev/null && echo $?
[2023/12/04 20:45:16] [ info] [fluent bit] version=2.2.1, commit=9016ad8f83, pid=45155
[2023/12/04 20:45:16] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/12/04 20:45:16] [ info] [cmetrics] version=0.6.5
[2023/12/04 20:45:16] [ info] [ctraces ] version=0.3.1
[2023/12/04 20:45:16] [ info] [input:lib:lib.0] initializing
[2023/12/04 20:45:16] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/12/04 20:45:16] [ info] [sp] stream processor started
[2023/12/04 20:45:16] [ info] [output:stdout:stdout.0] worker #0 started
[2023/12/04 20:45:16] [ warn] [engine] service will shutdown in max 5 seconds
[2023/12/04 20:45:17] [ info] [engine] service has stopped (0 pending tasks)
[2023/12/04 20:45:17] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/12/04 20:45:17] [ info] [output:stdout:stdout.0] thread worker #0 stopped
0

[nrdvana]

Packaging

I assume this counts as a change to packages, since there is a new additional file, but I couldn't get the packaging script to run.

+++ dirname -- packaging/local-build-all.sh
++ cd -- packaging
++ pwd
+ SCRIPT_DIR=/fluent-bit/packaging
+ JSON_FILE_NAME=/fluent-bit/packaging/build-config.json
+ PACKAGING_OUTPUT_DIR=test
+ echo 'Cleaning any existing output'
Cleaning any existing output
+ rm -rf 'test/*'
+ jq -cr '.linux_targets[]' /fluent-bit/packaging/build-config.json
+ read -r DISTRO
+ echo '{"target":"amazonlinux/2","type":"rpm"}'
{"target":"amazonlinux/2","type":"rpm"}
+ FLB_OUT_DIR=test
+ /bin/bash /fluent-bit/packaging/build.sh -d '{"target":"amazonlinux/2","type":"rpm"}'
+++ dirname -- /fluent-bit/packaging/build.sh
++ cd -- /fluent-bit/packaging
++ pwd
+ SCRIPT_DIR=/fluent-bit/packaging
+ FLB_DISTRO=
+ FLB_OUT_DIR=test
+ FLB_NIGHTLY_BUILD=
+ FLB_JEMALLOC=On
+ FLB_ARG=
+ getopts v:d:b:t:o: option
+ case "${option}" in
+ FLB_DISTRO='{"target":"amazonlinux/2","type":"rpm"}'
+ getopts v:d:b:t:o: option
+ '[' -z '{"target":"amazonlinux/2","type":"rpm"}' ']'
+ '[' -n test ']'
+ out_dir=test
+ volume='/fluent-bit/packaging/packages/{"target":"amazonlinux/2","type":"rpm"}/test/'
+ mkdir -p '/fluent-bit/packaging/packages/{"target":"amazonlinux/2","type":"rpm"}/test/'
+ echo 'FLB_DISTRO            => {"target":"amazonlinux/2","type":"rpm"}'
FLB_DISTRO            => {"target":"amazonlinux/2","type":"rpm"}
+ echo 'FLB_OUT_DIR           => test'
FLB_OUT_DIR           => test
+ MAIN_IMAGE='flb-{"target":"amazonlinux/2","type":"rpm"}'
+ IMAGE_CONTEXT_DIR='/fluent-bit/packaging/distros/{"target":"amazonlinux/2","type":"rpm"}'
+ [[ ! -d /fluent-bit/packaging/distros/{"target":"amazonlinux/2","type":"rpm"} ]]
+ IMAGE_CONTEXT_DIR='/fluent-bit/packaging/distros/{"target":"amazonlinux'
+ FLB_ARG=' --build-arg BASE_BUILDER={"target":"amazonlinux-2","type":"rpm"}-base --target builder'
+ [[ ! -f /fluent-bit/packaging/distros/{"target":"amazonlinux/Dockerfile ]]
+ echo 'Unable to find /fluent-bit/packaging/distros/{"target":"amazonlinux/Dockerfile'
Unable to find /fluent-bit/packaging/distros/{"target":"amazonlinux/Dockerfile
+ exit 1

[nrdvana]

Documentation

Assuming the change is accepted, the documentation at https://docs.fluentbit.io/manual/development/ingest-records-manually should be updated to refer to #include <fluent-bit-minimal.h>

Back-porting

Currently, libfluent-bit.so is not usable for compilation on any platform (afaik), due to the missing headers. It wouldn't hurt to back-port these header changes to past versions, if that would help it get into package repositories where a past version is considered stable. That would allow people to compile new software using libfluent-bit.so on those platforms. However, since it hasn't been possible so far, there probably isn't much demand for that.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[nrdvana]

I'd still appreciate it if this got merged.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[nrdvana]

Anyone able to review this yet?

[nrdvana]

I'd still appreciate it if this work didn't go to waste.