filter_kubernetes: add kubernetes_namespace metadata

[ryanohnemus]

This provides the ability to have the filter_kubernetes plugin fetch namespace labels & annotations and add them to the record under a kubernetes_namespace key.

• Added 3 new configuration options:

  • kube_meta_namespace_cache_ttl - FLB_CONFIG_MAP_TIME - the TTL for the namespace cache
  • namespace_annotations - FLB_CONFIG_MAP_BOOL - On if you want namespace annotations added to the record. Default: Off
  • namespace_labels - FLB_CONFIG_MAP_BOOL - On if you want namespace labels added to the record. Default: Off

• Fetching namespace metadata can not use kubelet so the shared variables around kubelet vs kubernetes api were completely split in this patch. This allows us to still use kubelet for pod information if Use_Kubelet On is set, while still being able to fetch namespace metadata from the upstream kubernetes api.

  • if neither namespace_labels or namespace_annotations are On then fetching of namespace metadata from the kubrenetes api will NOT occur.

• a second cache (flb_hash_table) and configuration for it's ttl was added specifically for caching namespace information. I found this important as you may want to refresh namespace label information every few minutes (in case namespace labels/annotations change), while only fetching new pod metadata if the cache is full. I defaulted the namespace cache to a 15min ttl.

• metadata fetching / extraction for pod vs namespace metadata is fairly separate intentionally as based on how this Pull Request goes, i would be willing to add the ability to add namespace metadata to non-pod logs (ie the ability to enrich kubernetes_events input with the same kubernetes_namespace data)

Closes #6544
Addresses #3865

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

[FILTER]
    Name kubernetes
    Match kube*
    K8S-Logging.Parser Off
    K8S-Logging.Exclude Off
    Use_Kubelet On
    Buffer_Size  1MB
    Annotations Off
    namespace_labels  On
    namespace_annotations On

• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

==18850== Memcheck, a memory error detector
==18850== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==18850== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==18850== Command: ./bin/flb-rt-filter_kubernetes
==18850== 
Test kube_core_base...                          [ OK ]
Test kube_core_no_meta...                       [ OK ]
Test kube_core_unescaping_text...               [ OK ]
Test kube_core_unescaping_json...               [ OK ]
Test kube_options_use-kubelet_enabled_json...   [ OK ]
Test kube_options_use-kubelet_disabled_json...  [ OK ]
Test kube_options_merge_log_enabled_text...     [ OK ]
Test kube_options_merge_log_enabled_json...     [ OK ]
Test kube_options_merge_log_enabled_invalid_json... [ OK ]
Test kube_options_merge_log_disabled_json...    [ OK ]
Test kube_options_merge_log_trim_enabled_json... [ OK ]
Test kube_options_merge_log_trim_disabled_json... [ OK ]
Test kube_options_merge_log_key_json...         [ OK ]
Test kube_options_keep_log_enabled_json...      [ OK ]
Test kube_options_keep_log_disabled_json...     [ OK ]
Test kube_options_k8s_logging_parser_disabled_text_stdout... [ OK ]
Test kube_options_k8s_logging_parser_disabled_text_stderr... [ OK ]
Test kube_options_k8s_logging_exclude_disabled_text_stdout... [ OK ]
Test kube_options_k8s_logging_exclude_disabled_text_stderr... [ OK ]
Test kube_annotations_invalid_text...           [ OK ]
Test kube_annotations_parser_regex_with_time_text... [ OK ]
Test kube_annotations_parser_regex_with_time_invalid_text_1... [ OK ]
Test kube_annotations_parser_json_with_time_json... [ OK ]
Test kube_annotations_parser_json_with_time_invalid_json_1... [ OK ]
Test kube_annotations_parser_invalid_text_stdout... [ OK ]
Test kube_annotations_parser_invalid_text_stderr... [ OK ]
Test kube_annotations_parser_stdout_text_stdout... [ OK ]
Test kube_annotations_parser_stdout_text_stderr... [ OK ]
Test kube_annotations_parser_stderr_text_stdout... [ OK ]
Test kube_annotations_parser_stderr_text_stderr... [ OK ]
Test kube_annotations_parser_multiple_1_container_1_stdout... [ OK ]
Test kube_annotations_parser_multiple_1_container_1_stderr... [ OK ]
Test kube_annotations_parser_multiple_1_container_2_stdout... [ OK ]
Test kube_annotations_parser_multiple_1_container_2_stderr... [ OK ]
Test kube_annotations_parser_multiple_1_container_3_stdout... [ OK ]
Test kube_annotations_parser_multiple_1_container_3_stderr... [ OK ]
Test kube_annotations_parser_multiple_1_container_4_stdout... [ OK ]
Test kube_annotations_parser_multiple_1_container_4_stderr... [ OK ]
Test kube_annotations_parser_multiple_1_container_5_stdout... [ OK ]
Test kube_annotations_parser_multiple_1_container_5_stderr... [ OK ]
Test kube_annotations_parser_multiple_2_container_1_stdout... [ OK ]
Test kube_annotations_parser_multiple_2_container_1_stderr... [ OK ]
Test kube_annotations_parser_multiple_2_container_2_stdout... [ OK ]
Test kube_annotations_parser_multiple_2_container_2_stderr... [ OK ]
Test kube_annotations_parser_multiple_2_container_3_stdout... [ OK ]
Test kube_annotations_parser_multiple_2_container_3_stderr... [ OK ]
Test kube_annotations_parser_multiple_2_container_4_stdout... [ OK ]
Test kube_annotations_parser_multiple_2_container_4_stderr... [ OK ]
Test kube_annotations_parser_multiple_2_container_5_stdout... [ OK ]
Test kube_annotations_parser_multiple_2_container_5_stderr... [ OK ]
Test kube_annotations_exclude_default_text...   [ OK ]
Test kube_annotations_exclude_invalid_text_stdout... [ OK ]
Test kube_annotations_exclude_invalid_text_stderr... [ OK ]
Test kube_annotations_exclude_stdout_text_stdout... [ OK ]
Test kube_annotations_exclude_stdout_text_stderr... [ OK ]
Test kube_annotations_exclude_stderr_text_stdout... [ OK ]
Test kube_annotations_exclude_stderr_text_stderr... [ OK ]
Test kube_annotations_exclude_multiple_1_container_1_stdout... [ OK ]
Test kube_annotations_exclude_multiple_1_container_1_stderr... [ OK ]
Test kube_annotations_exclude_multiple_1_container_2_stdout... [ OK ]
Test kube_annotations_exclude_multiple_1_container_2_stderr... [ OK ]
Test kube_annotations_exclude_multiple_1_container_3_stdout... [ OK ]
Test kube_annotations_exclude_multiple_1_container_3_stderr... [ OK ]
Test kube_annotations_exclude_multiple_1_container_4_stdout... [ OK ]
Test kube_annotations_exclude_multiple_1_container_4_stderr... [ OK ]
Test kube_annotations_exclude_multiple_2_container_1_stdout... [ OK ]
Test kube_annotations_exclude_multiple_2_container_1_stderr... [ OK ]
Test kube_annotations_exclude_multiple_2_container_2_stdout... [ OK ]
Test kube_annotations_exclude_multiple_2_container_2_stderr... [ OK ]
Test kube_annotations_exclude_multiple_2_container_3_stdout... [ OK ]
Test kube_annotations_exclude_multiple_2_container_3_stderr... [ OK ]
Test kube_annotations_exclude_multiple_2_container_4_stdout... [ OK ]
Test kube_annotations_exclude_multiple_2_container_4_stderr... [ OK ]
Test kube_annotations_exclude_multiple_3_container_1_stdout... [ OK ]
Test kube_annotations_exclude_multiple_3_container_1_stderr... [ OK ]
Test kube_annotations_exclude_multiple_3_container_2_stdout... [ OK ]
Test kube_annotations_exclude_multiple_3_container_2_stderr... [ OK ]
Test kube_annotations_exclude_multiple_3_container_3_stdout... [ OK ]
Test kube_annotations_exclude_multiple_3_container_3_stderr... [ OK ]
Test kube_annotations_exclude_multiple_3_container_4_stdout... [ OK ]
Test kube_annotations_exclude_multiple_3_container_4_stderr... [ OK ]
Test kube_annotations_exclude_multiple_4_container_1_stdout... [ OK ]
Test kube_annotations_exclude_multiple_4_container_1_stderr... [ OK ]
Test kube_annotations_exclude_multiple_4_container_2_stdout... [ OK ]
Test kube_annotations_exclude_multiple_4_container_2_stderr... [ OK ]
Test kube_annotations_exclude_multiple_4_container_3_stdout... [ OK ]
Test kube_annotations_exclude_multiple_4_container_3_stderr... [ OK ]
Test kube_annotations_exclude_multiple_4_container_4_stdout... [ OK ]
Test kube_annotations_exclude_multiple_4_container_4_stderr... [ OK ]
Test kube_systemd_logs...                       [ OK ]
SUCCESS: All unit tests have passed.
==18850== 
==18850== HEAP SUMMARY:
==18850==     in use at exit: 0 bytes in 0 blocks
==18850==   total heap usage: 277,552 allocs, 277,552 frees, 102,355,837 bytes allocated
==18850== 
==18850== All heap blocks were freed -- no leaks are possible
==18850== 
==18850== For lists of detected and suppressed errors, rerun with: -s
==18850== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

fluent/fluent-bit-docs#1280

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[ryanohnemus]

Added a second commit for the following... thought it was probably a good idea to include at least one test 😄 :

filter_kubernetes: Add Tests for namespace labels & annotations
    - also updates error text for kubelet vs kube api upstream errors
    - get_namespace_api_server_info passes in meta->namespace
      instead of ctx->namespace which may not be set at time of call

[ryanohnemus]

Initial test run failed due to missing test file. I didn't catch the .gitignore ignores *.log :). Just pushed an amended 2nd commit with the missing file. 🤞

[patrick-stephens]

@ryanohnemus I think we can link out to tests proving this now? That will demonstrate both the current behaviour working and this new behaviour right? If you can link that with some example output as well to show @edsiper this is well tested that will help.

[ryanohnemus]

@patrick-stephens @edsiper - yes, i added CI tests for the kubernetes_filter here: https://github.com/fluent/fluent-bit-ci/blob/main/tests/kubernetes-plugins/full.bats and we have an example of those passing (also attached to this PR via the checks) here: https://github.com/fluent/fluent-bit/actions/runs/7654984585/job/20860421190?pr=8279

[patrick-stephens]

@ryanohnemus post the log contents as well for posterity as they'll expire eventually

[ryanohnemus]

@patrick-stephens understood!

---

1. namespace_labels only test, where a label this_is_a_namespace_label: true exists. The filter config is:

    [FILTER]
        Name             kubernetes
        Alias            k8s_namespace_labels_only
        Match            kube.*k8s-namespace-label-tester*
        Kube_URL         https://kubernetes.default.svc:443
        Kube_CA_File     /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File  /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix  kube.var.log.containers.
        Merge_Log        Off
        Annotations      Off
        Labels           Off
        Namespace_labels On
        Namespace_annotations Off

And the output is:

{"date":1706799783.840397,"time":"2024-02-01T15:03:03.840397339Z","stream":"stdout","_p":"F","log":"hello world from k8s-namespace-label-tester","kubernetes_namespace":{"name":"test","labels":{"kubernetes.io/metadata.name":"test","this_is_a_namespace_label":"true"}}}

You should see a kubernetes_namespace (namespace labels) map and no kubernetes (pod labels) map in this one.

---

2. namespace labels and pod labels test

config:

    [FILTER]
        Name             kubernetes
        Alias            k8s_namespace_and_pod_labels_without_kubelet
        Match            kube.*k8s-pod-and-namespace-label-tester*
        Kube_URL         https://kubernetes.default.svc:443
        Kube_CA_File     /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File  /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix  kube.var.log.containers.
        Merge_Log        Off
        Annotations      Off
        Labels           On
        Namespace_labels On

output:

{"date":1706799799.653302,"time":"2024-02-01T15:03:19.653302082Z","stream":"stdout","_p":"F","log":"hello world from k8s-pod-and-namespace-label-tester","kubernetes":{"pod_name":"k8s-pod-and-namespace-label-tester","namespace_name":"test","pod_id":"2203df11-9417-47f6-b4de-e698191cce14","labels":{"this_is_a_test_label":"true"},"host":"kind-worker2","container_name":"k8s-pod-and-namespace-label-tester","docker_id":"616979d997d0a7e4c517c15da07424506e9e8d3410c3c55dd2f69e590aab8c6b","container_image":"docker.io/library/alpine:latest"},"kubernetes_namespace":{"name":"test","labels":{"kubernetes.io/metadata.name":"test","this_is_a_namespace_label":"true"}}}

You should see both a kubernetes_namespace (namespace labels) map and a kubernetes (pod labels) map in this one.

[ryanohnemus]

ci test failure looks like a flake on gke runner, and mac os unit test is a failure on flb-it-log

[patrick-stephens]

Yeah @ryanohnemus I'm happy you've done your best with the flakes :)
There's a fair bit of stuff to review but this is in the pipeline

[edsiper]

thank you!

[patrick-stephens]

Well done @ryanohnemus!