out_parseable: Plugin for sending logs to Parseable

[AdheipSingh]

Send kubernetes logs to https://www.parseable.com/

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[edsiper]

can you please provide the link to the API spec ?

[AdheipSingh]

• Here is the API doc for ingesting into parseable.

https://www.postman.com/parseable/parseable/request/au1fn5d/send-logs-to-a-log-stream-with-http-header

• Here is the sample config i am using.

      inputs: |
      [INPUT]
          Name tail
          Path /var/log/containers/*.log
          multiline.parser docker, cri
          Tag kube.*
          Mem_Buf_Limit 5MB
          Skip_Long_Lines On

      [INPUT]
          name kubernetes_events
          tag k8s_events

      [INPUT]
          Name systemd
          Tag host.*
          Systemd_Filter _SYSTEMD_UNIT=kubelet.service
          Read_From_Tail On

    ## https://docs.fluentbit.io/manual/pipeline/filters
    filters: |
      [FILTER]
          Name                kubernetes
          Match               kube.*
          Merge_Log           On
          Keep_Log            Off
          K8S-Logging.Parser  On
          K8S-Logging.Exclude On

    ## https://docs.fluentbit.io/manual/pipeline/outputs
    outputs: |
      [OUTPUT]
          Name parseable
          Match kube.*
          Server_Host parseable.parseable.svc.cluster.local
          Username admin
          Password admin
          Server_Port 80
          Stream $NAMESPACE
          Exclude_Namespaces kube-system, default

      [OUTPUT]
          Name parseable
          Match k8s_events
          Server_Host parseable.parseable.svc.cluster.local
          Server_Port 80
          Username admin
          Password admin
          Stream k8s-events 

[patrick-stephens]

Does this build for all targets and do we need any extra dependencies? We probably need some unit tests at least too.

I think at the moment you've not actually enabled it to build, only for the container but then we don't test or exercise that in unit tests or builds.

[patrick-stephens]

Is this what should be used in the cmake config? It's already defaulting to on

[AdheipSingh]

feel free to suggest the best practice here. I tried to build without it but was not able to run the plugin.

[AdheipSingh]

Does this build for all targets and do we need any extra dependencies? We probably need some unit tests at least too.

I think at the moment you've not actually enabled it to build, only for the container but then we don't test or exercise that in unit tests or builds.

Please suggest some examples on unit tests and what can be done on building the ideal way. I'll implement this. ( i am not an expert in C codebase, so whatever is the best practice, ill will try to adhere to it ).

[AdheipSingh]

Hi @patrick-stephens review comments have been addressed. Over to you :)
cc @nitisht

[nitisht]

@edsiper @patrick-stephens would be great to have a next round of review please