PoC for log tailing shipment via log gateway

[a-thaler]

Description
As outlined in #556 it would be best to have a central log gateway. An optional additional component should be the log agent log agent, which is tailing the application logs and forwarding them to the gateway. The approach of separating the setup for log tailing in a agent and gateway part will have the advantage:

• to have well-defined responsibilities of the components (log enrichment happens only in the gateway)
• the setup will be consistent with the metrics scenario
• only one component dealing with backend connectivity (secret handling)

However, it might

• have a resource-consumption impact as two components need to process log data instead of 1 as it is in the current fluentbit based setup.
• have a scaling implication as not many pods (the agent instances) are sending to the backend anymore but only few (the gateway instances)
• have a durability impact, as there are more places where logs could be dropped

Goal:
Build up knowledge in regards to retry handling and buffering and make experiments in order to come up with a decision on if the downsides are acceptable/can be mitigated so that we can forward with the target design.

Note:

• Using a batch processor in the agent will not cause a pause of the tailing, maybe we need to remove the batching there
• A re-enqueueing seems not to happen as in fluentbit

Criteria

• Document the most relevant findings in regards to otel-collector behaviour
• Create a concept on the new architecture incl. buffering/persistence setup
• Answers to the questions:
  • Agent will ship via gateway or not
  • Where do we have persistent buffers (agent and/or gateway)
  • Can we keep our current guarantees (see limitation section) or do we need adjustments

Reasons

Attachments

Release Notes

[chrkl]

PoC results have been documented with #843.