Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent information about loss of data on buffering for tail input plugin #1529

Open
MarcioJales opened this issue Dec 13, 2024 · 0 comments

Comments

@MarcioJales
Copy link

Hi!

I've found what I think it is some inconsistency about the possible data loss on the tail input plugin in respect of buffering.

In the "backpressure" documentation, it is stated that "The tail plugin can handle pauses without data
ingloss (sic), storing its current file offset and resuming reading later":

(for example, TCP input plugin). The tail plugin can handle pauses without data

In the other hand, the "Buffering and Storage" documentation says the opposite, "For some inputs, such as TCP and tail, pausing the input will almost certainly lead to log loss":

In a high load environment with backpressure the risks of having high memory usage is the chance of getting killed by the Kernel \(OOM Killer\). A workaround for this backpressure scenario is to limit the amount of memory in records that an input plugin can register, this configuration property is called `mem_buf_limit`. If a plugin has enqueued more than the `mem_buf_limit`, it won't be able to ingest more until that data can be delivered or flushed properly. In this scenario the input plugin in question is paused. When the input is paused, records will not be ingested until it is resumed. For some inputs, such as TCP and tail, pausing the input will almost certainly lead to log loss. For the tail input, Fluent Bit can save its current offset in the current file it is reading, and pick back up when the input is resumed.

I'd like to know, then, what is the behavior expected in this realm?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant