Tail plugin incorrectly removing entries from the database file during startup

[OS-tomaslaw]

Bug Report

Describe the bug

As part of the changes introduced in #8062 (in this commit c60999c), Fluent Bit now deletes items that are not being monitored when it starts.

However, this functionality causes an undesired behavior in the scenario where:

• there is more than 1 instance of the Tail input plugin being used (each monitoring a specific pattern of file names)
• there is a single database file to keep track of all monitored files and offsets

It seems that the changes introduced in 3.0.2 (#8062), cause the Tail plugin to compare the currently monitored files with the entries in the database file. If there are entries in the database file that don't have a corresponding monitored file, those entries are removed and any tracking information on that file is lost.

To Reproduce
Use the following configuration:

[SERVICE]
    log_level    error

[INPUT]
    Name           tail
    Read_from_Head true
    Path           /tmp/foo.txt
    db             /tmp/fluentbit.db

[INPUT]
    Name           tail
    Read_from_Head true
    Path           /tmp/bar.txt
    db             /tmp/fluentbit.db

[OUTPUT]
    Name stdout
    Match *

Where foo.txt and bar.txt contain:
foo.txt

foo

bar.txt

bar

(don't forget the newline after the text)

Notice that on the first Fluent Bit start, the output contains the expected result.

[0] tail.0: [[1717786236.421992900, {}], {"log"=>"foo"}]
[0] tail.1: [[1717786236.422190800, {}], {"log"=>"bar"}]

Notice that after the first restart on Fluent Bit, the output contains the result for the bar.txt file again.

[0] tail.1: [[1717786246.720946400, {}], {"log"=>"bar"}]

Notice that after the second restart on Fluent Bit, the output contains the result for the foo.txt and bar.txt files again.

[0] tail.0: [[1717786252.299557300, {}], {"log"=>"foo"}]
[0] tail.1: [[1717786252.299697800, {}], {"log"=>"bar"}]

So it seems that the Tail plugin is having issues keeping track of the files/offset after Fluent Bit is restarted.

Looking at the log file, we can see these relevant entries:

[debug] [input:tail:tail.0] 1 new files found on path '/tmp/foo.txt'
[info] [input:tail:tail.0] db: delete unmonitored stale inodes from the database: count=0
[debug] [input:tail:tail.1] 1 new files found on path '/tmp/bar.txt'
[ info] [input:tail:tail.1] db: delete unmonitored stale inodes from the database: count=1

Expected behavior

The database file would not lose information on the files being monitored by Fluent Bit.

Additional context

I can see that a possible solution would be to use a separate database file for each Tail plugin instance.
But performing that change to the configuration now would mean that Fluent Bit would start with a new database file. So any tracking information would be lost and all files would be processed again.
Is there any suggestion on how I could tackle this issue without causing files to be reprocessed and duplicating information?

Your Environment

• Version used: 3.0.4
• Operating System and version: Windows
• Filters and plugins: Tail

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[lchoudhu-tibco]

I think , this can be solved by soft delete

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[briandefiant]

We're seeing this same issue when using the same offsets DB for multiple tail inputs. The simplest fix for us will be using a separate DB per input, but this behavior was a bit unexpected and the documentation isn't really clear on whether using the same DB for multiple inputs is supported or not.