fluent-bit memory not released when setting up the input log file path with wild card

[vikas271019]

Bug Report

fluent-bit process memory not being released even if we deleted the complete list of log file path.
Describe the bug

when deploying fluent-bit sidecar with loggen microservice with the input log file path being set as wildcard /test*.log.
we observed while creating a thousands of log files from test1.log to tet1000.log, fluent-bit process memory increases for instance from 13660 KB to 23792 KB. After sometime we manually deleted all the log files including the default one of test0.log. we found the memory of the fluent-bit process is not decreasing.
When we checked to identify the file descriptor of the deleted file is still open using the below command
lsof | grep deleted
--> output of the command as --1 /tmp/loggen/loggen.bin test0.log (deleted)
that means above output will list all processes with open file descriptor that are linked to deleted files.
Overall this means, deleted file still being open and hence fluent-bit process memory not being released.

we also tried ways or configuration from fluent-bit to release the memory when files are being deleted.
configure Ignore_Older – prevent fluent-bit from tailing old or deleted file
Exit_On_Eof fluent-bit to stop reading a file after deletion

Even after following the above recommendation , there is release in memory of fluent-bit process.

Expected behavior
Ideally there should be a way or parameter from fluent-bit configuration to release the memory.

Your Environment
please refer the below config map of fluent-bit sidecar

[INPUT]
    name              tail
    tag               event.fluent-bit
    alias             log_shipper
    buffer_chunk_size 32k
    buffer_max_size   32k
    path              /logs/logshipper.log
    path_key          filename
    read_from_head    true
    refresh_interval  5
    rotate_wait       10
    skip_empty_lines  off
    skip_long_lines   off
    key               message
    db                /logs/logshipper.db
    db.sync           normal
    parser            json
    mem_buf_limit     1MB
[INPUT]
    name              tail
    tag               event.file0
    alias             file0
    buffer_chunk_size 32k
    buffer_max_size   32k
    read_from_head    true
    refresh_interval  5
    rotate_wait       10
    skip_empty_lines  off
    skip_long_lines   off
    key               message
    db                /logs/file0.db
    db.sync           normal
    db.locking        true
    db.journal_mode   off
    path              /logs/test*.log
    path_key          filename
    exclude_path      /logs/logshipper.log
    mem_buf_limit     1MB

[fluent bit] version=3.0.5

[sasikiranvaddi]

Hi @lecaros ,

Have you had an opportunity to review this issue? If so, I would appreciate any updates or insights you could provide. Thank you!"

[sasikiranvaddi]

Hi @lecaros ,
We feel when files that are matching wildcard is deleted (around 500files), from fluent-bit we could see watchers are removed for those files and then from source code 1 we see a inotify remove watch signal is sent to linux via sys/inotify.h, this will remove the kernel-side resources related to the inotify watch are released but user space memory that is allocated for the watcher file should be explicitly freedup, this might not be happening here.

{"timestamp":"2024-12-02T08:30:22.153+00:00","severity":"info","message":"[input:tail:file0] inotify_fs_remove(): inode=959 watch_fd=926"}
{"timestamp":"2024-12-02T08:30:22.154+00:00","severity":"info","message":"[input:tail:file0] inotify_fs_remove(): inode=960 watch_fd=927"}

bash-4.4$ ldd --version
ldd (GNU libc) 2.38

[lecaros]

hi @sasikiranvaddi, this should be the next on my list (unless something unexpected pushes it back). Thanks for your patience!

[lecaros]

Hi @sasikiranvaddi,
I'm not able to reproduce this behavior. I've used a small set of files, but then used 30k, just in case it was observed with higher numbers.
I can see that after deleting the files (rm or rm -f rm -rf logs/test92*), Fluent Bit no longer follows such files (➜ ~ lsof -p $(pidof fluent-bit) | grep -i test92 returns nothing).
Also lsof | grep deleted returns no matches of any log file.

In another run with log_level debug in the input plugin, I can see the files being removed from tail and the db file, along with the decrease in memory.

[2024/12/07 09:30:28] [debug] [input:tail:file0] inode=519746754 removing file name ./logs/test9995.log
[2024/12/07 09:30:28] [debug] [input:tail:file0] purge: monitored file has been deleted: ./logs/test9996.log
[2024/12/07 09:30:28] [debug] [input:tail:file0] db: file deleted from database: ./logs/test9996.log
[2024/12/07 09:30:28] [debug] [input:tail:file0] inode=519746755 removing file name ./logs/test9996.log

Are you able to reproduce this with vanilla Fluent Bit (running the binary outside of kubernetes).

I used version: version=3.2.3, commit=9acc096d0c

[sasikiranvaddi]

Hi @lecaros,
We have tried with 3.0.5 version of fluent-bit. We will give a try with 3.2.3 and see how it behaves.
We have not tried outside of Kubernetes.