Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filecount input plugin timeout #16357

Open
RomKimchi opened this issue Dec 30, 2024 · 1 comment
Open

filecount input plugin timeout #16357

RomKimchi opened this issue Dec 30, 2024 · 1 comment
Labels
feature request Requests for new plugin and for new features to existing plugins

Comments

@RomKimchi
Copy link

Use Case

filecount can very easily over stress a machine resources like CPU.

Expected behavior

It would be great if it will have a timeout mechanism so it wont count files until collection interval is reached.
i imagine it something like this

  [[inputs.filecount]]
    directories = ["D:/AidocData/Temp/IncomingDicoms"]
    name = "*.dcm"
    size = "0B"
    timeout = "10s"

or, another possible solution is setting a maximum of files it can collect and stopping after it (also adding a field that indicates that the maximum is reached)

  [[inputs.filecount]]
    directories = ["D:/Temp/IncomingData"]
    name = "*.txt"
    size = "0B"
    maximum_files = "1000000"

example:

filecount,directory=/var/cache/apt is_maximum_reached=true,count=1000000,size_bytes=7438336i,oldest_file_timestamp=1507152973123456789i,newest_file_timestamp=1507152973123456789i 1530034445000000000

Actual behavior

feature doesnt exist - telegraf cpu is spiking when dealing with tens of millions of files

Additional info

No response

@RomKimchi RomKimchi added the feature request Requests for new plugin and for new features to existing plugins label Dec 30, 2024
@danWalt
Copy link

danWalt commented Dec 30, 2024

Agree!
I ran into a similar problem which is currently limiting me from widely deploying and using this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Requests for new plugin and for new features to existing plugins
Projects
None yet
Development

No branches or pull requests

2 participants