Support configuring a poll interval in s3 Kafka Connect Source

[jessebye]

What is the expected behaviour?

We'd like to be able to configure a polling interval for the s3 Kafka Connect source so we can reduce AWS i/o charges incurred by frequent polling.

What was observed?

There is no way to configure the polling interval. I'm not sure what the default is, but it's very frequent and causing high AWS bills in our development environments.

[stheppi]

Hi @jessebye ,

Would a backoff also be useful for you? I am thinking if there are no new objects to slow down the polling interval up to a given max threshold.

[jessebye]

@stheppi Backoff would be nice to have. I could see that being useful in prod configurations too.

However, the main issue we'd like to address is our dev environment costs. Currently it's about $300 USD per day to keep a few s3 connectors active because they poll every 40ms or so. If we could configure that to poll every 60s, it would cut our bills dramatically.