Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: increase chunk_size for lightweight monitors !! #978

Open
shahzad31 opened this issue Nov 1, 2024 · 1 comment
Open

feat: increase chunk_size for lightweight monitors !! #978

shahzad31 opened this issue Nov 1, 2024 · 1 comment

Comments

@shahzad31
Copy link
Contributor

We have 100 default chunk_size for pushing monitors. This makes sense for browser monitors, since there bundle size can be big or small. So we set a very small number of 100, but it doesn't make sense for lightweight monitors, which are very small in size.

Why

small chunk_size add extra load on kibana in terms of processing with subsequent requests, we make bulk operations in kibana and having size of 100 or 1000 won't make a difference but having back to back 10 requests makes batching useless in some aspect.

Solution

We should increase size of chunk for lightweight monitors to 1000 and for browser monitors we can keep it to 100.

Kibana changes

We will also need to update kibana to accept more monitors, since it's also set to 250 at the moment.

@vigneshshanmugam
Copy link
Member

I like the idea of having different limits for the Lightweight and Browser monitors. But we need to do some stress testing on the Kibana side as it might cause some memory spikes when processing 1000s of monitors s well as the how it affects the Integration policies on the Private Locations.

@shahzad31 Is there a plan to also increase the limits between Kibana and our globally managed locations? I believe that is already at 250 per location?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants