Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runbook: clarify MimirIngesterReachingSeriesLimit errors and retries #9410

Merged
merged 3 commits into from
Dec 20, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/sources/mimir/manage/mimir-runbooks/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,15 @@ If nothing obvious from the above, check for increased load:

### MimirIngesterReachingSeriesLimit

This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester is reaching the limit. Once the limit is reached, writes to the ingester will fail (5xx) for new series, while appending samples to existing ones will continue to succeed.
This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester is reaching the limit.
bboreham marked this conversation as resolved.
Show resolved Hide resolved
The threshold is set at 80%, to give some chance to react before the limit is reached.
bboreham marked this conversation as resolved.
Show resolved Hide resolved
Once the limit is reached, writes to the ingester will fail for new series. Appending samples to existing ones will continue to succeed.
bboreham marked this conversation as resolved.
Show resolved Hide resolved

Note that the error responses sent back to the sender are classed as "server error" (5xx), which should result in a retry by the sender.
bboreham marked this conversation as resolved.
Show resolved Hide resolved
While this situation continues, these retries will stall the flow of data, and newer data will queue up on the sender.
bboreham marked this conversation as resolved.
Show resolved Hide resolved
If the condition is cleared in a short time, service can be restored with no data loss.
bboreham marked this conversation as resolved.
Show resolved Hide resolved

This is different to what happens when the `max_global_series_per_user` is exceeded, which is considered a "client error" (4xx) where excess data is discarded.
bboreham marked this conversation as resolved.
Show resolved Hide resolved

In case of **emergency**:

Expand Down
Loading