[server] Reset current version replica to ERROR on ingestion error #1387
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
December 12, 2024 09:52
majisourav99
force-pushed
the
helixError
branch
from
cbf1dda to
a4382cb
Compare
majisourav99
changed the title
xunyin8
reviewed
Dec 13, 2024
| @@ -1439,10 +1439,23 @@ private void processIngestionException() { | |||
| // DaVinci is always a follower. | |||
| subscribePartition(pubSubTopicPartition, false); | |||
| } | |||
| } else if (isCurrentVersion.getAsBoolean() && resetErrorReplicaEnabled && !isDaVinciClient) { | |||
There was a problem hiding this comment.
Is this the right place to mark replicas ERROR? IIUC we may not call processIngestionException if lastStoreIngestionException is set.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reset current version replica to ERROR on ingestion error
For hybrid stores whose current version is ingesting the real time data from a real time topic, if there is any exception/error thrown during ingestion they would log error message, sit still and stop ingestion, resulting in stale replicas.
The mitigation mechanism is for oncall to manually restart every impacted cluster, which is very labor-intensive.
This PR tries to resolve that by marking such replicas to ERROR state, which later in a controller task to reset such error replicas, will try to initiate new state transition of those replicas which can possibly mitigate such stale replica issue.
How was this PR tested?
GHCI
Does this PR introduce any user-facing changes?