Fix retry for UniqueConstraintViolation in RouteCreate (v3) #3697
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
philippthun
force-pushed
the
create-route-retries
branch
from
7ac4973 to
488de6d
Compare
The creation of internal routes depends on finding the next vip_offset.
When creating multiple routes in parallel, they find the same vip_offset
and only the very first write operation succeeds. All the others fail
with a UniqueConstraintViolation error. For this error a retry is
triggered in the VCAP::CloudController::RouteCreate action (v3).
This retry did not work due to a bug ("undefined local variable or
method 'user_audit_info'"). Besides fixing this bug a test has been
added to ensure that the parallel creation of internal routes is
working.
When running 5 parallel threads and each thread is creating an internal
route, the following logs are produced:
Duplicate entry '1' for key 'routes.routes_vip_offset_index', retrying
Duplicate entry '1' for key 'routes.routes_vip_offset_index', retrying
Duplicate entry '1' for key 'routes.routes_vip_offset_index', retrying
Duplicate entry '1' for key 'routes.routes_vip_offset_index', retrying
Duplicate entry '2' for key 'routes.routes_vip_offset_index', retrying
Duplicate entry '2' for key 'routes.routes_vip_offset_index', retrying
Duplicate entry '2' for key 'routes.routes_vip_offset_index', retrying
Duplicate entry '3' for key 'routes.routes_vip_offset_index', retrying
Duplicate entry '3' for key 'routes.routes_vip_offset_index', retrying
Duplicate entry '4' for key 'routes.routes_vip_offset_index', retrying
This indicates that all the write operations are being executed at the
same point in time and only one of them succeeds whereas the others are
failing, i.e. 4 out of 5. For each retry there is one conflicting route
less, so the number of errors goes down from 4 to 3 to 2 to 1 until the
last route could be saved (after 4 retries).
philippthun
force-pushed
the
create-route-retries
branch
from
488de6d to
a8fe74e
Compare
tcdowney
approved these changes
Mar 28, 2024
There was a problem hiding this comment.
This looks good to me.
My question is, why are we even assigning VIPs to internal routes anymore? I thought that was something for Istio Copilot.
🤔 https://www.pivotaltracker.com/story/show/161849240
Update: Created #3712 to track that.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The creation of internal routes depends on finding the next
vip_offset. When creating multiple routes in parallel, they find the samevip_offsetand only the very first write operation succeeds. All the others fail with aUniqueConstraintViolationerror. For this error a retry is triggered in theVCAP::CloudController::RouteCreateaction (v3).This retry did not work due to a bug (
undefined local variable or method 'user_audit_info'). Besides fixing this bug a test has been added to ensure that the parallel creation of internal routes is working.When running 5 parallel threads and each thread is creating an internal route, the following logs are produced:
This indicates that all the write operations are being executed at the same point in time and only one of them succeeds whereas the others are failing, i.e. 4 out of 5. For each retry there is one conflicting route less, so the number of errors goes down from 4 to 3 to 2 to 1 until the last route could be saved (after 4 retries).
I have reviewed the contributing guide
I have viewed, signed, and submitted the Contributor License Agreement
I have made this pull request to the
mainbranchI have run all the unit tests using
bundle exec rakeI have run CF Acceptance Tests