Speed up FindByPath.find 1000x on K8s/RDS environments #1163
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Trello card
Benchmarking under sustained load on staging after deploying content-store-proxy PR #48 showed that we had optimised to the point where the bottleneck was now the PostgreSQL content-store itself (see secondary_response_time in this Kibana screenshot)
Some more benchmarking in a console on the staging pods showed an unexpected slow point in
FindByPath.find()
- theif matches.any?
call.On the original Mongoid (main) application, this line is operating on an array, so
any?
is vanishingly fast.However, in ActiveRecord, this is operating on an
ActiveRecord::Relation
, and it is much slower. Experimentation showed that the fastest functionally-equivalent call is not even.exists?
butcount > 0
The difference is hardly noticeable on local developer laptops, but huge on staging - where the database is remote (RDS) and the container is resource-constrained in a highly-contended cluster.
Before:
After:
So, roughly 1000x faster
This application is owned by the publishing platform team. Please let us know in #govuk-publishing-platform when you raise any PRs.
Follow these steps if you are doing a Rails upgrade.