Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up FindByPath.find 1000x on K8s/RDS environments #1163

Merged
merged 1 commit into from
Oct 12, 2023

Conversation

aldavidson
Copy link
Contributor

@aldavidson aldavidson commented Oct 12, 2023

Trello card

Benchmarking under sustained load on staging after deploying content-store-proxy PR #48 showed that we had optimised to the point where the bottleneck was now the PostgreSQL content-store itself (see secondary_response_time in this Kibana screenshot)

Screenshot from 2023-10-11 17-12-05

Some more benchmarking in a console on the staging pods showed an unexpected slow point in FindByPath.find() - the if matches.any? call.

On the original Mongoid (main) application, this line is operating on an array, so any? is vanishingly fast.
However, in ActiveRecord, this is operating on an ActiveRecord::Relation, and it is much slower. Experimentation showed that the fastest functionally-equivalent call is not even .exists? but count > 0

The difference is hardly noticeable on local developer laptops, but huge on staging - where the database is remote (RDS) and the container is resource-constrained in a highly-contended cluster.

Before:

> Benchmark.measure{ 10.times{ FindByPath.new(ContentItem).find('/government/policies/improving-local-transport/supporting-pages/increasing-the-use-of-buses') } }
=> #<Benchmark::Tms:0x00007f9a4f4270e8 @cstime=0.0, @cutime=0.0, @label="", @real=23.315657227300107, @stime=0.0032899999999999874, @total=0.03041800000000003, @utime=0.02712800000000004>

After:

> Benchmark.measure{ 10.times{ FindByPath.new(ContentItem).find('/government/policies/improving-local-transport/supporting-pages/increasing-the-use-of-buses') } }
=> #<Benchmark::Tms:0x00007f9a50b67528 @cstime=0.0, @cutime=0.0, @label="", @real=0.02926267310976982, @stime=0.0, @total=0.009113000000000149, @utime=0.009113000000000149>

So, roughly 1000x faster

This application is owned by the publishing platform team. Please let us know in #govuk-publishing-platform when you raise any PRs.

⚠️ This repo is Continuously Deployed: make sure you follow the guidance ⚠️

Follow these steps if you are doing a Rails upgrade.

@aldavidson aldavidson changed the title Speed up FindByPath.find on K8s/RDS environments Speed up FindByPath.find 1000x on K8s/RDS environments Oct 12, 2023
@aldavidson aldavidson force-pushed the optimise-find-by-path branch from 7fab734 to 1b86cf7 Compare October 12, 2023 09:24
@aldavidson aldavidson merged commit 8c0c135 into port-to-postgresql Oct 12, 2023
4 checks passed
@aldavidson aldavidson deleted the optimise-find-by-path branch October 12, 2023 09:25
@sengi
Copy link
Contributor

sengi commented Oct 12, 2023

Nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants