Stress Testing Throughput for Club Submission Applications #606

shiva-menta · 2024-01-24T05:22:53Z

Conduct customizable stress tests using python manage.py stress_test. Change parameters including number of users, applications per user, number of clubs, etc.

HTTP requests are mocked using APIRequestFactory. Tested minimally before and after #603, and saw some performance gains in SQLite, but SQLite's concurrency features should be considered (thread explaining how SQLite handles concurrent writes!). Have not tested on Docker Postgres yet.

gitguardian · 2024-01-24T05:22:57Z

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	Secret	Commit	Filename
9335321	Generic Private Key	`ede0a26`	nginx/certs/_wildcard.localhost.net-key.pem	View secret
9335321	Generic Private Key	`33aa378`	nginx/certs/_wildcard.localhost.net-key.pem	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}

rohangpta · 2024-01-24T18:09:22Z

backend/clubs/management/commands/stress_test.py

+
+        self.uri = "/users/question_response/"
+        self.factory = APIRequestFactory()
+        self.view = UserViewSet.as_view({"post": "question_response"})


This will create objects for real right? So we want to tear down?

Yeah the tearDown() function handles that, but if it needs to be more explicit I can do that.

rohangpta · 2024-01-24T18:18:28Z

backend/clubs/management/commands/stress_test.py

+        self.users = list(User.objects.filter(username__startswith=self.user_prefix))
+        print("Finished setting up users.")
+
+    @sync_to_async


This decorator is misleading. All this is doing is taking your function and convering it to an awaitable. It unfortunately doesn't make your code "async for free". In fact, with thread_sensitive = True (which is the default behaviour), I believe these functions will all run in the main thread, and probably even slower then usual since it goes through the overhead of "asyncifying". Some more information here

It isn't worth changing it back since this is just a test but I thought I should let you know about this behaviour. IIf you wanted better async, you might want to use httpx or something like that (I think newer versions of Django support Async more natively).

Yeah I was starting to notice that in the end. I originally was designing it as sending HTTP requests and was going to use httpx but then the mocking seemed like a better option. Thanks for sharing!

rohangpta · 2024-01-24T18:20:08Z

backend/clubs/management/commands/stress_test.py

+        return end_time - start_time
+
+    def tearDown(self):
+        Club.objects.filter(code__startswith=self.club_prefix).delete()


Might have stray ClubApplication, ApplicationSubmission, etc objects which you'll need to tear down

Checking from the models, this should all be handled with on_delete behavior? It should form separate "trees" with roots of Clubs and Users.

Sure, this makes sense. Please do double check empirically though; we should definitely avoid excess footprint on the DB, especially if we test in prod multiple times.

Yeah just double checked, no rows are left over. The only concern I guess could be the autoincrementing index and offsetting that by the number of test instances. Should not be an issue at all, but if we want full isolation, we could create a separate database on the same instance.

I think this should be fine! Thank you for double checking

rohangpta · 2024-01-24T18:20:59Z

Can you post your test results here?

rohangpta · 2024-01-24T18:35:19Z

Great start Shiva! I think we're ready to take some steps towards productionising this test, some notes:

Test overall looks good to me, I like your choice of mocking HTTP request objects and overall the code is cleanly structured.
How long does this test take to run locally? Could you add some sleeps between submissions to make it more realistic?
Can you check whether your asyncifying is doing anything? (you can do this by noting the current end_time - start_time as a baseline, and comparing that against N * (individual end_time - start_time) values. I'm curious if sync_to_async is actually allowing parallel processing here, although my understanding is not.
Let's talk through the points here and here on Friday: some points on optimisation and DB configuration!
Before we productionise this, we should enable as many AWS insight metrics as possible so we can see where exactly the load is landing.

shiva-menta · 2024-01-24T18:58:48Z

Can you post your test results here?

Yeah I'm going to conduct more testing tonight, but I wanted to address the async behaviors first and potentially consider doing HTTP requests instead, but if our goal is to isolate database behavior, mocking makes more sense. Will show benchmarks for SQLite and PostgreSQL!

shiva-menta · 2024-01-25T00:13:27Z

Great start Shiva! I think we're ready to take some steps towards productionising this test, some notes:

Test overall looks good to me, I like your choice of mocking HTTP request objects and overall the code is cleanly structured.

How long does this test take to run locally? Could you add some sleeps between submissions to make it more realistic?

Can you check whether your asyncifying is doing anything? (you can do this by noting the current end_time - start_time as a baseline, and comparing that against N * (individual end_time - start_time) values. I'm curious if sync_to_async is actually allowing parallel processing here, although my understanding is not.

Let's talk through the points here and here on Friday: some points on optimisation and DB configuration!

Before we productionise this, we should enable as many AWS insight metrics as possible so we can see where exactly the load is landing.

Took a look, and yeah you're completely correct about sync_to_async, I made the incorrect assumption of seeing a difference in thread completion times implied concurrent connections; I believe it just indicated the overhead. When I set thread_sensitive=False, I get a bunch of database is locked errors due to the concurrent writes. I can't seem to configure SQLite's timeout setting properly so the database writes aren't blocking until the database is unlocked from the main thread.

I believe the implementation will be highly dependent on the actual database for now. I'm going to try seeing if PostgreSQL functions as is, because I'm not sure how informative SQLite will be if our main goal is to test a database with concurrency capabilities (table-locking vs. row-locking).

rohangpta · 2024-01-25T05:07:09Z

Yeah sounds good. I don't think thread_sensitive=False will end up working here since database adapters in Django require that they are accessed in the same thread that they were created in, but I agree that it's probably best to save the bulk of our testing for Postgres

rohangpta · 2024-02-05T15:30:29Z

@shiva-menta did we ever have any results on this? Curious about how things match up since we didn't actually seem to have any DB issues in prod this cycle.

Shiva Menta and others added 9 commits December 1, 2023 18:11

Initial Setup

fb3acda

Merge branch 'master' into stress_test

e530b44

Merge branch 'master' into stress_test

ede0a26

Write Initial Testing Script

a523da5

Finish Working Script

42e9fb1

Lint

3ad10be

Merge branch 'master' into stress_test

33aa378

Remove Extra Space

fcba047

Refactor Subset Size

38a3ba6

shiva-menta requested review from rohangpta, rm03 and aviupadhyayula January 24, 2024 05:22

shiva-menta added 3 commits January 24, 2024 10:18

Fix Deletion and Question Data

d537a6e

Add Bulk Creations for Setup

0de7037

Bulk Creation User Model

dcf81ab

rohangpta reviewed Jan 24, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stress Testing Throughput for Club Submission Applications #606

Stress Testing Throughput for Club Submission Applications #606

shiva-menta commented Jan 24, 2024

gitguardian bot commented Jan 24, 2024 •

edited

Loading

rohangpta Jan 24, 2024

shiva-menta Jan 24, 2024

rohangpta Jan 24, 2024 •

edited

Loading

shiva-menta Jan 24, 2024

rohangpta Jan 24, 2024

shiva-menta Jan 24, 2024

rohangpta Jan 24, 2024

shiva-menta Jan 24, 2024

rohangpta Jan 24, 2024

rohangpta commented Jan 24, 2024

rohangpta commented Jan 24, 2024 •

edited

Loading

shiva-menta commented Jan 24, 2024

shiva-menta commented Jan 25, 2024 •

edited

Loading

rohangpta commented Jan 25, 2024 •

edited

Loading

rohangpta commented Feb 5, 2024

Stress Testing Throughput for Club Submission Applications #606

Are you sure you want to change the base?

Stress Testing Throughput for Club Submission Applications #606

Conversation

shiva-menta commented Jan 24, 2024

gitguardian bot commented Jan 24, 2024 • edited Loading

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohangpta Jan 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohangpta commented Jan 24, 2024

rohangpta commented Jan 24, 2024 • edited Loading

shiva-menta commented Jan 24, 2024

shiva-menta commented Jan 25, 2024 • edited Loading

rohangpta commented Jan 25, 2024 • edited Loading

rohangpta commented Feb 5, 2024

gitguardian bot commented Jan 24, 2024 •

edited

Loading

rohangpta Jan 24, 2024 •

edited

Loading

rohangpta commented Jan 24, 2024 •

edited

Loading

shiva-menta commented Jan 25, 2024 •

edited

Loading

rohangpta commented Jan 25, 2024 •

edited

Loading