Load test Authenticated GET requests to various content types

[ElijahLynn]

In #8080 we got some baseline metrics for anonymous GET requests to the Benefits Detail Page which was around 217/second. We now need to test authenticated GET requests to various content types as that will be reflective of real world traffic when the new Next.js build system evolves to allowing for public requests.

AC

• Determine the average RPS (requests per second) for authenticated GET requests for a node of the Benefits Detail Page content type. This should be with a warmed cache.

Tasks

• Get Umami demo working via simplytest.me (quick hello-world example, not sustained, avoid DDoS)
• Write a test file for to login based on the Umami login code on local DEV
• Document initial setup and how to run load test
• Loadtest the Benefits Detail Page on staging.cms.va.gov
• Get SOCKS passthrough working ([docs] Run requests through a SOCKS proxy tag1consulting/goose#490) with example test file
• Post results to this thread, or maybe Confluence

Originally posted by @ElijahLynn in #8050 (comment)

We discussed in planning now, that the public API requests will be authenticated so it is still worth testing authenticated GET requests. I finally got Goose Load Testing working with concurrent requests on Friday so we should still pull in a new story this sprint to get these metrics on the authN GET requests.

[ElijahLynn]

Working on getting requests routed through our SOCKS proxy with Goose > tag1consulting/goose#490

[ElijahLynn]

cd tests/loadtest
# Start SOCKS PROXY FIRST
export ADMIN_USERNAME="content_build_api"
export ADMIN_PASSWORD="drupal8"
cargo run --release -- --host https://staging.cms.va.gov --report-file report-file.html --running-metrics 1 --users 125 --hatch-rate 10 --run-time 3m --no-reset-metrics

=== PER SCENARIO METRICS ===                                                  
------------------------------------------------------------------------------
Name                     |  # users |  # times run | scenarios/s | iterations 
------------------------------------------------------------------------------
1: Admin user            |      125 |        12914 |       66.23 |     103.31 
------------------------------------------------------------------------------
Name                     |    Avg (ms) |        Min |         Max |     Median
------------------------------------------------------------------------------
  1: Admin user          |        1785 |        449 |       3,699 |        449
                                                                              
=== PER TRANSACTION METRICS ===                                               
------------------------------------------------------------------------------
Name                     |   # times run |        # fails |  trans/s |  fail/s
------------------------------------------------------------------------------
1: Admin user                                                                 
  1: auth login          |           125 |         0 (0%) |     0.64 |    0.00
  2: auth /disability/.. |        13,039 |         0 (0%) |    66.87 |    0.00
-------------------------+---------------+----------------+----------+--------
Aggregated               |        13,164 |         0 (0%) |    67.51 |    0.00
------------------------------------------------------------------------------
Name                     |    Avg (ms) |        Min |         Max |     Median
------------------------------------------------------------------------------
1: Admin user                                                                 
  1: auth login          |        1827 |        887 |       5,201 |        887
  2: auth /disability/.. |        1785 |        449 |       3,699 |        449
-------------------------+-------------+------------+-------------+-----------
Aggregated               |        1785 |        449 |       5,201 |        449
                                                                              
=== PER REQUEST METRICS ===                                                   
------------------------------------------------------------------------------
Name                     |        # reqs |        # fails |    req/s |  fail/s
------------------------------------------------------------------------------
GET auth /disability/h.. |        13,039 |         0 (0%) |    66.87 |    0.00
GET auth login           |           125 |         0 (0%) |     0.64 |    0.00
GET static asset         |        79,984 |         0 (0%) |   410.17 |    0.00
POST auth login          |           125 |         0 (0%) |     0.64 |    0.00
-------------------------+---------------+----------------+----------+--------
Aggregated               |        93,273 |         0 (0%) |   478.32 |    0.00
------------------------------------------------------------------------------
Name                     |    Avg (ms) |        Min |         Max |     Median
------------------------------------------------------------------------------
GET auth /disability/h.. |      873.67 |        300 |       3,365 |        900
GET auth login           |      183.67 |         73 |       3,097 |         84
GET static asset         |      139.68 |          9 |       3,210 |         99
POST auth login          |      527.18 |        276 |       3,346 |        330
-------------------------+-------------+------------+-------------+-----------
Aggregated               |      242.86 |          9 |       3,365 |        130
------------------------------------------------------------------------------
Slowest page load within specified percentile of requests (in ms):            
------------------------------------------------------------------------------
Name                     |    50% |    75% |    98% |    99% |  99.9% | 99.99%
------------------------------------------------------------------------------
GET auth /disability/h.. |    900 |  1,000 |  1,000 |  1,000 |  2,000 |  3,000
GET auth login           |     84 |     95 |  2,000 |  3,000 |  3,000 |  3,000
GET static asset         |     99 |    200 |    500 |    600 |    800 |  3,000
POST auth login          |    330 |    600 |  3,000 |  3,000 |  3,000 |  3,000
-------------------------+--------+--------+--------+--------+--------+-------
Aggregated               |    130 |    270 |  1,000 |  1,000 |  1,000 |  3,000
------------------------------------------------------------------------------
Name                     |                                        Status codes
------------------------------------------------------------------------------
GET auth /disability/h.. |                                        13,039 [200]
GET auth login           |                                           125 [200]
GET static asset         |                                        79,984 [200]
POST auth login          |                                           125 [200]
-------------------------+----------------------------------------------------
Aggregated               |                                        93,273 [200]
                                                                              
=== OVERVIEW ===                                                              
------------------------------------------------------------------------------
Action       Started               Stopped             Elapsed    Users       
------------------------------------------------------------------------------
Increasing:  2022-06-03 14:49:39 - 2022-06-03 14:49:52 (00:00:13, 0 -> 125)   
Maintaining: 2022-06-03 14:49:52 - 2022-06-03 14:52:52 (00:03:00, 125)        
Decreasing:  2022-06-03 14:52:52 - 2022-06-03 14:52:54 (00:00:02, 0 <- 125)   

Target host: https://staging.cms.va.gov/                                      

Load on STAGING:

[ElijahLynn]

I cannot upload HTML files but here is the HTML report zipped up from this command, and I took a full page screenshot too:

 cargo run --release -- --host https://staging.cms.va.gov --report-file report-file.html --running-metrics 1 --users 125 --hatch-rate 10 --run-time 3m --no-reset-metrics

goose-load-test-report-20220603-staging.cms.va.gov.zip