[PERF] improve quads-web render performance for large amounts of hosts

[sadsfae]

Summary

In QUADS 2.1.0 we moved to generating and hosting the dynamic wiki, inventory and pages directly from flask and our APIv3 via a separate gunicorn process e.g. quads-web instead of generating the same data in markdown as a separate process and then pushing that over a Python XMLRPC interface into Wordpress via an API.

Problem Details

We are seeing slow page render times for /assignments and /inventory when a large amount of hosts (1000+ or more) are actively scheduled. This has evaded us in our testing in smaller environments due to not running this at such a large scale in the Scale Lab (1200+ hosts, actively scheduled). This is not evident in smaller environments.

Page Load Observations

Page	Avg Load Time	Nginx Payload Render	Page Size	Scheduled Hosts	Total Hosts	Active Env	All Hosts Render per Host	Per Scheduled Host Render
/assignments	50 sec	1-3 sec	2584 KB	1064	1172	37	40ms	44ms
/inventory	35 sec	1-3 sec	829 KB	1064	1172	37	27ms	30ms

Page Render Mechanics

• For /assignments each host across each active environment has to be queried for its schedule start, schedule end which is done sequentially for each active environment. Across a lot of active hosts with schedules this adds up. There is other date calculation and content structuring but this is relatively fast. Nginx also delivers great performance and most of the time is simply waiting for the payload to generate so it can be rendered.

https://github.com/redhat-performance/quads/blob/latest/src/quads/web/app.py#L163
https://github.com/redhat-performance/quads/blob/latest/src/quads/web/controller/CloudOperations.py#L95
https://github.com/redhat-performance/quads/blob/latest/src/quads/web/controller/CloudOperations.py#L44
https://github.com/redhat-performance/quads/blob/latest/src/quads/web/controller/CloudOperations.py#L109

There is other date calculation and content structuring but this is relatively fast. Nginx also delivers great performance and most of the time is simply waiting for the payload to generate so it can be rendered.

• For /inventory each host has its cloud owner and active current cloud environment queried sequentially and iterated by rack.

https://github.com/redhat-performance/quads/blob/latest/src/quads/web/app.py#L256
https://github.com/redhat-performance/quads/blob/latest/src/quads/web/app.py#L279

Suggestions for Improvement

Ajax / JQuery / Intelligent Rendering

• Use Ajax / jquery 1 2 to only render the summary parts of /assignments and only when someone drills into a specific environment is that part of the page loaded or otherwise use intelligent / background loading on-demand.

Look into asyncio for potentially expensive iterative loops

• Look into using asyncio in the jinja2 content rendering calls and methods where possible, this is done in many other parts of the code-base with great results.

Other Rendering Optimizations

• Potentially look at rendering /inventory first by cloud membership (--cloud-only) to potentially lessen amount of calls, then sort by rack to speed up render times for jinja2 content generation.

• Add additional helper unification methods for src/quads/web/app.py to reduce the amount of Rest API calls required in pulling in all the data needed for page render payloads.