Improve search index performance with lazy loading and compression

[seansica]

The current search solution for a website involves a FlexSearch Document that has two search indexes for page titles and content. This is used to index around 3704 objects on 1852 pages. It takes approximately 5-6 seconds to index all objects using the SearchService.initializeAsync method, which causes a noticeable delay for website visitors when the website is loaded for the first time.

The delay is caused by two things: downloading the JSON files containing the index-able items and the time it takes to index them in the FlexSearch Document. However, the delay only persists the first time a user visits the website because the search index and the index-able items are cached in IndexedDB, which allows the search index to be restored without having to re-download the items and re-generate the search index.

To improve the user experience, we can potentially reduce the time it takes to generate the search index by assigning each search index to a web worker and increasing the number of search indexes. This will enable lazy loading of search results as they are returned from each individually threaded search index, resulting in an improved user experience because results can be rendered as they are yielded. This is faster than generating and querying one massive search index.

Additionally, we can potentially reduce the time incurred from downloading JSON files by reducing the size of the files and using compression.