migration: performance issues

[jacquerie]

For the past week nightly migration has been unusually slow. While previously migration of all records terminated around 3:00 AM, now, by the time we get back at the office, only half of it is done.

Here's a profile of migrate('dumps/all_XXX.xml.gz', wait_for_results=True), obtained with the techniques described in 9b0f4a8: https://cernbox.cern.ch/index.php/s/D0rPrCM68FaqGPh

It's clear that the problem lies in get_record_collections, part in _build_cache, and part in _find_matching_collections_internally.

• Diagnose the performance issue
• Document the procedure followed to diagnose (docs: profiling #1637)
• Solve the performance issue in _build_cache (ext: configurable cache addition inveniosoftware/invenio-collections#69)
• Solve the performance issue in _find_matching_collections_internally (ext: configurable matcher addition inveniosoftware/invenio-collections#73)
• Release invenio-collections (WIP release: v1.0.0a4 inveniosoftware/invenio-collections#74)
• Implement a lighter matcher in inspire-next

[jacquerie]

By manually adding a cache to invenio-collections with

>>> from inspirehep.modules.cache import current_cache
>>> app.extensions['invenio-collections'].cache = current_cache

we were able to work around the performance issue in _build_cache. Here's a new profile that proves it: https://cernbox.cern.ch/index.php/s/XfnXKM9i8Qggd0c

This workaround has to be translated in a proper solution.

[jacquerie]

This workaround has to be translated in a proper solution.

@jirikuncar suggests:

[...] you can make the COLLECTION_CACHE configurable using similar pattern as we do in Invenio-Access (https://github.com/inveniosoftware/invenio-access/blob/master/invenio_access/ext.py#L54-L59).

[jacquerie]

Despite what the docstring says, setting COLLECTIONS_USE_PERCOLATOR = True appears to have improved a lot with the performance. Here's a newer profile that proves it: https://cernbox.cern.ch/index.php/s/klvHbciBjgx2TqE (note that this is also using the workaround described in #1640 (comment)).

Now I just manually started a nightly build. Let's see how many records are going to be there at 9.

[jacquerie]

Now I just manually started a nightly build. Let's see how many records are going to be there at 9.

The problem with this is that the percolator puts too much pressure on ES, and record inserts are lost, so we cannot go this route.

I outlined an alternative fix here: inveniosoftware/invenio-collections#72